[TODO] Implement complete grammar

[amynbe]

Tutorial:

• https://github.com/JetBrains/Grammar-Kit/blob/master/README.md
• https://github.com/tinku99/ahklexers
• https://www.autohotkey.com/docs/Language.htm

Difficulty: Hard, but:

How Important: Very. It's the foundation for:

• Highlighting,
• Correct Psi tree, which is needed for:
  • references/usages/symbol indexing
  • syntax errors
  • more complex semantic inspections

After looking into this, I came to the conclusion that the grammar can't be done incrementally. We should come up with a really good grammar from the beginning and then improve the remaining bits, not the other way around. Otherwise it'll get messy, inconsistent, and many cases will be overlooked.

To avoid reinventing the wheel, see the tinku99/ahklexers repo: it contains 3 things: a bnf file and 2 lexers for respectively a python and a php library. I think it's simpler to start by converting the bnf file to GrammarKit syntax, even though it's apparently an old grammar (without objects and arrays).

[takanuva15]

Thanks for posting the bnf/lex links; surprisingly it seems the bnf I wrote so far matches up decently with what's there (too bad it's a few yrs old lol). I was looking at how to parse functions/expressions and I was coming to a similar conclusion that it will need to be done mostly jointly for correct parsing.

We do need to be careful not to be hasty with a complete-write of the grammar. The original bnf that was committed in the first few commits of this plugin was attempted to be written in a similar fashion (ie all-at-once), but it became buggy/unwieldy and threw a bunch of errors. (Our plugin users immediately raised 3+ bugs and the bnf was reverted back to just highlighting comments.)

I will plan to work on this along with #53 asap.

[amynbe]

Sounds good.
I was mainly looking at references support and this is where we need to have the parser create appropriate PsiElements for function calls and function declarations, to be able to navigate from one to the other. Similarly from variable usage to the variable declaration.
Sadly the language has such a messy syntax (legacy vs new syntax, etc), and is quite dynamic in its syntax and type system, that it's going to be tricky to get this right...