Enhance grammar by supporting Regex's curly bracket quantifiers

[Mihaiii]

Thanks for the grammar functionality, it's great! :)

Right now we have only the following Regex quantifiers: "+", "*", "?".

Considering real-world applications, integrating Regex's curly bracket quantifiers could be beneficial. For instance, I aim to have my model detail its thought process and present results in a format suitable for parsing and subsequent coding tasks. This format assurance is crucial for me, even if the model doesn't complete its entire "step by step" reasoning.

The model I'm currently working with uses up the context length with its reasoning, especially given my extensive prompts.

Introducing curly bracket quantifiers might lead to abrupt interruptions in the model's reasoning. However, my primary interest isn't the reasoning output itself, but ensuring the end output's format accuracy, which I derive from the grammar's end rules (which hold the format of the final response I can parse).

I believe this isn't just a specific need of mine. Given that grammars are predominantly used for integrations and instructing the model to reason "step by step" is a recognized technique to enhance reasoning accuracy.

@ejones What do you think?

[Mihaiii]

I could make a PR with this abstraction myself, but I'll need a review from @ejones - will you be ok to review it? (sorry for tagging you again - in case this message is ignored, I promise I won't bother you with this issue again).

One concern I have is performance - what negative impact the change I'd like to do would have, since there will be much more subrules?

My intention is to make the changes here:

{n} will be a for loop n times (ex: S{3} will become S S S)
{n,} will become a for loop for n times then same logic for n*.
{n,m} will become a for loop for n times then (S | ) | (S S | ) | (S S S | ) etc. OR, in case it's interpreted the same (is it?), it could become (S | ) m-n times.

If this is the right part of the code for the change, we might not need a code modification. Instead, we can use detailed grammar rules, but this could make them hard to read due to repetitions.

[ejones]

Hey, yes this is definitely needed and a contribution here would be welcome! And your approach sounds right, that is basically what I had in mind, using rewrite rules rather than changing the binary format. Need to wrap my head around the {n,m} case. Additionally, for higher numbers I was thinking the rewrite approach could leverage a tree of rules with some high branching factor, e.g. with 16:

S-16: S S S # ... 16 times
S-256: S-16 S-16 # ... 16 times