Create README.md for V2Index and TokenDFA

Author: agourdel

src/tokens_dfa/README.md

@@ -0,0 +1,69 @@
+# V2Index and TokensDFA
+
+> The current Index is a naive implementation. It means for a given DFA build from a regex it will 'bruteforce' 
+> each state encountered during progression in the graph with all the tokens in order to build the tokens transitions table.
+> This results in a complexity proportional to the size of the model vocabulary, the average size of the tokens in bytes and the complexity of the regex. (The complexity of a regex will be defined later.)
+> The following is the will of  build an approach that takes the behaviors of DFA for regexes and extends them to the token scale in order to be less burdened by the complexity of regexes and the size of vocabularies.
+> 
+> At the end, the V2Index has much better compile-time performance than its predecessor, much better performance in serving the list of allowed tokens for each state, and takes up less memory in most cases.
+ ---
+
+ ## TokensDFA
+
+This new version of Index includes a TokensDFA object.
+This TokenDFA can be seen as an extension of DFA in that it leverages DFA optimizations to reduce the computational complexity of constructing the tokens transitions table.
+The trade-off that is made is to spend time upstream of the construction of the transition table in order to gain advantages during construction.
+
+***Regex's world is such a childish world. Only 256 different values to manage, all of them with one byte size. 
+Tokens world has no limit of different values with no limit of size. Dante described it as "Malebolge"***
+
+
+```rust
+pub struct TokensDFA
+ {
+    pub eos_token_id:u32,
+    pub eos_class_id:u32,
+    pub start_state: StateId,
+    pub final_states: HashSet<StateId>,
+    pub transitions_table: MasksTable,  
+}
+``` 
+The structure of the TokensDFA is very similar to the current index. The difference lies in the initialization.
+A series of five optimizations has been implemented:
+
+### 1. Reduce Vocabulary size
+
+A static analysis of the regex is made in order to make the list of the 'dead bytes'.
+'dead bytes' are bytes that will not be allowed at any place in the regex.
+It allows us to quickly discriminate all the tokens that have at least one of the dead bytes.
+```rust
+let byte_classes = dfa.byte_classes();
+let mut dead_byte_classes:HashSet<u8> = compile_dead_byte_classes(&muted_regex, &byte_classes);
+```
+Before going further, one thing very important to know about DFA is, when it compile,  it tries to regroup bytes by class.
+Bytes in the same class has same effect on the regex's graph.
+```regex
+"^[a-z]$"
+```
+In this example, all the char from 'a' to 'z' has the same class because they trigger the same behavior.
+So, there are 2 states and only one transition.
+Conversely, with the regex `"^a[a-z]$"` the char 'a' will have a different class than the chars 'b' to 'z'. 
+Because only the 'a' is allowed as transition at state 0. Then, two classes are allowed. The one of 'a' and the one of [b-z].
+It allows the DFA to reduce drastically the number of transitions by considering classes as transitions values.
+
+We will use and abuse of these classes.
+
+### 2. Tokens Classification
+
+We take the ByteClasses of the DFA and we construct the class of each token by concating the classes of each of their byte.
+In other world, if the range of bytes `[a-z]` has the class `[a]`, the token `'man'` will have the class `[a][a][a]` like all the 
+tokens of 3 letters. 
+We reduce the size 
+
+
+    
+
+
+ 
+ 
+