File tree
9 files changed
+161
-20
lines changed- Sources/Tokenizers
- Tests
- NormalizerTests
- TokenizersTests
- Resources
9 files changed
+161
-20
lines changedLines changed: 42 additions & 8 deletions
Original file line number | Diff line number | Diff line change | |
---|---|---|---|
| |||
10 | 10 |
| |
11 | 11 |
| |
12 | 12 |
| |
13 |
| - | |
| 13 | + | |
14 | 14 |
| |
15 | 15 |
| |
16 | 16 |
| |
| |||
30 | 30 |
| |
31 | 31 |
| |
32 | 32 |
| |
33 |
| - | |
| 33 | + | |
| 34 | + | |
34 | 35 |
| |
35 | 36 |
| |
36 | 37 |
| |
| 38 | + | |
37 | 39 |
| |
38 | 40 |
| |
39 | 41 |
| |
| |||
52 | 54 |
| |
53 | 55 |
| |
54 | 56 |
| |
55 |
| - | |
| 57 | + | |
| 58 | + | |
56 | 59 |
| |
57 | 60 |
| |
58 | 61 |
| |
| |||
154 | 157 |
| |
155 | 158 |
| |
156 | 159 |
| |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
157 | 166 |
| |
158 | 167 |
| |
159 | 168 |
| |
160 |
| - | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
| 175 | + | |
| 176 | + | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
161 | 180 |
| |
162 |
| - | |
163 |
| - | |
| 181 | + | |
164 | 182 |
| |
165 | 183 |
| |
166 | 184 |
| |
167 | 185 |
| |
168 | 186 |
| |
169 | 187 |
| |
170 |
| - | |
171 |
| - | |
| 188 | + | |
| 189 | + | |
172 | 190 |
| |
173 | 191 |
| |
174 | 192 |
| |
| |||
187 | 205 |
| |
188 | 206 |
| |
189 | 207 |
| |
| 208 | + | |
| 209 | + | |
| 210 | + | |
| 211 | + | |
| 212 | + | |
| 213 | + | |
| 214 | + | |
| 215 | + | |
| 216 | + | |
| 217 | + | |
| 218 | + | |
| 219 | + | |
| 220 | + | |
| 221 | + | |
| 222 | + | |
| 223 | + | |
190 | 224 |
| |
191 | 225 |
| |
192 | 226 |
| |
|
Lines changed: 1 addition & 1 deletion
Original file line number | Diff line number | Diff line change | |
---|---|---|---|
| |||
58 | 58 |
| |
59 | 59 |
| |
60 | 60 |
| |
61 |
| - | |
| 61 | + | |
62 | 62 |
| |
63 | 63 |
| |
64 | 64 |
| |
|
Lines changed: 7 additions & 9 deletions
Original file line number | Diff line number | Diff line change | |
---|---|---|---|
| |||
146 | 146 |
| |
147 | 147 |
| |
148 | 148 |
| |
149 |
| - | |
| 149 | + | |
150 | 150 |
| |
151 | 151 |
| |
152 | 152 |
| |
153 | 153 |
| |
154 | 154 |
| |
155 |
| - | |
156 | 155 |
| |
| 156 | + | |
157 | 157 |
| |
158 | 158 |
| |
159 | 159 |
| |
| |||
164 | 164 |
| |
165 | 165 |
| |
166 | 166 |
| |
167 |
| - | |
| 167 | + | |
168 | 168 |
| |
169 | 169 |
| |
170 | 170 |
| |
| |||
219 | 219 |
| |
220 | 220 |
| |
221 | 221 |
| |
222 |
| - | |
223 |
| - | |
224 |
| - | |
225 |
| - | |
226 |
| - | |
227 |
| - | |
| 222 | + | |
| 223 | + | |
| 224 | + | |
| 225 | + | |
228 | 226 |
| |
229 | 227 |
| |
230 | 228 |
| |
|
Lines changed: 4 additions & 1 deletion
Original file line number | Diff line number | Diff line change | |
---|---|---|---|
| |||
63 | 63 |
| |
64 | 64 |
| |
65 | 65 |
| |
| 66 | + | |
| 67 | + | |
66 | 68 |
| |
67 | 69 |
| |
68 | 70 |
| |
| |||
270 | 272 |
| |
271 | 273 |
| |
272 | 274 |
| |
273 |
| - | |
| 275 | + | |
| 276 | + | |
274 | 277 |
| |
275 | 278 |
| |
276 | 279 |
| |
|
Lines changed: 37 additions & 0 deletions
Original file line number | Diff line number | Diff line change | |
---|---|---|---|
| |||
120 | 120 |
| |
121 | 121 |
| |
122 | 122 |
| |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
123 | 136 |
| |
124 | 137 |
| |
125 | 138 |
| |
| |||
133 | 146 |
| |
134 | 147 |
| |
135 | 148 |
| |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
136 | 173 |
| |
137 | 174 |
| |
138 | 175 |
| |
|
Lines changed: 1 addition & 0 deletions
Original file line number | Diff line number | Diff line change | |
---|---|---|---|
| |||
| 1 | + |
Lines changed: 1 addition & 0 deletions
Original file line number | Diff line number | Diff line change | |
---|---|---|---|
| |||
| 1 | + |
Lines changed: 1 addition & 1 deletion
Large diffs are not rendered by default.
Lines changed: 67 additions & 0 deletions
Original file line number | Diff line number | Diff line change | |
---|---|---|---|
| |||
60 | 60 |
| |
61 | 61 |
| |
62 | 62 |
| |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
63 | 75 |
| |
64 | 76 |
| |
65 | 77 |
| |
| |||
108 | 120 |
| |
109 | 121 |
| |
110 | 122 |
| |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
| 175 | + | |
| 176 | + | |
| 177 | + | |
111 | 178 |
| |
112 | 179 |
| |
113 | 180 |
| |
|
0 commit comments