What's Changed
- docs: update chonkie link on tutorial readme by @iaurg in #235
- Fix dates in README.md by @Pringled in #238
- fix: add default arg for push_to_hub by @stephantul in #240
- fix: remove direct dependency on specific hf utils by @stephantul in #244
- feat: smaller tokenizers by @stephantul in #243
- feat: update lock by @stephantul in #246
- feat: allow passing validation set explicitly by @JarbasAl in #245
- docs: Added multilingual results by @Pringled in #247
- fix: distillation for models without card by @JarbasAl in #248
- feat: add supertokenizers by @stephantul in #236
- clean-up print statement by @stephantul in #249
- fix: small typing issue by @stephantul in #250
- docs: Added new logo by @Pringled in #252
- fix: missing unk, fix bug by @stephantul in #251
- bump version by @stephantul in #258
- feat: make normalization dependent on spacing by @stephantul in #259
New Contributors
Full Changelog: v0.5.0...v0.6.0
Contributors
stephantul, iaurg, and 2 other contributors