You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
[3rdparty] Bump tokenizer-cpp to enable SentencePiece by default (#3025)
This PR bumps the 3rdparty/tokenizer-cpp.
The SentencePiece tokenizer was disabled by default to
reduce the binary size of the built library, while it causes
some error when users expect to use the SentencePiece
tokenizer.
This PR enables it by default. And we will need to manually
disable it if we need to reduce its binary size.
0 commit comments