[3rdparty] Bump tokenizer-cpp to enable SentencePiece by default (#3025)

MasterJH5574 · web-flow · commit 53966dd9d98c · 2024-11-14T10:13:43.000-05:00
This PR bumps the 3rdparty/tokenizer-cpp.

The SentencePiece tokenizer was disabled by default to
reduce the binary size of the built library, while it causes
some error when users expect to use the SentencePiece
tokenizer.

This PR enables it by default. And we will need to manually
disable it if we need to reduce its binary size.
diff --git a/3rdparty/tokenizers-cpp b/3rdparty/tokenizers-cpp
@@ -1 +1 @@
-Subproject commit c0fab1e14a9421c1501acee5b7703e5dafa60479
+Subproject commit 528d695f48c03b25d47b1c9fd2ff2d052f5ea26a
