sentencepiece is not the same #89
Description
I used sentencepiece model with google lib
but inference of the rust-tokenizers is different. What do I need to do to get the output to be the same as the google library?
{{ message }}
I used sentencepiece model with google lib
but inference of the rust-tokenizers is different. What do I need to do to get the output to be the same as the google library?