fix: pretokenize tokens before checking vocabulary#215

stephantul merged 1 commit intomainMinishLab/model2vec:mainfrom

fix-duplicate-tokensMinishLab/model2vec:fix-duplicate-tokensCopy head branch name to clipboard