Search results

Add GPTNeoXTokenizerFast tokenizer.

Status: Open.

#104 In guillaume-be/rust-tokenizers;

· donpellegrino opened on May 25, 2025

Evaluate Profile-Guided Optimization (PGO) performance benefits for the library

Status: Open.

#103 In guillaume-be/rust-tokenizers;

· zamazan4ik opened on Apr 11, 2024

Port of 'rust-tokenizer' to C#/.NET

Status: Open.

#102 In guillaume-be/rust-tokenizers;

· vermorel opened on Apr 8, 2024

sentencepiece is not the same

Status: Open.

#89 In guillaume-be/rust-tokenizers;

· igor-yusupov opened on May 22, 2023

Reading SentencePieceVocab from text file

Status: Open.

#51 In guillaume-be/rust-tokenizers;

· MikaelCall opened on Apr 19, 2021