Skip to content

Commit e1fdf35

Browse files
fairydreamingsszymczy
authored andcommitted
llama : implement Unigram tokenizer needed by T5 and FLAN-T5 model families (ggml-org#5763)
* llama : add T5 model architecture, tensors and model header parameters * llama : add implementation of Unigram tokenizer with SentencePiece-like text normalization using precompiled charsmap --------- Co-authored-by: Stanisław Szymczyk <sszymczy@gmail.com>
1 parent 9e3d7d6 commit e1fdf35

File tree

4 files changed

+587
-39
lines changed

4 files changed

+587
-39
lines changed

0 commit comments

Comments
 (0)