This repo contains pre-loaded tokenizers files from Hugging Face for the following models:
- Meta-Llama-3-8B (with extended vocab)
- Meta-Llama-3-8B-Instruct (with extended vocab)
- Meta-Llama-3.1-8B
- Meta-Llama-3.1-8B-Instruct
- Meta-Llama-3.1-70B
- Meta-Llama-3.1-70B-Instruct
- Meta-Llama-3.2-1B
- Meta-Llama-3.2-1B-Instruct
- Meta-Llama-3.2-3B
- Meta-Llama-3.2-3B-Instruct
- Mistral-7B-v0.3:
- Mistral-7B-Instruct-v0.3
- GPT2