llama3 model conversion to gguf format for llama.cpp

[cc-lai]

Hi,

Hope I can get some pointers to get me in the right direction.

Background:
I understand that meta's llama3 model just gone GA not long ago. Have been using llama.cpp (it works great, thank you!) on gguf format of llama2 model. Wanted to try out the llama3, so conversion is required to obtain the gguf format of the model.

Progress:
Have used (convert_llama_weights_to_hf.py) to convert from Meta format (.pth) to hf (.safetensors, why is it not pytorch bin format??) format successfully. I understand from README that convert.py does not work for llama3 model at the moment, have to use 'convert-hf-to-gguf.py' instead.

When I run 'convert-hf-to-gguf.py' something like following, it throws an error about MODEL_ARCH. What did I miss?
Don't think I am using ORION.

--------- console output --------------->
[userid]$ python convert-hf-to-gguf.py hf-model-path/ --outfile output-file-path --outtype bf16
Traceback (most recent call last):
File "/home/myhome/convert-hf-to-gguf.py", line 787, in
class OrionModel(Model):
File "/home/myhome/convert-hf-to-gguf.py", line 788, in OrionModel
model_arch = gguf.MODEL_ARCH.ORION
^^^^^^^^^^^^^^^^^^^^^
AttributeError: type object 'MODEL_ARCH' has no attribute 'ORION'

I have read "Tutorial: How to convert HuggingFace model to GGUF format #2948", couldn't find anything that works.

[cloak1505]

Did you clone llama.cpp?

Without convert.py, when you run convert-hf-to-gguf.py you get ModuleNotFoundError: No module named 'convert'.
Without gguf-py folder, you get AttributeError: type object 'MODEL_ARCH' has no attribute 'ORION'.
I'm not sure what models folder and convert-hf-to-gguf-update.py do or if they are needed. As a casual user I have specifically made Llama 3 bf16.gguf with convert-hf-to-gguf.py and quantized with quantize.exe.

[cc-lai]

Thanks! It brings me closer, I knew something was missing.

Now I am getting this error:
WARNING:hf-to-gguf:**************************************************************************************
WARNING:hf-to-gguf:** WARNING: The BPE pre-tokenizer was not recognized!
WARNING:hf-to-gguf:** There are 2 possible reasons for this:
WARNING:hf-to-gguf:** - the model has not been added to convert-hf-to-gguf-update.py yet
WARNING:hf-to-gguf:** - the pre-tokenization config has changed upstream
WARNING:hf-to-gguf:** Check your model files and convert-hf-to-gguf-update.py and update them accordingly.
WARNING:hf-to-gguf:** ref: #6920
WARNING:hf-to-gguf:**
WARNING:hf-to-gguf:** chkhsh: c136ed14d01c2745d4f60a9596ae66800e2b61fa45643e72436041855ad4089d
WARNING:hf-to-gguf:**************************************************************************************
WARNING:hf-to-gguf:

Traceback (most recent call last):
File "llama.cpp/convert-hf-to-gguf.py", line 1287, in set_vocab
self. _set_vocab_sentencepiece()
File "llama.cpp/convert-hf-to-gguf.py", line 572, in _set_vocab_sentencepiece
raise FileNotFoundError(f"File not found: {tokenizer_path}")
FileNotFoundError: File not found: ../Meta-Llama-3-8B-hf/tokenizer.model

If I copy the tokenizer.model file distributed by Meta, the following error will show up:
RuntimeError: Internal: could not parse ModelProto from ../Meta-Llama-3-8B-hf/tokenizer.model

Do I have to run convert-hf-to-gguf-update.py? What is the syntax like?

[Indirajith-jithu]

Hi Any update on i got the same issiue

[cloak1505]

Hopefully someone smarter than me can answer about tokenizer stuff. I have no knowledge on actually making/finetuning models, etc. My use case is to simply download existing models. I have never touched convert-hf-to-gguf-update.py.

?/Meta-Llama-3-8B and Undi95/Meta-Llama-3-8B-hf (ignore Original folder) worked for me without warnings or errors.

[nick-ivanov-edb]

Looks like the gguf module on PyPI has not been updated in a while -- if that's how you install it, perhaps grab the module directly from the llama.cpp repo

[Neeraj-Natu]

It's 2025 now is there anyone found a consistent solution for converting the .pth files into .gguf format which could be used along with different LLM frameworks.