Error mapping tensor using convert_hf_to_gguf.py

[vcc715]

Hi,
I tried running python3 convert_hf_to_gguf.py ./lora_fused_model but I get this message indicating a ValueError mapping 'model.embed_tokens.biases' :

INFO:hf-to-gguf:Loading model: lora_fused_model
INFO:gguf.gguf_writer:gguf: This GGUF file is for Little Endian only
INFO:hf-to-gguf:Set model parameters
INFO:hf-to-gguf:gguf: context length = 8192
INFO:hf-to-gguf:gguf: embedding length = 4096
INFO:hf-to-gguf:gguf: feed forward length = 14336
INFO:hf-to-gguf:gguf: head count = 32
INFO:hf-to-gguf:gguf: key-value head count = 8
INFO:hf-to-gguf:gguf: rope theta = 500000.0
INFO:hf-to-gguf:gguf: rms norm epsilon = 1e-05
INFO:hf-to-gguf:gguf: file type = 1
INFO:hf-to-gguf:Set model tokenizer
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
INFO:gguf.vocab:Adding 280147 merge(s).
INFO:gguf.vocab:Setting special token type bos to 128000
INFO:gguf.vocab:Setting special token type eos to 128009
INFO:gguf.vocab:Setting chat_template to {% set loop_messages = messages %}{% for message in loop_messages %}{% set content = '<|start_header_id|>' + message['role'] + '<|end_header_id|>

'+ message['content'] | trim + '<|eot_id|>' %}{% if loop.index0 == 0 %}{% set content = bos_token + content %}{% endif %}{{ content }}{% endfor %}{% if add_generation_prompt %}{{ '<|start_header_id|>assistant<|end_header_id|>

' }}{% endif %}
INFO:hf-to-gguf:Exporting model...
INFO:hf-to-gguf:gguf: loading model weight map from 'model.safetensors.index.json'
INFO:hf-to-gguf:gguf: loading model part 'model-00001-of-00003.safetensors'
Traceback (most recent call last):
File "./llama.cpp/convert_hf_to_gguf.py", line 3631, in
main()
File "./llama.cpp/convert_hf_to_gguf.py", line 3625, in main
model_instance.write()
File "./llama.cpp/convert_hf_to_gguf.py", line 337, in write
self.write_tensors()
File "./llama.cpp/convert_hf_to_gguf.py", line 1507, in write_tensors
super().write_tensors()
File "./llama.cpp/convert_hf_to_gguf.py", line 274, in write_tensors
for new_name, data in ((n, d.squeeze().numpy()) for n, d in self.modify_tensors(data_torch, name, bid)):
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "./llama.cpp/convert_hf_to_gguf.py", line 1504, in modify_tensors
return [(self.map_tensor_name(name), data_torch)]
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "./llama.cpp/convert_hf_to_gguf.py", line 192, in map_tensor_name
raise ValueError(f"Can not map tensor {name!r}")
ValueError: Can not map tensor 'model.embed_tokens.biases'

would appreciate any help with this issue. Thanks.

[compilade]

Which model is this?

I want to know if model.embed_tokens.biases should be added to the ignored tensor list, and if it's model-specific, for which model architecture. (EDIT: from the line numbers, it seems like a Llama-like model)

[vcc715]

The model is derived from Llama-3-8B-Instruct and here is the path to it:
https://huggingface.co/mlx-community/log-parser

[compilade]

Thanks. In this case, it seems like model.embed_tokens.weight is pre-quantized, since that tensor is in U32, and it's accompanied with scales and biases.

The convert script does not yet support that, unfortunately. The model.embed_tokens.weight tensor would need to be dequantized first by applying model.embed_tokens.scales and model.embed_tokens.biases to it (not sure how exactly).