question about applying convert.py on mixed precision models like meta-llama/Llama-2-13b-chat-hf

[rchen19]

As I understand it, models like meta-llama/Llama-2-13b-chat-hf contains both fp16 and fp32 tensors, so I am wondering:

1. When I set --outtype fp16, do all the fp32 tensors in the model gets converted to fp16 and the tensors that are already fp16 gets no change?
2. And When I set --outtype fp32, do all the fp16 tensors in the model gets converted to fp32, and the fp32 tensors does not change?

Thanks!

[slaren]

I think that all the tensors in the llama-2 model files distributed by meta are BF16. When converting or quantizing the model to GGUF, some of these tensors are always exported as FP32, regardless of the --outtype parameter or the quantization format. The reason for this is due to an implementation detail of ggml - these tensors are used in a ggml_mul operation that does not support FP16 or quantized types. These tensors are very small, so this does not affect significantly the size of the model files.

[rchen19]

I got the idea of llama being mixed precision from here: https://huggingface.co/meta-llama/Llama-2-13b-chat-hf, on the right side of the page it says Tensor type F32 FP16, not sure if there's a difference from the original meta's own version. I am converting from the HF version though.

[rchen19]

OK, I think the original meta version of Llama-2 was released in bf16, but the HuggingFace hosted version is mostly fp16 with some layers being f32:

https://magazine.sebastianraschka.com/p/the-missing-bits-llama-2-weights

[devzzzero]

Hi, looking through the gguf-py code a bit, it looks like gguf does NOT support bf16 at all, only fp16 and fp32. And I didn't see any tests where the bf16's exp field is examined to see if a fp32 would be better (i.e. to not lose bits in the exponent). Assuming I'm correct, does this mean that the as is, it seems like there is a built in conversion error when moving from original tensors in bf16 (e.g. llama 3) to gguf.

Am I correct in this? @ggerganov ? Can you demystify?

Thank you.

[compilade]

@devzzzero There has been recent changes in convert-hf-to-gguf.py and gguf-py to support bf16 conversion, see #7158.
And bf16 is supported in GGUF since #6412, which was merged 5 days ago.

[devzzzero]

Wow! @compilade @ggerganov Thank you. I'll pull and recompile.