GGUF

[imkebe]

I was trying to convert the GGUF model but there is a major limitation that the script is not handling.
The RKLLM is only cappable of converting the output weigths of q_4_0 and fp16 quants. It's not about the generic GGUF quant defined in filename but the output weight.

For example there GGUF files from q_2 to q_6 of Qwen 2.5 has output of q_6. Only the q_8 has q_8 output quant and then the fp16 has fp16.

In fact there is no way of converting Qwen2.5 using GGUF other than fp16... In this case i would use HF instead.

Second thing is that the GGUF logic downloads all files while the GGUF loading function expects only the one specific filename - it should use q_4_0 probably however in real scenario we should first determine if the output quant is q_4_0. I don't know if the HF library has an option to read metadata ? Because there is the weights detailed information.

[c0zaut]

I had that epiphany last night - I need to provide some logic for finding/selecting the specific GGUF file. I read the documentation and the quant support with GGUF doesn't seem great, which is why I have been largely focusing on converting the Huggingface versions. Once I have had a chance to update for the new 1.1.4 library to support vision models, I'm going to circle back around to GGUF.

Once this that IS very interesting, though, is if you look at the Rockchip LoRA example, it actually loads a GGUF file: https://github.com/airockchip/rknn-llm/blob/8623edd0559a07e7127876d685f2b7ca8b83590c/examples/rkllm_api_demo/src/llm_demo.cpp#L144

I haven't actually tested directly loading a GGUF, but it does seem interesting.

[imkebe]

I've tested the GGUF. I've modifired you code heavly, but no luck with conversion since the models i've used like Qwen 2.5 have the Q_6 output quants. Not Q_4. Also i don't see any reason to use GGUF - the usage is limited (to q_4), the compatible models architecure is limited, all are avaible basically as a HF safetensors. With just FP16 and int4 the memory footprint will not be reduced (while the int4 will be only for the RK3576).

You need first to load the metadata of the GGUF to verify the output.weight for Q_4. Only then download the GGUF. I don't feel it's worth it.

[c0zaut]

I agree - which is kind of silly given that the whole thing is GGUF-based (I think the models are encrypted with rknn, which is why the metadata can't be read with strings)