explanation of the choice behind selective quantization + IQ quant grid

[afsara-ben]

A few questions-

1. For any quantization type (K-quant or IQ quant), different layers of a particular model are quantized to different bits such as, output layer of Q2_K is Q6_K and IQ1_M is Q5_k for llama 70b model. My question is how was this choice made? Was it empirical? And where in the codebase is the defined that say quantizing to Q2_K means this and this layers will be of this and this type (its not in quantize.cpp). How can I make my custom choice of quantized layers (using the --tensor-type)?

2. In IQ quant, (in ggml-quants.c), the kgrid_2bit_256 and other grid arrays are defined. How was this grid created?

3. I find mixed answers for whether K-quants employ K-means clustering or not, from the code I dont see any clustering implementation, what is K supposed to mean here?

[steampunque]

A few questions-

1. For any quantization type (K-quant or IQ quant), different layers of a particular model are quantized to different bits such as, output layer of Q2_K is Q6_K and IQ1_M is Q5_k for llama 70b model. My question is how was this choice made? Was it empirical? And where in the codebase is the defined that say quantizing to Q2_K means this and this layers will be of this and this type (its not in `quantize.cpp`). How can I make my custom choice of quantized layers (using the `--tensor-type`)?

src/llama-quant.cpp is where the quantization layer logic lives. This file is full of a bunch of empirical heuristics which modify the quantization of various tensors as a function of layer number and selected quant for file. If you want to switch things up from the default heuristics this is where the mods will go, or you can try overriding with the regexp thing recently added (I don't use it, I found it much easier to directly modify llama-quant.cpp for my needs).

I have been investigating hybrid layer quants (using less bits at deep layers and more bits at cortex layers) for some of the SOTA models and finding quite good results. The hybrid quant models can be made significantly smaller than homogeneous layer quants while performing just as well in my tests. Started with Llama scout and moving to Qwen 3, Mistral Small and Gemma 3. I summarize the layer quants I am using in each model on my model page at https://huggingface.co/steampunque. Original Scout experiment discussion #13040.

[afsara-ben]

thanks @steampunque. i also notice a new quant type iqk from ikawrakow/ik_llama.cpp. Do you know how it differs from a regular k quant?

[steampunque]

thanks @steampunque. i also notice a new quant type iqk from ikawrakow/ik_llama.cpp. Do you know how it differs from a regular k quant?

@afsara-ben I don't know the details of the quants except at a high level you don't want to be using IQ quants on CPU or old GPUs because they will be slow. Thus I stick with k quants for my hybrid layer quants. The hybrid layer quants can beat homogeneous IQ4_XS and probably the new iqk also on perplexity metric because they step outside of the uneeded constraint box of layer quant homogeneity. I don't use perplexity metric in assessing model quality when I optimize the hybrid layer quants, instead I use a small set of curated prompts covering knowledge, reasoning, code, and possibly vision abilities which help tell me if the quant is working as I want it to or if I went too far and lobotomized the model too much with the quant. The Exllama guy is also going off on a new tangent with some new QTIP quant if you are really interested in this stuff his github is another good one to check out https://github.com/turboderp-org/exllamav3.