What are the types of quanitzation scheme used in llama.cpp

[mrtpk]

Hello Team,

I have been learning about transformer quantization and is particularly interested in full integer 8-bit quantization. What quantization scheme (like GPTQ, AWQ, SmoothQuant) is supported in full integer quantization in llama.cpp? To give a context, TFLite uses symmetric int8 quantization for calibration and inference.

Or does llama.cpp support any quantization type as long as the format is in GGUF or GGML? I really appreciate any help or pointers. Thanks.

[BarfingLemurs]

https://newsletter.maartengrootendorst.com/p/a-visual-guide-to-quantization

Breakdown for K quantization, but more information about them is in pull requests.

[mrtpk]

Thank you @BarfingLemurs for taking the time to respond; It took me a couple of days to digest all these.

Please correct me if I'm wrong. I understand that in llama.cpp the quantization supported is K-bit quantization. The quantization is done by calculating an importance matrix (it can be hessian matrix if the method is GPTQ). We can employ any quantization scheme like AWQ and use it in llama.cpp if we respect the container format - GGUF or GGML.

One question that I am still trying to figure out is the following:

When the matrix multiplication is done, will the quantized value (value can be in bfloat16, or int8) transformed to float32 for the calculation? or does llama.cpp follow a different calculation mechanism?

From my dig, the quantized value is converted into float32 for calculations. See ggml_vec_dot_q8_0_q8_0.

I'm Looking forward to your response.

Thank you.