Why do all Q dot product methods use Q8 for non-model tensors?

[tjake]

Looking at ggml code:

https://github.com/ggerganov/llama.cpp/blob/master/ggml.c#L1675

Why do all the quantized dot products encode working tensors (non-model) into q8 first?
Isn't this more work and memory bandwidth than just sending the F16 or F32 directly?

Thanks!

[slaren]

These tensors are usually very small, and the cost of quantizing them is low compared to the overall cost of the matrix multiplication. For example, during generation with llama 7B, most matrix multiplications have dimensions [4096,4096] x [4096]. By quantizing these tensors to integers, it is possible to use mostly integer ops during the dot products, with only a few floating point operations in the end to scale the result. This is usually a lot faster than using only floating points operations for the entire matrix multiplication. There may also be a small performance gain due to reduced memory bandwidth requirements, but I believe that most of it is from the use of integer ops.