No quantization for activations?

[Z-KN]

There are a lot of quantization options for weights, I wonder whether there is a quantization process for activations?

When I add printf in ggml_compute_forward_mul_mat function, it shows the src0 tensor has data type of either 1, 2, or 14 (meaning fp16, q4_0, and q6_k respectively), while src1 always has data type of 0, which stands for fp32. So I infer that llama.cpp never quantizes activations? All tokens and activations are computed using floating points?

    const enum ggml_type type = src0->type;
    const enum ggml_type type1 = src1->type;
    printf("type = %d, type1 = %d\n", type, type1);  // shows type = 1/2/14, type1 = 0

[KerfuffleV2]

So I infer that llama.cpp never quantizes activations?

1D tensors don't get quantized since they're so small. There's basically no advantage to reducing the quality because you don't really gain anything. I don't think it has anything to do with activations, just the size of certain tensors. edit: Please disregard.

[Z-KN]

But with fixed point activations, say int8, can't we get faster inference speed? Not merely about the size.

[KerfuffleV2]

I don't know about that part. Also, as far as I know during inference the values get dequantized to 32bit before the operations are performed. You might be right about squeezing more performance out, but it would need to be something that worked with the various architectures (CPU, Metal, CUDA, etc) llama.cpp supports if it was implemented at the quantization level. edit: Please disregard.

[rikoras]

I guess there may be some unpredictable troubles applying integer quantization to activations, since some non-linear operators may cause math-related issues.
There is a paper which may provide a better illustration.
https://proceedings.mlr.press/v139/kim21d.html

[ggerganov]

@KerfuffleV2
The term "activations" refers to the intermediate results obtained during the evaluation of the transformer. It does not mean the 1D tensors in the model. This is a terminology that I also recently learned

The activations in ggml are generally quantized when:

• running on the CPU
• running with CUDA

They are not quantized yet with Metal.

On the CPU, even though src1 has type F32, it is still being quantized internally in the matrix multiplication call:

https://github.com/ggerganov/llama.cpp/blob/a40f2b656fab364ce0aff98dbefe9bd9c3721cc9/ggml.c#L11333-L11349

The activations are always quantized to 8-bits (.vec_dot_type):

https://github.com/ggerganov/llama.cpp/blob/a40f2b656fab364ce0aff98dbefe9bd9c3721cc9/ggml.c#L1616-L1781

For more info:

• Investigate alternative ggml_compute_forward_mul_mat_q_f32() implementation #909
• Add Q8_0 quantization for intermediate results #951

[KerfuffleV2]

Thanks! Sorry for the misinformation. I edited those comments.

[Z-KN]

Thank you very much!! I failed to read the code in more detail.

[manu-web]

How do we verify for sure what is the activation quantization?
And also for AMD GPUs what is the quantization that we are using?