Which quantized gguf size infers at INT8?

[segmond]

I see references to GGUF_TYPE_INT8 in ggml.c. Is there any particular gguf size that is guaranteed to be in that or does it have to be generated a certain way?

[sorasoras]

I think that's ARM Cpu thing.

[segmond]

Got it, would it be possible to infer in int8? and how can we support that? Telsa P40 int8 yields about 47 teraflops, RTX 3090 f16/f32 have about 35+ teraflops. Theoretically it sounds like we should see better performance from the P40 than 3090 if we have our data in the right format.

[BarfingLemurs]

https://github.com/ggerganov/llama.cpp/pull/4966/files

It should work on certain ARM platforms with Q4_0 and Q8_0 (at least) as shown in the discussions.