How to use 7-bit integer quantization?

[GermanAizek]

ٴ

[cmp-nct]

If you use the available Q6_K type quantization you'll end up with around 6.6 bpw. It shows performance equal to fp8 and very similar to fp16.
You can quantize models using the "quantize" binary and you can load them just like any gguf