SOTA Quants

[joseph777111]

@ikawrakow Are we able to us quantize to quantize gguf models using the new SOTA Quant methods that you implemented for Q2_K? Or, are they that still in the works? 🤔

[sorasoras]

I believe it is still need clean up before publish or he could open his repo.

[ikawrakow]

You are now able to quantize all IQ2_ and k-quants using an importance matrix, which leads to improved results. See PRs #4930, #4897, #4861, #4856, and #4773. Also, for convenience, I have uploaded importance matrix data for some models to this repo on Huggingface. If the importance matrix for your favorite model(s) is not there, you can compute using the new imatrix tool (see #4861).

[Nexesenex]

What is the matrix (dataset, context and chunks) you used to quantize your models in your SOTA directory on HF, @ikawrakow?
The quants of the Llama 2 70b you made are very good (benchs and use both), notably the IQ2_XS and Q2_K_S, the latter which usually shows only a marginal benefit vs IQ2_XS, but with yours actually behaves as expected.
Some PPL numbers :
Llama-v2-70b-Q2_K_S-2.66bpw.gguf,-,wikitext,4.2056,512
Llama-v2-70b-IQ2_XS-2.36bpw.gguf,-,wikitext,4.347,512
Llama-v2-70b-IQ2_XXS-2.12bpw.gguf,-,wikitext,4.7042,512

[ikawrakow]

https://huggingface.co/datasets/ikawrakow/imatrix-from-wiki-train

[joseph777111]

@ikawrakow, forgive me if I misunderstand, but: is the imatrix computed per model (eg: Nous-2-Hermes-Yi-34B) or per model architecture (eg: LLAMA-2, Mistral-7B, etc...)? I see in your huggingface repo that you have imatrices for both LLAMA-2-13B (base model) and also Nous-Hermes-2-Yi-34B (fine-tune). Can imatrices for base models be used for fine-tunes of the same model architecture and parameter size, or is it best to compute an imatrix for every model that you wish to quantize and improve (lower) the perplexity for? 🤔

Ps... I greatly appreciate the work that you, and the rest of the llama.cpp team have done. I never thought that I would be able to run a quantized version of a 34B parameter model within 16GB of RAM, but you guys made it possible! You're all awesome! 😁🚀

[ikawrakow]

You need a separate imatrix for each model. The imatrix of the base model does not work for a fine-tuned version (I tried once and it made it worse compared to not using imatrix at all).

[joseph777111]

@ikawrakow, thanks that's what I was thinking. But, I wanted to make sure before I computed the imatrix for the base model or vice versa. 😁