Measured Quantization probability differences for Mistral 7b and Llama 13b

[kalomaze]

https://www.reddit.com/r/LocalLLaMA/comments/1816h1x/how_much_does_quantization_actually_impact_models/

Mistral 7b average KL divergence:

Llama 13b average KL divergence:

[BarfingLemurs]

Does this work as a benchmark between models trained on the same data, like llama 7B and 13B?

[kalomaze]

Depends on whether or not you consider the base model of 13b objectively superior in every way, which is hard to quantify I'd assume. At its core, the graph is only measuring how different each quantization is from the base model on average