Perplexity evaluator

[ghost]

Is there a working perplexity evaluator for exl2? The one from the exllama1 repo does not seem to work on exl2 quants.

[turboderp]

You can test perplexity against a Parquet file with:

test_inference.py -m <model_dir> -ed <dataset.parquet>

Note that it tokenizes the parquet file in the same way as the quantizer, by concatenating all the text and shaping it into a block of 128 rows of 2048 tokens each. You can override that with -er <num_rows> and/or -el <tokens_per_row>.

[ghost]

Just tried requantizing one using the regular measurements.json and -hb 8 -b 8 and it fixes the low context perplexity problem but still gets me A LOT (5.8) ppl when benchmarked. I don't know if I messed something up along the way in the quantizing or measuring but it shouldn't be so high.

[ghost]

Ok, no, I'm doing something seriously wrong because even the GPTQs get me way too much perplexity. Anyway, -hb 8 and -b 8 might fix the low context ppl problem but they don't get all the layers at 8.13 32g. Can you tell me the way to do it without the apparently broken 8.13 measurements.json from QMB15?

[ghost]

Managed to get the one in ooba's working and it finally gets me correct values. Still, the max version has more ppl than the -hb 8 -b 8 one so apparently the measurements are indeed broken. I'd appreciate the correct way to get all those layers at 8.13 32g.

[turboderp]

There are some subtleties to consider with perplexity and quantization, specifically with the error correction that GPTQ does. There's a risk of slightly overfitting to the calibration data, and this will be less pronounced in 8-bit than it would be in, say, 6-bit quantization, since the 8-bit matrices will have less error to correct for. As a result you might see lower perplexity from a 6-bit quant than from an 8-bit quant under some circumstances, especially if the test dataset aligns with the calibration dataset used (or if they're the same).

To produce a "max quality" quant you could simply specify -b 9, and it will select the "best possible" settings for each layer. The problem (if you consider it a problem) is that, for some layers, a mix of 8 and 6 bits is so accurate that the quantizer can't measure any difference compared to 8 bits 32g:

 -- 0.05:8b/0.05:6b/0.9:5b 32g s4  5.38 bpw    rfn_error: 0.00045
 -- 0.4:6b/0.6:5b 32g s4           5.50 bpw    rfn_error: 0.00047
 -- 0.1:8b/0.3:6b/0.6:5b 32g s4    5.81 bpw    rfn_error: 0.00043
 -- 1.0:6b 128g s4                 6.03 bpw    rfn_error: 0.00035
 -- 1.0:6b 32g s4                  6.13 bpw    rfn_error: 0.00046
 -- 0.1:8b/0.9:6b 128g s4          6.28 bpw    rfn_error: 0.00027
 -- 1.0:8b 32g s4                  8.13 bpw    rfn_error: 0.00040

All of the above results are effectively the same precision, with the different measurements resulting from "random jitter" in the FP16 math more than anything else.

If you want to force it to use 8-bit 32g for every layer there are a couple of ways to go about it, I guess. One would be to remove all other options from conversion/qparams.py, then running a new measurement. You can also play around with different mixes in there if you like, though not everything would be guaranteed to work.

Another option is to modify the measurement.json file so that the 8b 32g measurements are always preferred even if they're not measured to be the best option, by setting the "err" value to zero for every layer:

                {
                    "desc": "1.0:8b 32g s4",
                    "bpw": 8.125732421875,
                    "total_bits": 34081792.0,
                    "err": 0.0,      <---- that one
                    "qparams": {
                        "group_size": 32,
                        "bits": [
                            8
                        ],
                        "bits_prop": [
                            1.0
                        ],
                        "scale_bits": 4
                    }
                }

Of course it's a big file so maybe you'd want to script that somehow. Afterwards convert with -b 9.

Alternatively you could hack the do_quant() function in convert/quantize.py, which accepts the chosen quantization parameters as an argument. Replace qp = QParams.from_dict(qparams) with qp = QParams(32, [8], [1.0], 4) and it should always produce only 8b 32g output layers.

[ghost]

Thanks for the extensive summary. So do you think there could be any practical benefit to actually forcing it to 8bit 32g all of the layers? I assume it couldn't hurt, could it?