2-bit integer quantization

[ggerganov]

Add Q2_0 and Q2_1 quantization support to ggml:

• Follow the existing Q4_0 and Q4_1 implementations
• Implement reference scalar quantization and dequantization routines
• I suspect we might have to use QK == 16 in this case to compensate for further accuracy losses
• Add SIMD support for a specific architecture - investigate best strategy to perform the ggml_vec_dot_q2() computation
• No need to implement ggml_vec_mad_q2() - these will be deprecated soon
• Compute perplexity scores

The expected model sizes for 7B and QK == 16 are:

• Q2_0 - 3.2 GB

For QK == 32 we have:

• Q2_0 - 2.4 GB
• Q2_1 - 3.2 GB

Before you send me papers that show 2-bit quantization does not work - no need. I want to have this supported anyway. I have something in mind. The efforts needed to add this support are so small that there is no reason not to do it.

[dakennedyd]

No 3-bit support?

[ggerganov]

No 3-bit support?

I don't think I can implement it efficiently, but if anyone wants to give it a try - sure

[Green-Sky]

65B using 32gig ram anyone? 😆

[prusnak]

I came up with a script that's able to compute RMS for various quantization methods - maybe it will come handy for experimenting: https://gist.github.com/prusnak/f54f8f33503458ca1aa9883f71897072

[sw]

Go home Q2, you're drunk ;-)

$ ./main -m ./models/7B/ggml-model-q2_0.bin -p "The efforts needed to add this support are so small that there is no reason not to do it." -n 64 -s 1679735763

The efforts needed to add this support are so small that there is no reason not to do it.
The efforts that we need the work to make sure that we can be sure that everything falls together with no additional and very little is reserved for a little or 1, or even less or 13 is 13, that in additionally or 1 month faster is 18 and or even faster

This is cherry-picked, often it goes to babbling numbers right away.

Q3 seems decent:

$ ./main -m ./models/7B/ggml-model-q3_0.bin -p "Building a website can be done in 10 simple steps:" -n 128 -s 1679739910

Building a website can be done in 10 simple steps:
Decide which web authoring software you're going to use.
Read up on what you need for the site you're building. Note that I am only referring to reading material on the web here; reading will build your knowledge without spending money on a book (or e-book). I would suggest looking into JavaScript, HTML5 and CSS3 before you launch into development of any kind. You can always test the waters of what you're working with against an online validator before you launch into production mode -- or you could just skip that part altogether until you get frustrated with having to use a browser

Both are very slow because I haven't found a good way to use AVX2 yet. Perplexity would probably take days if not weeks.

I used float for the scale in Q2 and FP16 in Q3, so the model files actually are the same size:

$ ls -gho models/7B/*q*
-rw-rw-r-- 1 3.2G Mär 25 10:43 models/7B/ggml-model-q2_0.bin
-rw-rw-r-- 1 3.2G Mär 25 10:45 models/7B/ggml-model-q3_0.bin
-rw-rw-r-- 1 4.0G Mär 24 11:52 models/7B/ggml-model-q4_0.bin
-rw-rw-r-- 1 4.8G Mär 22 13:08 models/7B/ggml-model-q4_1.bin

For Q2 I deviated slightly from the standard calculation of the factors. If you want to have a zero value and symmetry in positive and negative range, that would have left only 3 values (-1 0 +1). Instead, I calculate the signed maximum (= value of largest magnitude, without applying fabsf), then I assign the value -2 to that maximum. The sign of the shared scaling factor is adjusted to give the right sign of the result. Without this modification, I couldn't get Q2 to output any semblance of english.

Code here: https://github.com/sw/llama.cpp/tree/q2q3

[sw]

Updated my branch with AVX optimizations, probably far from perfect.

Still quite slow...
Q2:

98.37 seconds per pass - ETA 17.90 hours
[1]147.6625,[2]136.8862,[3]132.6015,[4]127.8629,[5]120.4091,[6]111.7640,[7]114.2548,[8]112.8951,

Q3:

203.61 seconds per pass - ETA 37.05 hours
[1]7.0481,[2]8.0335,[3]8.8317,[4]10.0700,[5]10.1138,[6]9.9850,[7]10.2314,[8]10.2057,

[CamiloMM]

Not nearly enough, we need support for 1-bit signed floats.

[Interpause]

Not nearly enough, we need support for 1-bit signed floats.

Swap that out for 1 qubit and now we're talking.

[prusnak]

Not nearly enough, we need support for 1-bit signed floats.

I think the best model size and performance will be achieved when 0-bit quantization is used.

[Lolagatorade]

Not nearly enough, we need support for 1-bit signed floats.

I think the best model size and performance will be achieved when 0-bit quantization is used.

Mhmm possibly -1...