Could performant GPU decompression algorithms increase memory?

[BarfingLemurs]

• Inspired by a discussion from the localllama subreddit.

my question is not identical to the idea explored above, which uses high bandwidth storage

If we loaded a 180B f16 model to RAM, pre-compressed by a factor of 20 (18GB total), then sent pieces to the GPU for decompression (1GB->20gb) and ran inference, would we be able to run this massive model at a tolerable speed?

[ggerganov]

The model data is not compressible - factor of 20x is impossible.

You are better off just running it on the CPU. With mmap, it should dynamically load from disk whatever is needed

[BarfingLemurs]

The model data is not compressible - factor of 20x is impossible.

I see.. Maybe if there is a lossy way to reconstruct the model layer(s)? But that hurts the whole pursuit of "quality" with massive models.

[easp]

Model quantization is a type of lossy compression. There are various approaches to minimize quality loss by adjusting the degree of quantization for various parts of the model.

[askmyteapot]

pre-compressed by a factor of 20

So how do you propose the data is compressed? Standard compression algorithms won't do much (7z best compression can only achieve ~4% compression). We already have a kind of compression byway of quants.

The other issue, loading over the PCIe bus is SLOW. PCIe gen 4 x16 is about 30GB/s. That's slower than DDR4 3200 in dual channel mode. It's also not factoring in any overheads.

Pretty much all LLM speed limitations come from memory bandwidth, not INT/FP calculations.

[BarfingLemurs]

If the PCIe bus is slow, then if a decompression or some transformation with lowdata were able to let the gpu inference the next layer, it could overcome that.

I'm not saying have any knowledge such a thing exists yet. But to make use of the bandwidth of high end GPUs with their vram limits would be great, and it would achieve something similar to 192gb Mac studio

[cmp-nct]

When you look at compression, in my view, 7zip/lempel-ziv and similar algorithms are made for documents mostly.
llama.cpp is basically built upon compression, it's just called quantization when it comes to numbers, quantization compresses a number from 16 or 32 bit down to a few bit and uses the information of surrounding numbers to compress stronger.

It would certainly be possible to compress this stronger, you could group the tensor data by amplitude and tricks like that but the required decompression performance would make it worthless. After all you need the data in a defined structure to work with it.