Borrow ideas from LLM in a flash

[bachittle]

I'm wondering if any of the techniques proposed in the following paper could be implemented here: https://huggingface.co/papers/2312.11514
https://arxiv.org/abs/2312.11514

This goes above my level of understanding, but I'm wondering if they give enough technical details to run this in a new example. The results were that they were able to load a model that was twice the size of available DRAM with major increases in inference speed. So figured I'd make a discussion post as an initial seed and see where this idea goes.

[rahulmanuwas]

Following! @bachittle if you find any implementations, please share!

[cmp-nct]

(Just my personal opinion) I don't think it contains anything of real use for now.
It reminds on PowerInfer just that it comes without code but uses a potentially better predictor model for sparse computation.
The "RAM" benefits come from only loading parts of a tensor.
Their predictor seems to use the "last 5 tokens" to get a quite accurate neuron activation pattern.
It will suffer from the same weakness, as in no gains during prompt batch processing.

Implementing it is impossible without code, given we already have all code for PowerInfer and even that is currently not on the table to really include .. that one is far from it.

[leemgs]

FYI. Here is a related topic discussed in the PowerInfer community.

• Combined with LLM in a flash SJTU-IPADS/PowerInfer#39