PowerInfer: Built on llama.cpp, Now 11x Faster

[bobqianic]

I think the main breakthrough is that it can arrange the position of weight parameters more scientifically based on the frequency of neuron activation, placing the frequently activated weights in faster-reading caches to improve inference speed. They developed a Neuron-aware Operator that can bypass neurons that are not activated, and also developed an offline profiling technique that allows the Neuron Placement Policy to be predetermined.

Github: https://github.com/SJTU-IPADS/PowerInfer

Paper: https://ipads.se.sjtu.edu.cn/_media/publications/powerinfer-20231219.pdf

Relevant discussions: #4543 #4542 #4534

[joseph777111]

@ggerganov,what are your thoughts on this? Could this be useful for llama.cpp? 🤔

[bobqianic]

See #4534 (reply in thread)

[joseph777111]

Thanks 🤘😁