Flash decoding (flash attention) speed increase for long context

[jjohare]

I just came across this and can't see the conversation elsewhere so...
From the blog post

"We present a technique, Flash-Decoding, that significantly speeds up attention during inference, bringing up to 8x faster generation for very long sequences. The main idea is to load the keys and values in parallel as fast as possible, then separately rescale and combine the results to maintain the right attention outputs."

Example forthcoming:
https://github.com/Dao-AILab/flash-attention/tree/main/examples/inference

[psugihara]

Looks like phind is using it to speed up inference https://news.ycombinator.com/item?id=38089451