Efficient Streaming Language Models with Attention Sinks [

[logikstate]

New paper with example code claims huge context with minimal changes

https://github.com/mit-han-lab/streaming-llm

[KerfuffleV2]

Hmm, either you misunderstood it or I did. https://github.com/mit-han-lab/streaming-llm#faq says that it doesn't actually increase context. As I understand it, this is basically #3377 - a more graceful way to do --keep rather than something that actually increases the context size.

[logikstate]

Yes, the author updated and added an faq. It seems similar to sliding window.

[logikstate]

https://news.ycombinator.com/item?id=37740932

[j7z]

For reference, Tom Aarsen also has an attention sinks project here: https://github.com/tomaarsen/attention_sinks