Context reuse / context shift for long prompts

[SamuelOliveirads]

Hi! — I'm coming from koboldcpp, and I've been testing this fork due to its optimizations.

One feature I found very useful in koboldcpp was the context shift functionality, which helps when working with very long context windows.

I noticed that llama.cpp implemented something similar in PR #9866, which allows for reusing the prompt cache more efficiently instead of regenerating the entire prompt every time the context overflows.

I searched through this repo but couldn’t find an equivalent implementation.

Here’s the issue I’m currently facing:

• I'm using a 62k context in Qwen 3.
• When the context overflows, the cache keeps my system prompt, but discards the conversation history.
• That leads to reprocessing ~58k tokens from scratch each time, which at ~40 tokens/sec takes several minutes per new message.
• With proper cache reuse (like in llama.cpp), this would take just seconds.

My question is:

• Is there already something similar to context reuse implemented here?
• If not, would this be something feasible to implement, perhaps inspired by how llama.cpp did it?

Thanks!

[mtcl]

This is a very useful usecase because of which I have been switching back and forth between ik_llama.cpp and llama.cpp. This works seamlessly with llama.cpp i have noticed. I always thought I am doing something wrong here and it is my user error, but apparantly it is not! Thank you for mentioning it here.

[cmoncure]

This would be a massive win for me. Currently PP is the millstone around the neck (for which you have had to endure many of my ignorant comments in support of a solution).

KV Cache reuse and tool calling would open up whole new worlds.

[mtcl]

I agree 100% with you. Given that I built my own tool calling solution for ik_llama.cpp, at this point of time kv cache reuse would mean an instant switch for me to this!

[SamuelOliveirads]

Glad to see that others are also interested in this feature! I was about to open an issue myself, but I noticed that @saood06 is already looking into something similar here — so now it’s just a matter of waiting.

By the way, @saood06, if you need any help with testing, I’d be happy to assist.

[ikawrakow]

Are we talking about the llama.cpp feature (taken from kobold.cpp) where if I have

aaaaccccbbbb

in the KV cache, and the new context is

aaaabbbb

I can reuse the full aaaabbbb (mainline llama.cpp) instead of just reusing aaaa as it happens here?

If so, here is an example:

KV cache: Yesterday I saw a movie. I absolutely enjoyed it. The main actor was ...
New context: Yesterday I saw a movie. The main actor was

Suppose New context is in the context of the worst movie you have ever seen, so you expect "a disaster" or some such.
The existing KV cache, despite context shifting and all that, will be heavily biased towards "brilliant", "amazing" and such.

Do you see the problem? You cannot undo the impact of the skipped tokens by just changing the position encoding via RoPE.

[saood06]

Are we talking about the llama.cpp feature (taken from kobold.cpp) where if I have

Yes that is what we are talking about. Thank you for the very clear example (so much better than what I was typing out).

I'm not sure this is from kobold.cpp. I know they offer a much better context shift where they effectively keep the context full at all times once you hit the limit unlike llama.cpp and here where the context shift unnecessarily removes far more tokens than is needed (I think half) and thus shifts are less frequent. Kobold.cpp on the other hand shifts every token which keeps the maximum information allowed at all times.

[cmoncure]

You cannot undo the impact of the skipped tokens by just changing the position encoding via RoPE.

So...

1. KV Cache is a Key-Value cache
2. KV Cache as a "memoization" technique stores the results of the expensive PP computation for reuse.
3. But the PP computation is cumulative in such a way that the presence and order of tokens matters.
4. Once a token has acted on the KV cache, its effect poisons the KV cache indelibly.

Questions:

1. Is the effect of tokens on the KV cache additive or multiplicative (or something else)? If additive, can the effect of tokens removed from the prompt be recalculated and their effect subtracted?
2. If the presence of token PP computation in the KV cache poisons it forever, then doesn't that imply that tokens outside the context window can continue to affect generation? That would contradict my mental model of how all this is supposed to work. Edit: I suppose that's why the whole thing must be scrapped each time when the context window fills up. It makes sense.

[saood06]

4. Once a token has acted on the KV cache, its effect poisons the KV cache indelibly.

Questions:

2. If the presence of token PP computation in the KV cache poisons it forever, then doesn't that imply that tokens outside the context window can continue to affect generation? That would contradict my mental model of how all this is supposed to work. Edit: I suppose that's why the whole thing must be scrapped each time when the context window fills up. It makes sense.

No. If that were the case then you could not have multiple slots which serve independent users that share the KV cache, but that is a well supported use case.

The tokens do not "poison" the cache, it is just that a token holds the information of all prior tokens from that sequence when it was calculated. If you get rid of tokens and then shift tokens that had come after the now deleted tokens in order to re-use them the shifted tokens will still contain the information from the deleted tokens.

To add to the the example given above with the movie, even though you removed the tokens "I absolutely enjoyed it.", their influence is not gone if you keep the tokens after and shift them.

If you shift "The main actor was" then you will see the influence of the removed tokens (but it will be much faster as you are not recomputing those tokens).

If you do recompute the tokens "The main actor was" and do not shift then it will be slower (as you have to actually compute the tokens again) but you will not experience the lingering impact of "I absolutely enjoyed it."

[cmoncure]

If you do recompute the tokens "The main actor was" and do not shift then it will be slower (as you have to actually compute the tokens again) but you will not experience the lingering impact of "I absolutely enjoyed it."

Forgive me if I've misunderstood. Suppose we have the following prompt:

AAAABBBBCCCC

Then we can understand the state of the fully processed KV cache to be something like the following, where some function f(X) :-> x gives the "effect" of the token on subsequent tokens:

A A A A Ba Ba Ba Ba Cab Cab Cab Cab

I'm stretching the truth a bit here for the purposes of a convenient representation. But the above illustrates that each part of the prompt carries with it information about the previous parts.

Suppose that our context grows and our A tokens must be pushed off the top of the context window. Then we have some intermediate state

Ba Ba Ba Ba Cab Cab Cab Cab D D D D

In order to create a properly functioning KV cache, we have to effectuate the following:

1. The effect of A tokens must be removed from B and C
2. D tokens must take into account B and C

So that finally, we have

B B B B Cb Cb Cb Cb Dbc Dbc Dbc Dbc

The way this is currently achieved is (if I am not mistaken) by dropping and re-processing the entire cache pertaining to the prompt, which is expensive, suggesting an algorithmic complexity of O(n^2). Can we not instead of re-processing the entire prompt, simply calculate f(A) and subtract it from the following tokens (or the inverse f'(A) and add it):

Ba Ba Ba Ba Cab Cab Cab Cab - f(A) => B B B B Cb Cb Cb Cb

Finally computing the rest of the prompt only against D:

D D D D + f(B) + F(C) => Dbc Dbc Dbc Dbc

Then concatenate the two to get the desired state? I'm still reading through llama.cpp... it's a lot.

[cmoncure]

Might have to do it myself.