Is there a way to lower VRAM usage without lowering context size?

[curvedinf]

I would like to use a very large context (100k+ tokens) with my models, but can not for even small models because of VRAM limitations. Is it possible to configure llama.cpp so it has lower VRAM cost per token of context size?

[zhentaoyu]

You can try -nkvo to store the kv cache on CPU and use -fa flash-attn to save CPU memory, especially for the first token. I did it on intel GPUs before and tried to store transposed kv cache and it could accelerate the next-token generation speed a little.

[curvedinf]

Thank you! Using phi3 mini at 12000 ctx is 13 GB with default options. It goes down to 9GB with flash attention and 4GB with offload kvcache off. Kvcache offloading speeds up inference a lot, so I will keep that on. FA has no downside it seems.