Adding support for DeepSeek-V2 MLA

[Azirine]

DeepSeek-V2-Chat-0628 currently uses excessive VRAM, possibly due to running as MHA instead of MLA.

Discussion here:
https://old.reddit.com/r/LocalLLaMA/comments/1e6ba6a/deepseekv2chat0628_weight_release_1_open_weight/ldtybpo/

[ggerganov]

I think we already support MLA. What makes you think that we use excessive VRAM?

[Azirine]

KV Cache sizes are extremely big, as seen here:
https://old.reddit.com/r/LocalLLaMA/comments/1e6ba6a/deepseekv2chat0628_weight_release_1_open_weight/ldv84z5/

I'm running Q3_K_S (101.7 GB) with M3 Max 128 GB memory and 122 GB allocated as VRAM, it swaps with small context size (<=2k, even with 256 for some reason). 4k/8k are unusable.

[ggerganov]

Ah yes, we are not storing the compressed data. This most likely has to wait after the KV cache functionality is refactored to allow overloading it more easily

[fairydreaming]

@Azirine check out this: #11446