[Feature] Support for unsloth/DeepSeek-R1-0528-GGUF

[344303947]

Checklist

• 1. If the issue you raised is not a feature but a question, please raise a discussion at https://github.com/kvcache-ai/ktransformers/discussions. Otherwise, it will be closed.
• 2. To help the community, I will use Chinese/English or attach an Chinese/English translation if using another language. Non-English/Chinese content without translation may be closed.

Motivation

DeepSeek-R1-0528-UD-Q2_K_XL

Related resources

loading blk.0.attn_q_a_norm.weight to cuda
loading blk.0.attn_kv_a_norm.weight to cuda
Process SpawnProcess-1:
Traceback (most recent call last):
File "/home/sean/miniconda3/envs/kt/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
self.run()
File "/home/sean/miniconda3/envs/kt/lib/python3.10/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/home/sean/miniconda3/envs/kt/lib/python3.10/site-packages/ktransformers/server/backend/interfaces/balance_serve.py", line 244, in run_engine
engine = Engine(args, token_queue, broadcast_endpoint)
File "/home/sean/miniconda3/envs/kt/lib/python3.10/site-packages/ktransformers/server/backend/interfaces/balance_serve.py", line 162, in init
optimize_and_load_gguf(self.model, optimize_config_path, gguf_path, config)
File "/home/sean/miniconda3/envs/kt/lib/python3.10/site-packages/ktransformers/optimize/optimize.py", line 131, in optimize_and_load_gguf
load_weights(module, gguf_loader)
File "/home/sean/miniconda3/envs/kt/lib/python3.10/site-packages/ktransformers/util/utils.py", line 125, in load_weights
load_weights(child, gguf_loader, prefix+name+".")
File "/home/sean/miniconda3/envs/kt/lib/python3.10/site-packages/ktransformers/util/utils.py", line 127, in load_weights
module.load()
File "/home/sean/miniconda3/envs/kt/lib/python3.10/site-packages/ktransformers/operators/base_operator.py", line 63, in load
utils.load_weights(child, self.gguf_loader, self.key+".")
File "/home/sean/miniconda3/envs/kt/lib/python3.10/site-packages/ktransformers/util/utils.py", line 125, in load_weights
load_weights(child, gguf_loader, prefix+name+".")
File "/home/sean/miniconda3/envs/kt/lib/python3.10/site-packages/ktransformers/util/utils.py", line 125, in load_weights
load_weights(child, gguf_loader, prefix+name+".")
File "/home/sean/miniconda3/envs/kt/lib/python3.10/site-packages/ktransformers/util/utils.py", line 125, in load_weights
load_weights(child, gguf_loader, prefix+name+".")
File "/home/sean/miniconda3/envs/kt/lib/python3.10/site-packages/ktransformers/util/utils.py", line 127, in load_weights
module.load()
File "/home/sean/miniconda3/envs/kt/lib/python3.10/site-packages/ktransformers/operators/base_operator.py", line 63, in load
utils.load_weights(child, self.gguf_loader, self.key+".")
File "/home/sean/miniconda3/envs/kt/lib/python3.10/site-packages/ktransformers/util/utils.py", line 125, in load_weights
load_weights(child, gguf_loader, prefix+name+".")
File "/home/sean/miniconda3/envs/kt/lib/python3.10/site-packages/ktransformers/util/utils.py", line 123, in load_weights
load_cur_state_dict(module, gguf_loader, prefix)
File "/home/sean/miniconda3/envs/kt/lib/python3.10/site-packages/ktransformers/util/utils.py", line 118, in load_cur_state_dict
raise Exception(f"can't find {translated_key} in GGUF file!")
Exception: can't find blk.0.attn_kv_b.weight in GGUF file!

[apalesnail]

Same issue observed in Q4_K_M deepseek r1 0528

[zhouye]

The UD-Q2_K_XL version of DeepSeek-R1 0528 splits the KV tensor into standalone K and V tensors. The reason of doing this might be K and V needs to be quantized into different data types.

The support of this feature was in the upstream llama.cpp (ggml-org/llama.cpp#12801) but not included in ktransformer master.

[qiyuxinlin]

The issue has been resolved.