Replies: 30 comments 161 replies
-
|
I trolled through some of the PRs you linked to me and pulled together this rough guide as my notes for getting started with The biggest hurdle so far is needing a custom quant for MLA support. I'll work on that another time as I'm using og unsloth My initial impression is with the right settings it can get faster prompt processing than ktransformers and about the same token generation. Looking forward to trying it with an MLA supported quant. |
Beta Was this translation helpful? Give feedback.
-
Glad I can be of help. I've seen a lot of people show interest in using ik_llama.cpp but the amount of options and the spread out documentation was a deterrent. This guide (even in it's current state) is a much better resource to give people than my explanations and links to PR's, so thank you for putting it together.
You seemed to have found all the huggingface MLA quants I know of but I forgot to mention that you can use the technique listed here in order to skip a step if you are going to manually convert from the original fp8 model files. (I've thought about porting that here but the triton dependence adds more complication than I think it is worth for most people, when more fp8 native models are released, I think something along the lines of this is the best path forward). I think reading through this discussion #242 (most relevant bits are this, this, and this but there are other bits of the discussion that are worth reading if you are making your own imatrix as you may run into similar issues, but as mentioned you can just use an imatrix from someone else, just make sure to set the new MLA tensors to high quant types as those won't be in any imatrix unless they created it with MLA. Making a custom quant has a lot of flexibility in terms of quality, size, and performance (for example the quant of the attention tensors and shared experts has much lower impact on size, but has larger impacts on quality and size, whereas the quant of the non-shared experts has a much larger impact on size, and a smaller impact on performance). This is demonstrated here where the custom blend that is smaller had lower PPL than the IQ4_KSS quant. There is a lot more discussion about quants in that thread (and it is where the issue of CUDA for certain tensors was first noticed).
I think ktransformers will outperform ik_llama.cpp without MLA for TG at higher context lengths as it uses MLA. The higher PP is nice, I wonder if the lead is still held with MLA. Also you may find #225 useful for benchmarking. |
Beta Was this translation helpful? Give feedback.
-
|
@saood06 please keep in mind that there is no such thing as comparing the performance of ik_llama.cpp with ktransformers. Simply because the ktransformers is using old fork of flashinfer (see 0.2.3). If simply put, you will get either crash in the sampler or the garbage output (or lost context). Yeah, I initially thought ik_llama.cpp suck because the decode speed is slower (esp. on a long context because they dont't use matrix absorption etrc.) .. but ... there is simply no way to run ktransformers with large context. ktransformers doesn't even have the --seed parameter implemented lol so each time the llm answers you you can't tell if its a right answer or its a garbage lol. ktransformers was written by script-kiddies (I looked at the code -- its awful). So please be serious. |
Beta Was this translation helpful? Give feedback.
-
Not sure why you are replying to old comments. I said in a later comment in this same discussion page, "Even then and still now I still see ktransformers as more of a performance demo because of how limited it is in what it supports both in hardware and the server/API they expose." |
Beta Was this translation helpful? Give feedback.
-
Well, you didn't say that ktransformers ARE unusable. I am saying that. Its not about the stuff that it supports or not. The problem is that they claiming to support 128k context when in reality it just crashes or outputs the garbage. So anyone reading this thread should be aware to not waste any time with ktransformers. That's it. |
Beta Was this translation helpful? Give feedback.
-
|
Thank you for these results.
#259 should remove this hurdle. With this PR models prepared with mainline |
Beta Was this translation helpful? Give feedback.
-
Just thought you'd want to know this, manually notifying you as edit's don't trigger notifications. |
Beta Was this translation helpful? Give feedback.
-
Yeah, looking more I see the full run is more useful for easy comparisons than just the first N chunks.
I see. Are you aware of other quants that throw It runs the There were no nans running it with vanilla See here for exact logs. Let me know if you think I should open an issue or of maybe just user error? `ik_llama.cpp llama-perplexity` logs.$ numactl -N 0 -m 0 \
./build/bin/llama-perplexity \
--model /mnt/ai/models/unsloth/DeepSeek-R1-GGUF/DeepSeek-R1-Q8_0/DeepSeek-R1.Q8_0-00001-of-00015.gguf \
-rtr \
-ctk q8_0 \
-mla 2 -fa \
-amb 512 \
-fmoe \
--ctx-size 512 \
--ubatch-size 512 \
-f wiki.test.raw \
--numa numactl \
--threads 128
# also similar results on `ik_llama.cpp@f2fb15de` without fancy options etc.
main: build = 3597 (127c6ee6) 20:14:51 [199/1921]
main: built with cc (Ubuntu 13.3.0-6ubuntu2~24.04) 13.3.0 for x86_64-linux-gnu
main: seed = 1742415291
WARNING: /proc/sys/kernel/numa_balancing is enabled, this has been observed to impair performance
llama_model_loader: additional 14 GGUFs metadata loaded.
llama_model_loader: loaded meta data with 48 key-value pairs and 1025 tensors from /mnt/ai/models/unsloth/DeepSeek-R1-GGUF/DeepSeek-R1-Q8_0/DeepSeek-R
1.Q8_0-00001-of-00015.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv 0: general.architecture str = deepseek2
llama_model_loader: - kv 1: general.type str = model
llama_model_loader: - kv 2: general.name str = DeepSeek R1 BF16
llama_model_loader: - kv 3: general.quantized_by str = Unsloth
llama_model_loader: - kv 4: general.size_label str = 256x20B
llama_model_loader: - kv 5: general.repo_url str = https://huggingface.co/unsloth
llama_model_loader: - kv 6: deepseek2.block_count u32 = 61
llama_model_loader: - kv 7: deepseek2.context_length u32 = 163840
llama_model_loader: - kv 8: deepseek2.embedding_length u32 = 7168
llama_model_loader: - kv 9: deepseek2.feed_forward_length u32 = 18432
llama_model_loader: - kv 10: deepseek2.attention.head_count u32 = 128
llama_model_loader: - kv 11: deepseek2.attention.head_count_kv u32 = 128
llama_model_loader: - kv 12: deepseek2.rope.freq_base f32 = 10000.000000
llama_model_loader: - kv 13: deepseek2.attention.layer_norm_rms_epsilon f32 = 0.000001
llama_model_loader: - kv 14: deepseek2.expert_used_count u32 = 8
llama_model_loader: - kv 15: general.file_type u32 = 7
llama_model_loader: - kv 16: deepseek2.leading_dense_block_count u32 = 3
llama_model_loader: - kv 17: deepseek2.vocab_size u32 = 129280
llama_model_loader: - kv 18: deepseek2.attention.q_lora_rank u32 = 1536
llama_model_loader: - kv 19: deepseek2.attention.kv_lora_rank u32 = 512
llama_model_loader: - kv 20: deepseek2.attention.key_length u32 = 192
llama_model_loader: - kv 21: deepseek2.attention.value_length u32 = 128
llama_model_loader: - kv 22: deepseek2.expert_feed_forward_length u32 = 2048
llama_model_loader: - kv 23: deepseek2.expert_count u32 = 256
llama_model_loader: - kv 24: deepseek2.expert_shared_count u32 = 1
llama_model_loader: - kv 25: deepseek2.expert_weights_scale f32 = 2.500000
llama_model_loader: - kv 26: deepseek2.expert_weights_norm bool = true
llama_model_loader: - kv 27: deepseek2.expert_gating_func u32 = 2
llama_model_loader: - kv 28: deepseek2.rope.dimension_count u32 = 64
llama_model_loader: - kv 29: deepseek2.rope.scaling.type str = yarn
llama_model_loader: - kv 30: deepseek2.rope.scaling.factor f32 = 40.000000
llama_model_loader: - kv 31: deepseek2.rope.scaling.original_context_length u32 = 4096
llama_model_loader: - kv 32: deepseek2.rope.scaling.yarn_log_multiplier f32 = 0.100000
llama_model_loader: - kv 33: tokenizer.ggml.model str = gpt2
llama_model_loader: - kv 34: tokenizer.ggml.pre str = deepseek-v3
llama_model_loader: - kv 35: tokenizer.ggml.tokens arr[str,129280] = ["<｜begin▁of▁sentence｜>", "<...
llama_model_loader: - kv 36: tokenizer.ggml.token_type arr[i32,129280] = [3, 3, 3, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
llama_model_loader: - kv 37: tokenizer.ggml.merges arr[str,127741] = ["Ġ t", "Ġ a", "i n", "Ġ Ġ", "h e...
llama_model_loader: - kv 38: tokenizer.ggml.bos_token_id u32 = 0
llama_model_loader: - kv 39: tokenizer.ggml.eos_token_id u32 = 1
llama_model_loader: - kv 40: tokenizer.ggml.padding_token_id u32 = 128815
llama_model_loader: - kv 41: tokenizer.ggml.add_bos_token bool = true
llama_model_loader: - kv 42: tokenizer.ggml.add_eos_token bool = false
llama_model_loader: - kv 43: tokenizer.chat_template str = {% if not add_generation_prompt is de...
llama_model_loader: - kv 44: general.quantization_version u32 = 2
llama_model_loader: - kv 45: split.no u16 = 0
llama_model_loader: - kv 46: split.count u16 = 15
llama_model_loader: - kv 47: split.tensors.count i32 = 1025
llama_model_loader: - type f32: 361 tensors
llama_model_loader: - type q8_0: 664 tensors
llm_load_vocab: special tokens cache size = 819
llm_load_vocab: token to piece cache size = 0.8223 MB
llm_load_print_meta: format = GGUF V3 (latest)
llm_load_print_meta: arch = deepseek2
llm_load_print_meta: vocab type = BPE
llm_load_print_meta: n_vocab = 129280
llm_load_print_meta: n_merges = 127741
llm_load_print_meta: vocab_only = 0
llm_load_print_meta: n_ctx_train = 163840
llm_load_print_meta: n_embd = 7168
llm_load_print_meta: n_layer = 61
llm_load_print_meta: n_head = 128
llm_load_print_meta: n_head_kv = 128
llm_load_print_meta: n_rot = 64
llm_load_print_meta: n_swa = 0
llm_load_print_meta: n_embd_head_k = 192
llm_load_print_meta: n_embd_head_v = 128
llm_load_print_meta: n_gqa = 1
llm_load_print_meta: n_embd_k_gqa = 24576
llm_load_print_meta: n_embd_v_gqa = 16384
llm_load_print_meta: f_norm_eps = 0.0e+00
llm_load_print_meta: f_norm_rms_eps = 1.0e-06
llm_load_print_meta: f_clamp_kqv = 0.0e+00
llm_load_print_meta: f_max_alibi_bias = 0.0e+00
llm_load_print_meta: f_logit_scale = 0.0e+00
llm_load_print_meta: n_ff = 18432
llm_load_print_meta: n_expert = 256
llm_load_print_meta: n_expert_used = 8
llm_load_print_meta: causal attn = 1
llm_load_print_meta: pooling type = 0
llm_load_print_meta: rope type = 0
llm_load_print_meta: rope scaling = yarn
llm_load_print_meta: freq_base_train = 10000.0
llm_load_print_meta: freq_scale_train = 0.025
llm_load_print_meta: n_ctx_orig_yarn = 4096
llm_load_print_meta: rope_finetuned = unknown
llm_load_print_meta: ssm_d_conv = 0
llm_load_print_meta: ssm_d_inner = 0
llm_load_print_meta: ssm_d_state = 0
llm_load_print_meta: ssm_dt_rank = 0
llm_load_print_meta: model type = 671B
llm_load_print_meta: model ftype = Q8_0
llm_load_print_meta: model params = 671.026 B
llm_load_print_meta: model size = 664.295 GiB (8.504 BPW)
llm_load_print_meta: repeating layers = 662.461 GiB (8.504 BPW, 669.173 B parameters)
llm_load_print_meta: general.name = DeepSeek R1 BF16
llm_load_print_meta: BOS token = 0 '<｜begin▁of▁sentence｜>'
llm_load_print_meta: EOS token = 1 '<｜end▁of▁sentence｜>'
llm_load_print_meta: PAD token = 128815 '<｜PAD▁TOKEN｜>'
llm_load_print_meta: LF token = 131 'Ä'
llm_load_print_meta: max token length = 256
llm_load_print_meta: n_layer_dense_lead = 3
llm_load_print_meta: n_lora_q = 1536
llm_load_print_meta: n_lora_kv = 512
llm_load_print_meta: n_ff_exp = 2048
llm_load_print_meta: n_expert_shared = 1
llm_load_print_meta: expert_weights_scale = 2.5
llm_load_print_meta: expert_weights_norm = 1
llm_load_print_meta: expert_gating_func = sigmoid
llm_load_print_meta: rope_yarn_log_mul = 0.1000
llm_load_tensors: ggml ctx size = 0.42 MiB
llm_load_tensors: CPU buffer size = 680237.97 MiB
....................................................................................................
============ llm_load_tensors: need to compute 61 wk_b tensors
Computed blk.0.attn_v_b.weight as 128 x 512 x 128 and stored in buffer CPU
Computed blk.1.attn_v_b.weight as 128 x 512 x 128 and stored in buffer CPU
Computed blk.2.attn_v_b.weight as 128 x 512 x 128 and stored in buffer CPU
Computed blk.3.attn_v_b.weight as 128 x 512 x 128 and stored in buffer CPU
Computed blk.4.attn_v_b.weight as 128 x 512 x 128 and stored in buffer CPU
Computed blk.5.attn_v_b.weight as 128 x 512 x 128 and stored in buffer CPU
Computed blk.6.attn_v_b.weight as 128 x 512 x 128 and stored in buffer CPU
Computed blk.7.attn_v_b.weight as 128 x 512 x 128 and stored in buffer CPU
Computed blk.8.attn_v_b.weight as 128 x 512 x 128 and stored in buffer CPU
Computed blk.9.attn_v_b.weight as 128 x 512 x 128 and stored in buffer CPU
Computed blk.10.attn_v_b.weight as 128 x 512 x 128 and stored in buffer CPU
Computed blk.11.attn_v_b.weight as 128 x 512 x 128 and stored in buffer CPU
Computed blk.12.attn_v_b.weight as 128 x 512 x 128 and stored in buffer CPU
Computed blk.13.attn_v_b.weight as 128 x 512 x 128 and stored in buffer CPU
Computed blk.14.attn_v_b.weight as 128 x 512 x 128 and stored in buffer CPU
Computed blk.15.attn_v_b.weight as 128 x 512 x 128 and stored in buffer CPU
Computed blk.16.attn_v_b.weight as 128 x 512 x 128 and stored in buffer CPU
Computed blk.17.attn_v_b.weight as 128 x 512 x 128 and stored in buffer CPU
Computed blk.18.attn_v_b.weight as 128 x 512 x 128 and stored in buffer CPU
Computed blk.19.attn_v_b.weight as 128 x 512 x 128 and stored in buffer CPU
Computed blk.20.attn_v_b.weight as 128 x 512 x 128 and stored in buffer CPU
Computed blk.21.attn_v_b.weight as 128 x 512 x 128 and stored in buffer CPU
Computed blk.22.attn_v_b.weight as 128 x 512 x 128 and stored in buffer CPU
Computed blk.23.attn_v_b.weight as 128 x 512 x 128 and stored in buffer CPU
Computed blk.24.attn_v_b.weight as 128 x 512 x 128 and stored in buffer CPU
Computed blk.25.attn_v_b.weight as 128 x 512 x 128 and stored in buffer CPU
Computed blk.26.attn_v_b.weight as 128 x 512 x 128 and stored in buffer CPU
Computed blk.27.attn_v_b.weight as 128 x 512 x 128 and stored in buffer CPU
Computed blk.28.attn_v_b.weight as 128 x 512 x 128 and stored in buffer CPU
Computed blk.29.attn_v_b.weight as 128 x 512 x 128 and stored in buffer CPU
Computed blk.30.attn_v_b.weight as 128 x 512 x 128 and stored in buffer CPU
Computed blk.31.attn_v_b.weight as 128 x 512 x 128 and stored in buffer CPU
Computed blk.32.attn_v_b.weight as 128 x 512 x 128 and stored in buffer CPU
Computed blk.33.attn_v_b.weight as 128 x 512 x 128 and stored in buffer CPU
Computed blk.34.attn_v_b.weight as 128 x 512 x 128 and stored in buffer CPU
Computed blk.35.attn_v_b.weight as 128 x 512 x 128 and stored in buffer CPU
Computed blk.36.attn_v_b.weight as 128 x 512 x 128 and stored in buffer CPU
Computed blk.37.attn_v_b.weight as 128 x 512 x 128 and stored in buffer CPU
Computed blk.38.attn_v_b.weight as 128 x 512 x 128 and stored in buffer CPU
Computed blk.39.attn_v_b.weight as 128 x 512 x 128 and stored in buffer CPU
Computed blk.40.attn_v_b.weight as 128 x 512 x 128 and stored in buffer CPU
Computed blk.41.attn_v_b.weight as 128 x 512 x 128 and stored in buffer CPU
Computed blk.42.attn_v_b.weight as 128 x 512 x 128 and stored in buffer CPU
Computed blk.43.attn_v_b.weight as 128 x 512 x 128 and stored in buffer CPU
Computed blk.44.attn_v_b.weight as 128 x 512 x 128 and stored in buffer CPU
Computed blk.45.attn_v_b.weight as 128 x 512 x 128 and stored in buffer CPU
Computed blk.46.attn_v_b.weight as 128 x 512 x 128 and stored in buffer CPU
Computed blk.47.attn_v_b.weight as 128 x 512 x 128 and stored in buffer CPU
Computed blk.48.attn_v_b.weight as 128 x 512 x 128 and stored in buffer CPU
Computed blk.49.attn_v_b.weight as 128 x 512 x 128 and stored in buffer CPU
Computed blk.50.attn_v_b.weight as 128 x 512 x 128 and stored in buffer CPU
Computed blk.51.attn_v_b.weight as 128 x 512 x 128 and stored in buffer CPU
Computed blk.52.attn_v_b.weight as 128 x 512 x 128 and stored in buffer CPU
Computed blk.53.attn_v_b.weight as 128 x 512 x 128 and stored in buffer CPU
Computed blk.54.attn_v_b.weight as 128 x 512 x 128 and stored in buffer CPU
Computed blk.55.attn_v_b.weight as 128 x 512 x 128 and stored in buffer CPU
Computed blk.56.attn_v_b.weight as 128 x 512 x 128 and stored in buffer CPU
Computed blk.57.attn_v_b.weight as 128 x 512 x 128 and stored in buffer CPU
Computed blk.58.attn_v_b.weight as 128 x 512 x 128 and stored in buffer CPU
Computed blk.59.attn_v_b.weight as 128 x 512 x 128 and stored in buffer CPU
Computed blk.60.attn_v_b.weight as 128 x 512 x 128 and stored in buffer CPU
============ Repacked 663 tensors
llama_new_context_with_model: n_ctx = 2048
llama_new_context_with_model: n_batch = 2048
llama_new_context_with_model: n_ubatch = 512
llama_new_context_with_model: flash_attn = 1
llama_new_context_with_model: mla_attn = 2
llama_new_context_with_model: attn_max_b = 512
llama_new_context_with_model: fused_moe = 1
llama_new_context_with_model: ser = -1, 0
llama_new_context_with_model: freq_base = 10000.0
llama_new_context_with_model: freq_scale = 0.025
llama_kv_cache_init: layer 0: n_embd_head_qk_rope = 64, kv_lora_rank = 512
llama_kv_cache_init: layer 1: n_embd_head_qk_rope = 64, kv_lora_rank = 512
llama_kv_cache_init: layer 2: n_embd_head_qk_rope = 64, kv_lora_rank = 512
llama_kv_cache_init: layer 3: n_embd_head_qk_rope = 64, kv_lora_rank = 512
llama_kv_cache_init: layer 4: n_embd_head_qk_rope = 64, kv_lora_rank = 512
llama_kv_cache_init: layer 5: n_embd_head_qk_rope = 64, kv_lora_rank = 512
llama_kv_cache_init: layer 6: n_embd_head_qk_rope = 64, kv_lora_rank = 512
llama_kv_cache_init: layer 7: n_embd_head_qk_rope = 64, kv_lora_rank = 512
llama_kv_cache_init: layer 8: n_embd_head_qk_rope = 64, kv_lora_rank = 512
llama_kv_cache_init: layer 9: n_embd_head_qk_rope = 64, kv_lora_rank = 512
llama_kv_cache_init: layer 10: n_embd_head_qk_rope = 64, kv_lora_rank = 512
llama_kv_cache_init: layer 11: n_embd_head_qk_rope = 64, kv_lora_rank = 512
llama_kv_cache_init: layer 12: n_embd_head_qk_rope = 64, kv_lora_rank = 512
llama_kv_cache_init: layer 13: n_embd_head_qk_rope = 64, kv_lora_rank = 512
llama_kv_cache_init: layer 14: n_embd_head_qk_rope = 64, kv_lora_rank = 512
llama_kv_cache_init: layer 15: n_embd_head_qk_rope = 64, kv_lora_rank = 512
llama_kv_cache_init: layer 16: n_embd_head_qk_rope = 64, kv_lora_rank = 512
llama_kv_cache_init: layer 17: n_embd_head_qk_rope = 64, kv_lora_rank = 512
llama_kv_cache_init: layer 18: n_embd_head_qk_rope = 64, kv_lora_rank = 512
llama_kv_cache_init: layer 19: n_embd_head_qk_rope = 64, kv_lora_rank = 512
llama_kv_cache_init: layer 20: n_embd_head_qk_rope = 64, kv_lora_rank = 512
llama_kv_cache_init: layer 21: n_embd_head_qk_rope = 64, kv_lora_rank = 512
llama_kv_cache_init: layer 22: n_embd_head_qk_rope = 64, kv_lora_rank = 512
llama_kv_cache_init: layer 23: n_embd_head_qk_rope = 64, kv_lora_rank = 512
llama_kv_cache_init: layer 24: n_embd_head_qk_rope = 64, kv_lora_rank = 512
llama_kv_cache_init: layer 25: n_embd_head_qk_rope = 64, kv_lora_rank = 512
llama_kv_cache_init: layer 26: n_embd_head_qk_rope = 64, kv_lora_rank = 512
llama_kv_cache_init: layer 27: n_embd_head_qk_rope = 64, kv_lora_rank = 512
llama_kv_cache_init: layer 28: n_embd_head_qk_rope = 64, kv_lora_rank = 512
llama_kv_cache_init: layer 29: n_embd_head_qk_rope = 64, kv_lora_rank = 512
llama_kv_cache_init: layer 30: n_embd_head_qk_rope = 64, kv_lora_rank = 512
llama_kv_cache_init: layer 31: n_embd_head_qk_rope = 64, kv_lora_rank = 512
llama_kv_cache_init: layer 32: n_embd_head_qk_rope = 64, kv_lora_rank = 512
llama_kv_cache_init: layer 33: n_embd_head_qk_rope = 64, kv_lora_rank = 512
llama_kv_cache_init: layer 34: n_embd_head_qk_rope = 64, kv_lora_rank = 512
llama_kv_cache_init: layer 35: n_embd_head_qk_rope = 64, kv_lora_rank = 512
llama_kv_cache_init: layer 36: n_embd_head_qk_rope = 64, kv_lora_rank = 512
llama_kv_cache_init: layer 37: n_embd_head_qk_rope = 64, kv_lora_rank = 512
llama_kv_cache_init: layer 38: n_embd_head_qk_rope = 64, kv_lora_rank = 512
llama_kv_cache_init: layer 39: n_embd_head_qk_rope = 64, kv_lora_rank = 512
llama_kv_cache_init: layer 40: n_embd_head_qk_rope = 64, kv_lora_rank = 512
llama_kv_cache_init: layer 41: n_embd_head_qk_rope = 64, kv_lora_rank = 512
llama_kv_cache_init: layer 42: n_embd_head_qk_rope = 64, kv_lora_rank = 512
llama_kv_cache_init: layer 43: n_embd_head_qk_rope = 64, kv_lora_rank = 512
llama_kv_cache_init: layer 44: n_embd_head_qk_rope = 64, kv_lora_rank = 512
llama_kv_cache_init: layer 45: n_embd_head_qk_rope = 64, kv_lora_rank = 512
llama_kv_cache_init: layer 46: n_embd_head_qk_rope = 64, kv_lora_rank = 512
llama_kv_cache_init: layer 47: n_embd_head_qk_rope = 64, kv_lora_rank = 512
llama_kv_cache_init: layer 48: n_embd_head_qk_rope = 64, kv_lora_rank = 512
llama_kv_cache_init: layer 49: n_embd_head_qk_rope = 64, kv_lora_rank = 512
llama_kv_cache_init: layer 50: n_embd_head_qk_rope = 64, kv_lora_rank = 512
llama_kv_cache_init: layer 51: n_embd_head_qk_rope = 64, kv_lora_rank = 512
llama_kv_cache_init: layer 52: n_embd_head_qk_rope = 64, kv_lora_rank = 512
llama_kv_cache_init: layer 53: n_embd_head_qk_rope = 64, kv_lora_rank = 512
llama_kv_cache_init: layer 54: n_embd_head_qk_rope = 64, kv_lora_rank = 512
llama_kv_cache_init: layer 55: n_embd_head_qk_rope = 64, kv_lora_rank = 512
llama_kv_cache_init: layer 56: n_embd_head_qk_rope = 64, kv_lora_rank = 512
llama_kv_cache_init: layer 57: n_embd_head_qk_rope = 64, kv_lora_rank = 512
llama_kv_cache_init: layer 58: n_embd_head_qk_rope = 64, kv_lora_rank = 512
llama_kv_cache_init: layer 59: n_embd_head_qk_rope = 64, kv_lora_rank = 512
llama_kv_cache_init: layer 60: n_embd_head_qk_rope = 64, kv_lora_rank = 512
llama_kv_cache_init: CPU KV buffer size = 72.91 MiB
llama_new_context_with_model: KV self size = 72.91 MiB, c^KV (q8_0): 72.91 MiB, kv^T: not used
llama_new_context_with_model: CPU output buffer size = 1.97 MiB
llama_new_context_with_model: CPU compute buffer size = 450.01 MiB
llama_new_context_with_model: graph nodes = 3487
llama_new_context_with_model: graph splits = 1
system_info: n_threads = 128 / 512 | AVX = 1 | AVX_VNNI = 1 | AVX2 = 1 | AVX512 = 1 | AVX512_VBMI = 1 | AVX512_VNNI = 1 | AVX512_BF16 = 1 | FMA = 1 |
NEON = 0 | SVE = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE =
1 |
perplexity: tokenizing the input ..
perplexity: tokenization took 888.249 ms
perplexity: calculating perplexity over 561 chunks, n_ctx=512, batch_size=2048, n_seq=4
perplexity: 14.92 seconds per pass - ETA 34.85 minutes
[1]nan,[2]nan,[3]nan,[4]nan,[5]nan,[6]nan,[7]nan,[8]nan,[9]nan,[10]nan,[11]nan,[12]nan,[13]nan,[14]nan,[15]nan,[16]nan,[17]nan,[18]nan,[19]nan,[20]nan
,[21]nan,[22]nan,[23]nan,[24]nan,[25]nan,[26]nan,[27]nan,[28]nan,[29]nan,[30]nan,[31]nan,[32]nan,[33]nan,[34]nan,[35]nan,[36]nan,[37]nan,[38]nan,[39]n
an,[40]nan,[41]nan,[42]nan,[43]nan,[44]nan,[45]nan,[46]nan,[47]nan,[48]nan,[49]nan,[50]nan,[51]nan,[52]nan,[53]nan,[54]nan,[55]nan,[56]nan,[57]nan,[58
]nan,[59]nan,[60]nan,[61]nan,[62]nan,[63]nan,[64]nan,[65]nan,[66]nan,[67]nan,[68]nan,[69]nan,[70]nan,[71]nan,[72]nan,[73]nan,[74]nan,[75]nan,[76]nan,[
77]nan,[78]nan,[79]nan,[80]nan,[81]nan,[82]nan,[83]nan,[84]nan,[85]nan,[86]nan,[87]nan,[88]nan,[89]nan,[90]nan,[91]nan,[92]nan,[93]nan,[94]nan,[95]nan
,[96]nan,[97]nan,[98]nan,[99]nan,[100]nan,^C
Another recent PR allows for |
Beta Was this translation helpful? Give feedback.
-
|
Hi Guys, I've been struggling on my dual Xeon 8558 (48cores) with 768Gb RAM and Quad 3090 with Q8 (that is on lamma.cpp mainline, Q4_K_S gives me 6-7 tk/s in real world prompting) - gives me nan's, can you recommend and help to create custom quants for my situation? I would like to get best performance and ik_llama.cpp seems on the edge, I've been following this thread but might get lost in details calculating and applying custom quants logic... |
Beta Was this translation helpful? Give feedback.
-
Heya, so assuming you have set BIOS to Also you don't have enough RAM to run ktransformers compiled with So your best bet is probably as follows:
To start out I'd recommend simply trying to run your existing CUDA_VISIBLE_DEVICES="0," \
numactl -N 0 -m 0 \
./build/bin/llama-server \
--alias somequant/DeepSeek-R1-Q4_K_S \
--model /models/somequant/DeepSeek-R1-Q4_K_S.gguf \
-rtr \
--ctx-size 32768 \
-ctk q8_0 \
-mla 2 -fa \
-amb 512 \
-fmoe \
--n-gpu-layers 63 \
--override-tensor exps=CPU \
--parallel 1 \
--numa numactl \
--threads 48 \
--host 127.0.0.1 \
--port 8080
# if you get assert error after `============ llm_load_tensors: need to compute 61 wk_b tensors`
# git checkout 68a5b604 # and try with that version
This is with Okay, holler if you get stuck and looking forward to hearing your results! Also feel free to chat about how to make quants, I put some rough notes in this guide where I'm stumbling through the process myself haha... |
Beta Was this translation helpful? Give feedback.
-
|
Well, assuming nan is a token with a single D (basically the output is DDDDD...) - I'm using vannilla llama.cpp@main the same way as with Q4_K_S, it loads, and start outputting D's with out any errors, after I close session it give me tok/s stats, prompt eval is also low vs Q4_K_S at around 0,57 tok/s |
Beta Was this translation helpful? Give feedback.
-
|
Ok here is a bit of testing - I was getting around 6-6.7 tok/s on vanilla llama and achieved 10.8 tok/s on ik_llama on 8192 context. That is Q4_K_S. I was getting assert errors so had to checkout the given branch. Currently I've done exact instructions besides I didnt isolate to 1 3090 but used all 4 - anyways its offloads whatever (not expert layers as these are CPU override) at around 11Gb on each GPUs - I'm looking in trying a single CPU 4677 motherboard with 2 dimms per channel - this will give me 768GB but for 1 NUMA node and I probably can try Q8 on it |
Beta Was this translation helpful? Give feedback.
-
None that still do that haven't been mentioned in the conversation already, there was an issue with IQ1_S_R4 but that was fixed here: #194
Everything looks reasonable to me (especially since you were thorough and tried a bunch of valid combinations, and any valid combination shouldn't NaN on perplexity, but since all of them do that might help narrow down where the problem lies).
Nice. |
Beta Was this translation helpful? Give feedback.
-
I don't think it's the size that is the issue, iq2_bn_r4 is a bitnet quant. I briefly tested an IQ1_S_R4 which didn't even have the benefit of going to q8_0 for the non expert tensors like you did and I still got FAR more reasonable perplexity numbers (exact values here, with the quant log here ) If you are still experimenting with quant types, you might be able to improve on your Q2_K_R4 at around the same size by replacing the q2_k_r4, and q3_k_r4 which are k quants with similar sized i quants or iqk quants instead of using k quants, this PR #85 has a really nice chart focusing on that quant range (caveat IQ3_KL is not a quant type, it is a quant recipe), and shows how the three different quant types (i, k and iqk) stack up. |
Beta Was this translation helpful? Give feedback.
-
Does WizardLM-2-8x22B or any other 8x22B interest you as that could fit, and someone tried it (albeit on llama.cpp) here and got good results.
Thanks, I periodically check on this page as github doesn't notify on edits. |
Beta Was this translation helpful? Give feedback.
-
|
I ran a quick comparison between the For this specific config seems like pp is slightly slower but tg is slightly faster! With basically the same perplexity and 5% smaller, these non-linear
|
Beta Was this translation helpful? Give feedback.
-
Yes, I basically always use IQK quants, and at higher bpw levels ( where I-quants do not exist) they are often a far better quality option at their size (see: the data in #83 and #89) which is why for models that I use in the 4.25-7 bpw range I make an IQK quant (with an imatrix). |
Beta Was this translation helpful? Give feedback.
-
Quantized 8x22B is something I can run on my Ryzen-5975WX. I get |
Beta Was this translation helpful? Give feedback.
-
|
With my paltry 16 GB RTX-4080 that is in the Ryzen-7950WX box, I get |
Beta Was this translation helpful? Give feedback.
-
They are actually great. But they are Bitnet quants, so quants for a model that has been trained such that model weights take one of 3 possible values (-1, 0, 1). Hence, they absolutely cannot be used for normal models trained using actual floats. But that does not make them not great. The ternary quants in this repo ( |
Beta Was this translation helpful? Give feedback.
-
|
Okay gotchu. Yeah I picked them hoping they were fast, but given R1 was not trained as a bitnet they are not the right match for this specific case. |
Beta Was this translation helpful? Give feedback.
-
|
The |
Beta Was this translation helpful? Give feedback.
-
|
@ubergarm huge thanks for this guide! Any chance you could publish the DeepSeek-R1_Q2_K_R4 quant described here? First of all, thanks for doing all the research on running DeepSeek-R1 locally and publishing high quality technical details. Your posts on level1techs and reddit are currently the only good sources of information available on the subject. My internet searches related to purchasing decisions for running DSR1 always end up on one of your posts! I started with a 7975wx system for CPU only inference, and overclocked the memory controller based on your benchmarking on level1techs. Then, based on this guide, I ended up shelling out for an RTX 5090. Switching from CPU only inferencw with ollama to CPU+GPU inferece with ik_llama resulted in a 5x inference speedup. The speed improvement are more pronounced for longer contexts, I am able to get roughly 10 tps inference on a 40k context with the unsloth/DeepSeek-R1-UD-Q2_K_XL quant. Since 5090 has more memory, I offloaded all the small layers onto the GPU with Would love to get my hands on the DeepSeek-R1_Q2_K_R4 quant! |
Beta Was this translation helpful? Give feedback.
-
|
Heya @anikiforovopensource , I appreciate the feedback, its been great working with tools provided by the great developers to push the envelope! Glad you have found some of this useful
I updated the guide with a link to the hugging face repo that contains a couple https://huggingface.co/ubergarm/DeepSeek-V3-0324-GGUF Sorry it is difficult to piece together all the bread crumbs across so many sites, but sounds like you are having good success.
The 5090 is pretty great size 32GB VRAM for the quants I made actually. Use the CPU+GPU example on the model card, you want to be using I would recommend:
I'd love to see any benchmark results, you can see how to run Cheers and good luck, sounds like you have a great rig to experiment! |
Beta Was this translation helpful? Give feedback.
-
Where are my 136k stars 😃 |
Beta Was this translation helpful? Give feedback.
-
|
Has something changed with how llama-quantize wants the Specifically, it gives me e.g. |
Beta Was this translation helpful? Give feedback.
-
|
There have been no changes related to custom quants. Can you post your full command? |
Beta Was this translation helpful? Give feedback.
-
|
Sure! I arrived at: It also doesn't like q6_k, but is ok with q4_0. I dug around a little, but |
Beta Was this translation helpful? Give feedback.
-
|
Oh, this is Kawrakow-style usability at its best! The "K" in k-quants need to be capitalized. So, This applies only to |
Beta Was this translation helpful? Give feedback.
-
|
Oh man, thanks. I actually tried different capitalizations, but hadn't gone as far as mixing them! |
Beta Was this translation helpful? Give feedback.
-
|
Ok, I run the benchmarks, results are below. System: 7975wx with FCLK=2100 , 768G RAM at 5600MHz, RTX 5090.
Partial benchmark logsunsloth/DeepSeek-R1-UD-Q2_K_XL--override-tensor down_exps=CPU,gate_exps=CPU,up_exps=CPU./build/bin/llama-sweep-bench main: n_kv_max = 32768, n_batch = 2048, n_ubatch = 512, flash_attn = 1, n_gpu_layers = 200, n_threads = 32, n_threads_batch = 128
--override-tensor exps=CPU./build/bin/llama-sweep-bench main: n_kv_max = 32768, n_batch = 2048, n_ubatch = 512, flash_attn = 1, n_gpu_layers = 200, n_threads = 32, n_threads_batch = 128
ubergarm/DeepSeek-V3-0324-IQ2_K_R4--override-tensor down_exps=CPU,gate_exps=CPU,up_exps=CPU./build/bin/llama-sweep-bench main: n_kv_max = 32768, n_batch = 2048, n_ubatch = 512, flash_attn = 1, n_gpu_layers = 200, n_threads = 32, n_threads_batch = 128
--override-tensor exps=CPU./build/bin/llama-sweep-bench
|
Beta Was this translation helpful? Give feedback.
-
So, where are my 13k stars? One also has a longer context and better quantization options available... |
Beta Was this translation helpful? Give feedback.
-
I said something to that tune on Feb 19, ik_llama.cpp has improved a lot since then. Even then and still now I still see ktransformers as more of a performance demo because of how limited it is in what it supports both in hardware and the server/API they expose.
I was never sure if you wanted more publicity, I always offered technical support and explanations whenever ik_llama.cpp was brought up and only brought it up when it was relevant to discussions, but there were times I felt like I could have posted about it and gotten strong reception but I never did because I wasn't sure if you wanted this project to be popular.
I find this repo amazing, and it is full of options, but popularity and quality aren't linked. Your bitnet implementation is far better than the popular Microsoft one, but the Microsoft one (which also has 13k stars), is far better known. |
Beta Was this translation helpful? Give feedback.
-
I'm not necessarily looking for popularity (as you say, the correlation between popularity and quality is not very strong), but KTransformers copying code from here without acknowledgement (see #319) does rub me the wrong way. You can for sure post about that. And I'm now thinking that if this repository was better known, perhaps they wouldn't do it so blatantly. They do acknowledge to have taken the CPU implementation from |
Beta Was this translation helpful? Give feedback.
-
I saw that discussion, and I wasn't really happy with it either, but that isn't the sort of thing I would post about. My potential posts were more feature/performance highlights.
That may have helped avoid the situation.
I really don't see the different inference engine as competitors, they just serve different niches. |
Beta Was this translation helpful? Give feedback.
-
|
@anikiforovopensource One last quick tip, if you want to sacrifice some quality in exchange for extra speed add |
Beta Was this translation helpful? Give feedback.
-
|
@ubergarm I incorporated some of your suggestions and re-run the benchmark.
I ran
From my tests,
I prefer to run R1 instead of V3, so I currently don't have the quant to utilize more RAM. I can run benchmarks on your Benchmark results (system: 7975wx with FCLK=2100 , RAM at 5600MHz, RTX 5090):
Partial benchmark logsGPU-ctk f16 -ctv f16, --override-tensor all_but_3_expsVRAM: 30G, RAM: 216G./build/bin/llama-sweep-bench main: n_kv_max = 32768, n_batch = 2048, n_ubatch = 512, flash_attn = 1, n_gpu_layers = 200, n_threads = 32, n_threads_batch = 32
GPU (best so far)-ctk f16 -ctv f16, --override-tensor down_exps=CPU,gate_exps=CPU,up_exps=CPUVRAM: 18.5G, RAM: 228G./build/bin/llama-sweep-bench main: n_kv_max = 32768, n_batch = 2048, n_ubatch = 512, flash_attn = 1, n_gpu_layers = 200, n_threads = 32, n_threads_batch = 32
GPU-ctk q8_0VRAM: 17.5G, RAM: 228G./build/bin/llama-sweep-bench main: n_kv_max = 32768, n_batch = 2048, n_ubatch = 512, flash_attn = 1, n_gpu_layers = 200, n_threads = 32, n_threads_batch = 32
CPU with ctk=f16./build/bin/llama-sweep-bench
|
Beta Was this translation helpful? Give feedback.
-
If you have the capability, I would recommend making your own quants, that way you can optimally make them exactly to your system specs. |
Beta Was this translation helpful? Give feedback.
-
|
I fixed some cooling issues with the system and re-run benmarks with Perplexity for
Benchmark results (system: 7975wx with FCLK=2100 , RAM at 5600MHz, RTX 5090): |
Beta Was this translation helpful? Give feedback.
-
|
@saood06 thanks, I'll try making my own quant targetting 32G VRAM. I could use some tips on how to validate it :) |
Beta Was this translation helpful? Give feedback.
-
|
@ubergarm I tested |
Beta Was this translation helpful? Give feedback.
-
|
i-quants tend to be slower than k-quants (the only exceptions being |
Beta Was this translation helpful? Give feedback.
-
This is only true when attention is computed on the GPU (on the GPU |
Beta Was this translation helpful? Give feedback.
-
|
It's interesting to see how applying one optimization immediately moves the bottleneck somewhere else, running these models is pushing the hardware limits in different ways. |
Beta Was this translation helpful? Give feedback.
-
|
Hello, I have a question. I'm using a laptop 2060 and I'm trying to speed up partial offloading for Gemma 3 12B. I've compiled your build of llama.cpp with CUDA and AVX2 to see if there's any improvement compared to mainline, however it was noticeably slower. In the readme it is mentioned that for CUDA you need to offload the token embeddings tensors to the GPU, but nowhere can I see the command to do that. I think its --override-tensor but I don't know the specific command. I tried ffn_down_exps=CUDA0 which resulted in a speedup almost on par with main, but using that and ffn_up_exps=CUDA0, gate_exps=CUDA0 results in a performance loss again (although I think the latter of which is only for MoE models?) What is the command for doing that? Thank you! |
Beta Was this translation helpful? Give feedback.
-
|
Well, RoPE can indeed only take |
Beta Was this translation helpful? Give feedback.
-
|
With mainline I'm not getting this error, but yes I'm pretty sure llama.cpp is using context shifting as a default. In ST there's also token padding. So llama.cpp is probably using ctx shift while your build uses RoPE, could that be it? |
Beta Was this translation helpful? Give feedback.
-
But are you using quantized KV cache with mainline? It is very slow, no? |
Beta Was this translation helpful? Give feedback.
-
Yes you are very right about that. I've took a cup of coffee and... waited until it was all said and done.
As you can see, Quant KV Cache + FA and Gemma 3 is completely unsuable with mainline llama.cpp. However, it doesn't throw the error that I've mentioned above. |
Beta Was this translation helpful? Give feedback.
-
This is interesting. I'll need to investigate. It is not that I couldn't implement RoPE for |
Beta Was this translation helpful? Give feedback.
-
|
Alright. I want to put down some baseline numbers. I've built a system with EPYC 9175F and 768 GB @5600, with 2x RTX 6000 Ada Generation for 96 GB VRAM. Due to my dumb ass and inexperience with this kind of hardware, I'm running without GPUs and RAM is configured at 3600 for the time being. Pulled down ubergarm/DeepSeek-V3-0324-IQ4_K_R4 and running it with ik_llama.cpp on master, with config flags: RTR seems to have a huge impact. Overall things are about 66% faster than mainline llama.cpp with the unsloth 4-bit quant. I'm actually okay with this TG, but I gotta get my PP up 😜; my use case requires trawling through a lot of context. I'll check back in when I get GPU working and RAM at expected speed. |
Beta Was this translation helpful? Give feedback.
-
|
Can you share your command line that resulted in dog slow performance with 2 GPUs? With that I can give you a more informed answer to your question about expected performance increase with a 48-core CPU. |
Beta Was this translation helpful? Give feedback.
-
|
Sorry I didn't comprehend all the "Case A, B, C...F" stuff above as it was too dense to read.
What really?? Oh, I found it in an AMD white paper, you're right:
If I didn't already mention it, can you configure your BIOS to |
Beta Was this translation helpful? Give feedback.
-
|
Hybrid LLM execution models.pdf Okay, I illustrated it. Hope it makes things more clear. |
Beta Was this translation helpful? Give feedback.
-
Yeah GGUF naming convention is a bit different than transformers convention.
You can learn more by using
Sorry, I appreciate the image but I don't understand what you're asking? Are you asking "what is the best way to run a particular LLM on my specific hardware with ik_llama.cpp right now?" ? In general just try some things out and A/B test with llama-sweep-bench to see what is faster and keep iterating. See what commands other folks are using and what they say is faster/better. Sorry I don't have more motivation for this big question. |
Beta Was this translation helpful? Give feedback.
-
I'll restate the thread of discussion from the beginning.
I wrote out six possible execution models (CASE "B" through "G") and asked at which case, execution becomes not supported or impossible in llama.cpp? |
Beta Was this translation helpful? Give feedback.
-
|
can you please add to the guide: llama-sweep-bench |
Beta Was this translation helpful? Give feedback.
-
|
The guide is missing a lot of things as this fork has been moving pretty quickly. Your best bet in general is to search closed PRs for more details. Regarding llama-sweep-bench:
I believe @saood06 introduced it in #225
On this fork it will be built and live in
consciousness. of what else could this universe be comprised? I have some examples in my recent speed benchmark methodology gist as well. You can use the python script that comes with it to make plots or vibe code your own plotting tool etc. Basically you figure out the command you want to use for your specific system then replace the binary with |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for putting this guide together! I have to say ik_llama.cpp has been a great experience so far for me:
I'm already very happy with the tokens/s I'm getting from ik_llama.cpp when using DeepSeek-R1-UD-Q2_K_XL: What I'd like to try to optimize now is the context size. Specs of the machine:
Current maximum context size I managed to get so far was 41000. Full ik_llama.cpp run arguments: Is there any way to squeeze a larger context size out of this hardware, while maintaining reasonable tokens/s (>15tps)? Thanks for any help and for working on this! |
Beta Was this translation helpful? Give feedback.
-
|
In addition to above recommendations, if you have configured BIOS to set each socket as a single NUMA node e.g. On intel Xeon in my limited experience the optimal number of threads for PP is larger than for TG. |
Beta Was this translation helpful? Give feedback.
-
|
@ubergarm thanks. I did disable NUMA in BIOS. With the options you suggested I'm getting ~10% faster PP: That's with |
Beta Was this translation helpful? Give feedback.
-
Great you got it going! As ik mentioned, if you have some VRAM left-over you might be able to offload another layer or so of experts to GPU another small boost and max out performance in this configuration e.g. I'm not sure on sapphire rapids intel xeon, but your BIOS may also have some kind of Finally, while |
Beta Was this translation helpful? Give feedback.
-
|
@ubergarm, thanks for those pointers! As for I tried to apply your suggestion to use Here are my ik_llama.cpp arguments with Corresponding TG speed: Then TG speed with all the same arguments, except for a lack of Does my |
Beta Was this translation helpful? Give feedback.
-
|
I'm not 100% on the best The idea being to assign just one or two or however many fit until you OOM of the routed expert layers (exps) onto specific GPUs with the balance being caught by the final regex and going to CPU/RAM. Implicitly everything not overridden like attention and shared experts (shexp) will be split normally as you used You'll probably have to fiddle with the regex as needed to catch the right tensors/layers for your remaining VRAM. Some folks like the [3-4[0-9] style and others like the (0|2|4|6|8) style depending on how your brain works haha... And finally you should be able to use Good luck! |
Beta Was this translation helpful? Give feedback.
-
|
Hi Everyone, Great thread on the subject, and was very helpful for me to optimize the oldish hardware I currently have to play with this. I wanted to share some of the results of my experiments after reading everything here, and see if anyone has any further suggestions on how to make things faster for CPU only? 1 - I'm using 2 Xeon Gold (Skylake) with 1TB of ram If I enable subcluster and leave interleaving disabled, the 2 CPUs will present 4 numa nodes. With subcluster disabled and interleaving disabled, I get 1 node per CPU. And finally, with numa disabled and interleaving enabled, I get a single node for both CPUs Using the intel mlc tool, the maximum bandwidth is achieved with 1 numa node per CPU, around 100gb / s each. Having a single node for both CPUs gives me around 130gb / s. In theory, going with 2 nodes should be faster, but in reality, it seems like having everything consolidated under a single numa node is the fastest option (around 30% faster). I'm using Windows, perhaps the results would be better on Linux? Best result I got so far: G:\ik_llama>llama-bench.exe --model "G:\Qwen3-235B-A22B-128K-Q8_0-00001-of-00006.gguf" -mla 3 -fa 1 -t 28 --run-time-repack 1
Any suggestions are appreciated! :-) |
Beta Was this translation helpful? Give feedback.
-
|
Hey glad you got it going on your system. Thanks a lot for the detailed explanation of the BIOS settings as I don't have access to intel xeon BIOS. I had never heard of "node interleaving" option and just assumed that dual socket intel had no equivalent of AMD Right, I watched a good deep dive on AMD Epyc server BIOS on level1techs youtube recently and the AMD engineers basically said "don't use NPS0 unless your workload is not optimized at all" and that is basically the case for all CPU inferencing engines so even though aggregate RAM bandwidth goes down it will likely be the fastest for now. You could compare a single numa node setup with having 1x numa node per socket and running with So quick possible optimizations thoughts for you given you are running CPU only:
Have fun tweaking! |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for providing such a detailed reply, this has been super helpful! I ended up spending some more time on this, and wanted to share my results: 1 - Windows turned out to be a big limitation, as it is not possible to control NUMA behavior the same way as you can in Linux. I also tried Proxmox, but could not figure out how to reach the maximum bandwidth in a Linux VM. I ended up installing Debian on bare metal, and easily got close to 200gb / s doing the Intel MLC test, with 2 numa nodes 2 - Equipped with Debian bare metal, I was now able to use numactl, and the best results were obtained with numactl --interleave=all and --numa distribute, I got over 4 tokens / s on the llama-bench. Not a spetacular result as I was expecting, but better than the max I had reached before with Windows 3 - I switched over to your model (Qwen3-235B-A22B-mix-IQ3_K) as you suggested, and that also helped in real world usage once I started the llama-server. After filling up the context, I can still get over 3 t/s, not bad! 4 - fmoe and ctk / ctv did not make much of a difference 5 - final startup command with best results: numactl --interleave=all /media/xyz/data/ik_llama/llama-bench --model /media/xyz/data/Qwen3-235B-A22B-mix-IQ3_K-00001-of-00003.gguf -mla 3 -fa 1 --run-time-repack 1 --numa distribute Also adding the screenshots below in case anyone has a similar system and wants to play with NUMA: Sub numa cluster settings if you want to reduce numa nodes from 4 to 2 (probably best option on Linux bare metal): Interleaving, probably the best option on windows if you want to present a single numa node: Sub numa cluster is disabled if you enable interleaving (as expected): |
Beta Was this translation helpful? Give feedback.
-
|
What's the easiest method to produce a file that simply applies the --runtime-repack transformation to an existing GGUF? I can run DeepSeek at Q_8 but the startup time is a killer. |
Beta Was this translation helpful? Give feedback.
-
I ran it once a few months ago but lost my logs and my rigs are tied up at the moment. Someone was asking me on reddit too: https://www.reddit.com/r/LocalLLaMA/comments/1kb97ys/comment/mvg837s/ If you want to repack everything for CPU inferencing, it is basically There is an option for regex matching if you only want to repack some tensors, check out |
Beta Was this translation helpful? Give feedback.
-
|
#274 and #272 are where you can find more details about this. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks @saood06 I couldn't find my old logs for this but apparently I'd buried a command in a detail fold over two months ago. So @cmoncure probably something like this would work if you want to repack all the attn/shexp layers to optimize for running without any GPU: Then you should be able to start up with mmap() and no longer need to wait for |
Beta Was this translation helpful? Give feedback.
-
|
Thank you, I will try it this evening and let you know. Much appreciated. |
Beta Was this translation helpful? Give feedback.
-
|
Hi everyone, First, I want to sincerely thank @ikawrakow for this amazing repo (definitely deserves much more attention!), and @ubergarm for his excellent guides, insights, and quants. Big appreciation also goes out to unsloth and bartowski. I'm currently building a new AI/LLM machine. Although it's still a WIP (with some cooling issues), I couldn't resist running some tests. The final setup will run Proxmox, and will have multiple GPUs, but for now, it is AMD Epyc 9355 with 768 GB RAM and single RTX 4090 running Windows. Without much expertise, I managed to compile the library with: cmake -B build -G Ninja ^
-DCMAKE_BUILD_TYPE=Release ^
-DLLAMA_CURL=OFF ^
-DGGML_CUDA=ON ^
-DGGML_BLAS=OFF ^
-DGGML_AVX512=ON ^
-DGGML_AVX512_VNNI=ON ^
-DGGML_AVX512_BF16=OFF ^
-DCMAKE_CUDA_ARCHITECTURES=89
cmake --build build --config Release -j $env:NUMBER_OF_PROCESSORSHonestly, I’m unsure if I'm losing performance by disabling With ik-llama finally running, I tested DeepSeek-V3 quants with various params, and ended up with these:
Results
Observations and Thoughts
Logs - ubergarmLogs - unslothLogs - bartowskiI have NPS0 set in BIOS, and "LLC as NUMA domain (ACPI SRAT L3 Cache as NUMA domain)" ENABLED. It might be worth re-testing with this option DISABLED. I will test smaller and larger quants, too, but downloads take ages 😃. Anyway, just wanted to say "thanks" and share my excitement 💯. |
Beta Was this translation helpful? Give feedback.
-
|
@sousekd Your While I can see that competition for PCI-E bandwidth/latency may hinder PP improvements, I'm not sure I understand why one cannot get TG speed improvement by having additional routed experts offloaded to the second GPU. No tensor data is copied from RAM to VRAM when generating tokens, so PCI-E shouldn't be a bottleneck, so I expect to see at least some TG speed improvement. I'm quite interested in improving the speed further if possible, so I think it would be useful for you to post what you have tried and the results. You may want to start a new discussion for that as this one is getting difficult to follow all comments. |
Beta Was this translation helpful? Give feedback.
-
|
Thank you, @ikawrakow for your thoughts. The system is an EPYC 9355 (32 cores) with 12x DDR5-6400, and the latest results above are from a single RTX 5090 on PCIe 5.0 x16. Previous results were from a single RTX 4090 on PCIe 4.0 x16. Combined - without much tuning of the parameters - both PP t/s and TG t/s were significantly lower than on a single GPU. Oh, and it's currently running on Windows Server - only temporarily. 
Yes, I will play with params and benchmark more and once I have some results, I will open a new discussion. The reason I post these results (and params) are meant to help other people. When I was deciding on what hardware to buy for running these huge models the lack of available information and real results on larger contexts was putting me off. All I was able to find is "MacBook Pro can run DeepSeek", but no information about how the performance is degrading with growing context... and k-transformers for AMX. Anyway, it is quite possible I am doing something wrong, or Windows. Thank you very much - the numbers are great as they are, but obviously one can always try to improve, and the fact the second GPU did not help surprised me. |
Beta Was this translation helpful? Give feedback.
-
|
Just helped some of the multi-gpu crew tune up their commands. Feel free to take a look on how they are achieving over 300 tok/sec PP and almost 20 tok/sec TG on my newest quants (using very fast IQ2_KS and the new IQ3_KS): https://huggingface.co/ubergarm/DeepSeek-TNG-R1T2-Chimera-GGUF/discussions/2 Yeah feel free to start a new discussion listing your hardware and multi-GPU arrangement as well as your current command and folks can help workshop it. There is a lot of confusion partially from my own older mistakes still floating around as well as the fact that Qwen3 has different tensor names than DeepSeek so the override-tensor regex commands look similar but are importantly different.
Yeah give your BIOS configuration as well e.g. if you have dual socket are you running I believe if you use dual GPU and are offloading efficiently TG should definitely be like ~1 tok/sec faster or so probably as a 4090 with ~1TB/sec VRAM bandwidth bets almost any CPU RAM speeds. |
Beta Was this translation helpful? Give feedback.
-
|
Btw, the other day I randomly came across a discussion in the KTransformers repository where 2 guys were thinking that |
Beta Was this translation helpful? Give feedback.
-
|
I attempted to address it there also: kvcache-ai/ktransformers#1417 (comment) I'll spend some time updating my huggingface model cards so hopefully people don't make this mistake and accidentally spread more misinformation. Kind of reminds me of "Brandolini's law" aka the "bullshit asymmetry principle":
thanks UPDATE:
|
Beta Was this translation helpful? Give feedback.
-
|
Thank you for the kind words!
Please post the compilation errors you get with There are places where I have added GEMM/GEMV implementations optimized for |
Beta Was this translation helpful? Give feedback.
-
Oh, you are 100% correct and I am an idiot. ik_llama.cpp builds perfectly fine with |
Beta Was this translation helpful? Give feedback.
-
|
I have a dual EPYC 9355 system which normally has 768gb of RAM across 24 channels and scores roughly 720gb/s memory bandwidth on the stream triad test. At the moment, I had a RDIMM failure, so I'm down a stick and I only have 23 channels and 736gb of system RAM. I also have a blackwell 6000 pro on this system. I run with NPS4 set in the system BIOS, so I have 8 numa domains. I typically run Deepseek-V3-0324 671b:Q4_K_XL, so that's the model I'll be showing benchmarks for here. I run this before every llama server startup: echo 0 | sudo tee /proc/sys/kernel/numa_balancing
echo 3 | sudo tee /proc/sys/vm/drop_cachesUsing ./build/bin/llama-batched-bench \
--model /data/DeepSeek-V3-0324-GGUF-UD/UD-Q4_K_XL/DeepSeek-V3-0324-UD-Q4_K_XL-00001-of-00008.gguf \
--numa numactl \
--threads 32 \
--ctx-size 163840 \
--n-gpu-layers 62 \
-ot ".ffn_.*_exps.=CPU" \
--seed 3407 \
--prio 3 \
--temp 0.3 \
--min-p 0.0 \
--flash-attn \
-npp 512 -ntg 128 -npl 1
main: n_kv_max = 163840, n_batch = 2048, n_ubatch = 512, flash_attn = 1, is_pp_shared = 0, n_gpu_layers = 62, n_threads = 32, n_threads_batch = 32
| PP | TG | B | N_KV | T_PP s | S_PP t/s | T_TG s | S_TG t/s | T s | S t/s |
|-------|--------|------|--------|----------|----------|----------|----------|----------|----------|
| 512 | 128 | 1 | 640 | 24.441 | 20.95 | 5.973 | 21.43 | 30.414 | 21.04 |With ./build/bin/llama-sweep-bench \
--model /data/DeepSeek-V3-0324-GGUF-UD/UD-Q4_K_XL/DeepSeek-V3-0324-UD-Q4_K_XL-00001-of-00008.gguf \
--alias DeepSeek-V3-0324:671b-q4_k_xl \
--numa numactl \
--threads 32 \
--ctx-size 163840 \
--n-gpu-layers 62 \
-ot ".ffn_.*_exps.=CPU" \
--seed 3407 \
--temp 0.3 \
--min-p 0.0 \
--flash-attn \
--host 0.0.0.0 \
-mla 3 \
-fmoe \
-rtr \
--port 11434
main: n_kv_max = 163840, n_batch = 2048, n_ubatch = 512, flash_attn = 1, n_gpu_layers = 62, n_threads = 32, n_threads_batch = 32
| PP | TG | N_KV | T_PP s | S_PP t/s | T_TG s | S_TG t/s |
|-------|--------|--------|----------|----------|----------|----------|
| 512 | 128 | 0 | 3.862 | 132.56 | 15.186 | 8.43 |
| 512 | 128 | 512 | 3.851 | 132.94 | 15.240 | 8.40 |
| 512 | 128 | 1024 | 3.873 | 132.19 | 15.232 | 8.40 |
| 512 | 128 | 1536 | 3.925 | 130.45 | 15.253 | 8.39 |I'm just curious: Why is generation tok/s so much lower in Thanks! |
Beta Was this translation helpful? Give feedback.
-
|
Hey thanks for taking some time to try this out. I too started using ktransformers but have since moved over to ik's for given he is the author on pretty much all the quants after the original
Both myself an fairydreaming have done a lot of research on the NUMA domain issue for both intel xeon and amd epyc dual socket rigs. the tl;dr; is I recommend you try out Generally PP will benefit from as much physical cores that you can throw at it, but TG will likely be fastest with some smaller number of threads so get the best of both worlds with I've been helping folks tune their exact command to get max speed, so I'll take a crack at yours as it stands assuming you are still running with 8 numa domains and haven't attempted the above BIOS optimizations yet: # build for RTX PRO Blackwel 96GB VRAM arch/capabilities 120 psure
cmake -B ./build -DGGML_CUDA=ON -DGGML_SCHED_MAX_COPIES=1 -DGGML_CUDA_IQK_FORCE_BF16=1 -DCMAKE_CUDA_ARCHITECTURES="120"
cmake --build ./build --config Release -j $(nproc)
# run on single CPU socket assuming NPS4 (4x domains per socket)
numactl --interleave=0,1,2,3 \
./build/bin/llama-sweep-bench \
--model /data/DeepSeek-V3-0324-GGUF-UD/UD-Q4_K_XL/DeepSeek-V3-0324-UD-Q4_K_XL-00001-of-00008.gguf \
-fa -mla 3 -fmoe -amb 512 -mg 0 \
--ctx-size 20480 \
-ngl 99 \
-ot "blk\.(3|4|5|6|7|8)\.ffn_.*=CUDA0" \
-ot exps=CPU \
--threads 32 \
--threads-batch 32 \
-ub 4096 -b 4096 \
-rtr \
--numa numactl \
--warmup-batchAdjust
Once you've dialed in the command you can then just switch out the executable back to Okay, let me know if u have any questions, you have a very nice rig! |
Beta Was this translation helpful? Give feedback.
-
|
Hi @createthis, I was able to achieve the following on (single) Epyc 9355 and RTX 5090:
As @ubergarm noted, try NPS0. Also, did you experiment with --numa param? I am not sure how/whether it is supported here. Edit: Huh, haven't seen @ubergarm's full response 😀. |
Beta Was this translation helpful? Give feedback.
-
|
I think you are observing a difference in GPU offload policy. In which for DeepSeek-R1/V3 translates to 1024 tokens. So, basically, in this benchmark you are not using the GPU at all, everything runs on the CPU when using
will give a nice table with PP and TG performance for 0...32k tokens in the KV cache. I think in Another comment related to the NUMA situation: I don't have access to a NUMA system myself, but people report that, sadly, on dual socket systems they get the best performance by disabling NUMA in the BIOS and running on a single CPU. @ubergarm has done quite a few experiments in that regard. I haven't followed what is happening in |
Beta Was this translation helpful? Give feedback.
-
|
But apart from everything else, worth pointing out that |
Beta Was this translation helpful? Give feedback.
-
|
Just to let you know guys, did some benchmarks on iklcpp on my setup (192GB RAM + 208GB VRAM) on DeepSeek V3/R1/Chimera of Q2_K_XL, IQ3_XXS, IQ3_KS, Q3_K_XL and IQ4_XS on reddit, if you want to take a look! https://www.reddit.com/r/LocalLLaMA/comments/1lwnj5x/performance_benchmarks_on_deepseek/ Performance of ikllamacpp for these kind of setups, is really impressive! |
Beta Was this translation helpful? Give feedback.
-
|
@ikawrakow here it is with NPS0: mla 3./build/bin/llama-sweep-bench \
--model /data/DeepSeek-V3-0324-GGUF-UD/UD-Q4_K_XL/DeepSeek-V3-0324-UD-Q4_K_XL-00001-of-00008.gguf \
--alias DeepSeek-V3-0324:671b-q4_k_xl \
--threads 32 \
--ctx-size 163840 \
--n-gpu-layers 62 \
-ot ".ffn_.*_exps.=CPU" \
--seed 3407 \
--temp 0.3 \
--min-p 0.0 \
--flash-attn \
--host 0.0.0.0 \
-mla 3 \
-fmoe \
-rtr \
--port 11434
main: n_kv_max = 163840, n_batch = 2048, n_ubatch = 512, flash_attn = 1, n_gpu_layers = 62, n_threads = 32, n_threads_batch = 32
| PP | TG | N_KV | T_PP s | S_PP t/s | T_TG s | S_TG t/s |
|-------|--------|--------|----------|----------|----------|----------|
| 512 | 128 | 0 | 3.677 | 139.23 | 12.996 | 9.85 |
| 512 | 128 | 512 | 3.994 | 128.19 | 13.160 | 9.73 |
| 512 | 128 | 1024 | 4.020 | 127.37 | 13.161 | 9.73 |
| 512 | 128 | 1536 | 4.279 | 119.65 | 13.426 | 9.53 |
| 512 | 128 | 2048 | 4.193 | 122.11 | 13.596 | 9.41 |
| 512 | 128 | 2560 | 3.868 | 132.38 | 12.987 | 9.86 |
| 512 | 128 | 3072 | 4.655 | 109.98 | 13.682 | 9.36 |
| 512 | 128 | 3584 | 4.291 | 119.31 | 13.344 | 9.59 |
| 512 | 128 | 4096 | 4.287 | 119.44 | 12.890 | 9.93 |
| 512 | 128 | 4608 | 4.221 | 121.29 | 12.835 | 9.97 |mla 2./build/bin/llama-sweep-bench \
--model /data/DeepSeek-V3-0324-GGUF-UD/UD-Q4_K_XL/DeepSeek-V3-0324-UD-Q4_K_XL-00001-of-00008.gguf \
--alias DeepSeek-V3-0324:671b-q4_k_xl \
--threads 32 \
--ctx-size 163840 \
--n-gpu-layers 62 \
-ot ".ffn_.*_exps.=CPU" \
--seed 3407 \
--temp 0.3 \
--min-p 0.0 \
--flash-attn \
--host 0.0.0.0 \
-mla 2 \
-fmoe \
-rtr \
--port 11434
main: n_kv_max = 163840, n_batch = 2048, n_ubatch = 512, flash_attn = 1, n_gpu_layers = 62, n_threads = 32, n_threads_batch = 32
| PP | TG | N_KV | T_PP s | S_PP t/s | T_TG s | S_TG t/s |
|-------|--------|--------|----------|----------|----------|----------|
| 512 | 128 | 0 | 3.766 | 135.95 | 12.805 | 10.00 |
| 512 | 128 | 512 | 3.774 | 135.66 | 12.753 | 10.04 |
| 512 | 128 | 1024 | 3.833 | 133.59 | 13.051 | 9.81 |
| 512 | 128 | 1536 | 4.051 | 126.38 | 13.200 | 9.70 |
| 512 | 128 | 2048 | 3.882 | 131.89 | 13.089 | 9.78 |
| 512 | 128 | 2560 | 3.887 | 131.71 | 13.085 | 9.78 |
| 512 | 128 | 3072 | 3.993 | 128.24 | 13.275 | 9.64 |
| 512 | 128 | 3584 | 4.380 | 116.89 | 13.879 | 9.22 |
| 512 | 128 | 4096 | 4.273 | 119.82 | 13.199 | 9.70 |
| 512 | 128 | 4608 | 4.115 | 124.41 | 12.996 | 9.85 |Doesn't seem to make much difference mla 2 vs 3. PP speed does continue to rise past 32 threads though, which is suprising: ./build/bin/llama-sweep-bench \
--model /data/DeepSeek-V3-0324-GGUF-UD/UD-Q4_K_XL/DeepSeek-V3-0324-UD-Q4_K_XL-00001-of-00008.gguf \
--alias DeepSeek-V3-0324:671b-q4_k_xl \
--threads 61 \
--ctx-size 163840 \
--n-gpu-layers 62 \
-ot ".ffn_.*_exps.=CPU" \
--seed 3407 \
--temp 0.3 \
--min-p 0.0 \
--flash-attn \
--host 0.0.0.0 \
-mla 2 \
-fmoe \
-rtr \
--port 11434
main: n_kv_max = 163840, n_batch = 2048, n_ubatch = 512, flash_attn = 1, n_gpu_layers = 62, n_threads = 61, n_threads_batch = 61
| PP | TG | N_KV | T_PP s | S_PP t/s | T_TG s | S_TG t/s |
|-------|--------|--------|----------|----------|----------|----------|
| 512 | 128 | 0 | 3.274 | 156.36 | 12.792 | 10.01 |
| 512 | 128 | 512 | 3.174 | 161.33 | 12.924 | 9.90 |
| 512 | 128 | 1024 | 3.099 | 165.22 | 13.011 | 9.84 |
| 512 | 128 | 1536 | 3.204 | 159.83 | 13.140 | 9.74 |
| 512 | 128 | 2048 | 3.196 | 160.22 | 13.131 | 9.75 |
| 512 | 128 | 2560 | 3.093 | 165.54 | 13.327 | 9.60 |
| 512 | 128 | 3072 | 3.443 | 148.70 | 13.393 | 9.56 |
| 512 | 128 | 3584 | 3.369 | 151.97 | 13.454 | 9.51 |
| 512 | 128 | 4096 | 3.413 | 150.02 | 13.577 | 9.43 | |
Beta Was this translation helpful? Give feedback.
-
|
Another sort of interesting result: This is NPS4 with ./build/bin/llama-batched-bench \
--model /data/DeepSeek-V3-0324-GGUF-UD/UD-Q4_K_XL/DeepSeek-V3-0324-UD-Q4_K_XL-00001-of-00008.gguf \
--numa numactl \
--threads 32 \
--ctx-size 163840 \
--n-gpu-layers 62 \
-ot "blk\.(3|4|5|6|7|8|9|10|11)\.ffn_.*=CUDA0" \
-ot exps=CPU \
-ub 4096 -b 4096 \
--seed 3407 \
--prio 3 \
--temp 0.3 \
--min-p 0.0 \
--flash-attn \
-npp 4096 \
-ntg 1024 \
-npl 1
main: n_kv_max = 163840, n_batch = 4096, n_ubatch = 4096, flash_attn = 1, is_pp_shared = 0, n_gpu_layers = 62, n_threads = 32, n_threads_batch = 32
| PP | TG | B | N_KV | T_PP s | S_PP t/s | T_TG s | S_TG t/s | T s | S t/s |
|-------|--------|------|--------|----------|----------|----------|----------|----------|----------|
| 4096 | 1024 | 1 | 5120 | 21.628 | 189.38 | 46.545 | 22.00 | 68.173 | 75.10 |The "real world" numbers, which are small context, as you pointed out: ./build/bin/llama-server \
--model /data/DeepSeek-V3-0324-GGUF-UD/UD-Q4_K_XL/DeepSeek-V3-0324-UD-Q4_K_XL-00001-of-00008.gguf \
--alias DeepSeek-V3-0324:671b-q4_k_xl \
--no-webui \
--numa numactl \
--threads 32 \
--ctx-size 163840 \
--n-gpu-layers 62 \
-ot "blk\.(3|4|5|6|7|8|9|10|11)\.ffn_.*=CUDA0" \
-ot exps=CPU \
-ub 4096 -b 4096 \
--seed 3407 \
--prio 3 \
--temp 0.3 \
--min-p 0.0 \
--log-colors \
--flash-attn \
--host 0.0.0.0 \
--jinja \
--port 11434
It is a huge time suck switching between NPS0 and NPS4. The machine takes like 10 minutes to reboot. @ubergarm I'm interested in trying out your llama.cpp sweep benchmark. I need to get some work done on a paid project at the moment, but I'll try to take a look later this weekend and report my findings. I'll also report higher context real world results as they come in. I don't have an agentic workload at the moment, so I'm not sure when that will be, but maybe I can fabricate one this weekend if nothing pops up today. Thanks for all the feedback and help thus far! |
Beta Was this translation helpful? Give feedback.
-
|
This is still NPS4 with I'm never 100% sure I'm reading these correctly, but I think this is performance at ./build/bin/llama-server \
--model /data/DeepSeek-V3-0324-GGUF-UD/UD-Q4_K_XL/DeepSeek-V3-0324-UD-Q4_K_XL-00001-of-00008.gguf \
--alias DeepSeek-V3-0324:671b-q4_k_xl \
--no-webui \
--numa numactl \
--threads 32 \
--ctx-size 163840 \
--n-gpu-layers 62 \
-ot "blk\.(3|4|5|6|7|8|9|10|11)\.ffn_.*=CUDA0" \
-ot exps=CPU \
-ub 4096 -b 4096 \
--seed 3407 \
--prio 3 \
--temp 0.3 \
--min-p 0.0 \
--log-colors \
--flash-attn \
--host 0.0.0.0 \
--jinja \
--port 11434
Not too shabby performance. EDIT: updated to be the same prompt as the below 47k context "real world" examples for an apples to apples comparison |
Beta Was this translation helpful? Give feedback.
-
|
"real world" NPS0 with ./build/bin/llama-server \
--model /data/DeepSeek-V3-0324-GGUF-UD/UD-Q4_K_XL/DeepSeek-V3-0324-UD-Q4_K_XL-00001-of-00008.gguf \
--alias DeepSeek-V3-0324:671b-q4_k_xl \
--no-webui \
--threads 32 \
--ctx-size 163840 \
--n-gpu-layers 62 \
-ot "blk\.(3|4|5|6|7|8|9|10|11)\.ffn_.*=CUDA0" \
-ot exps=CPU \
-ub 4096 -b 4096 \
--seed 3407 \
--prio 3 \
--temp 0.3 \
--min-p 0.0 \
--log-colors \
--flash-attn \
--host 0.0.0.0 \
--jinja \
--port 11434
This is in-line with my original findings. |
Beta Was this translation helpful? Give feedback.
-
|
"real world" NPS0 ./build/bin/llama-server \
--model /data/DeepSeek-V3-0324-GGUF-UD/UD-Q4_K_XL/DeepSeek-V3-0324-UD-Q4_K_XL-00001-of-00008.gguf \
--alias DeepSeek-V3-0324:671b-q4_k_xl \
-ot "blk\.(3|4|5|6|7|8|9|10|11)\.ffn_.*=CUDA0" \
-ot exps=CPU \
--threads 32 \
--threads-batch 64 \
-ub 4096 -b 4096 \
--ctx-size 163840 \
--n-gpu-layers 62 \
--seed 3407 \
--temp 0.3 \
--min-p 0.0 \
--flash-attn \
--host 0.0.0.0 \
-mla 3 \
-fmoe \
-amb 512 \
-mg 0 \
-rtr \
--port 11434
This performance is quite good. PP is slightly better than NPS4 |
Beta Was this translation helpful? Give feedback.
-
|
@createthis as related to the comparison of ik_llama.cpp and llama.cpp. The following likely unrelated to your case, but I will mention it just in case someone else would have the issue. Today I was installing the ik_llama.cpp and was unable to [do] it. It was falling out with: after stracing it I realized that the compiled ik_llama.cpp binary is trying to pickup the /usr/local/lib/libggml.so and /usr/local/lib/libggml-base.so. I realized that these are from the old installation of ollama! Hence please make sure that the ik_llama.cpp doesn't pick up the libraries from the llama.cpp and wise versa lol! Again, it might be absolutely unrealed but still. |
Beta Was this translation helpful? Give feedback.
-
|
transferring from kvcache-ai/ktransformers#1417 Short story -- I would like to switch to the ik_llama.cpp from ktransformers (the ktransformers are having huge problems with the stability). I would like to know how I can run Deepseek R1/V3 with 128k context and more. In the ktransformers they used the matrix absorption trick ( https://docs.flashinfer.ai/api/mla.html, https://github.com/madsys-dev/deepseekv2-profile/blob/main/workspace/blog/optimizing-mla.md ) -- that is, the flashinfer allows to use one 24GB GPU to prefill up to 128k context (i never tried more because I didn't know the Deepseek supports 163k). So what can be done currently in my case to support large context? I have a various machines mostly with Threadripper Pro 3995wx (inc. lenovo-locked), overclocked Samsung ECC RAM up to 3200 MT/s and currently up to 3 GPUs RTX 3090 FE per workstation with p2p enabled: Currently researching what @ubergarm suggested and actually trying to fix the bug in ktransformers. Please advise what can be done. [EDIT]: Currently doing this: Its running well on a single GPU but its only 41k context. [EDIT2]: it seems to be that lots of people having trouble using flashinfer instead of flash attention. For example: https://github.com/turboderp-org/exllamav3
The same thing goes for ik_llama.cpp etc. -- the matrix absorption trick in flash infer is not available in flashattn hence the for the full context in ik_llama.cpp its required to have at least 48 GB VRAM which is not ideal. |
Beta Was this translation helpful? Give feedback.
-
Hello there! Wondering, what was your command to test PPL? I want to try with some models I have but I get just "nan" for some reason, so maybe it's an issue on my end (highly factible). And these models work perfectly on normal usage. |
Beta Was this translation helpful? Give feedback.
-
|
You just get what? The docs on Perplexity is in this current thread (see above). quote:
|
Beta Was this translation helpful? Give feedback.
-
|
The quoted comments about NaNs and |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for the result on that perplexity score @magikRUKKOLA it lines up with my own estimates of the smaller quants. That guide is indeed hopelessly outdated already haha.. Using q8_0 quantized cache will drop the score just a tiny bit, and mla 3 is pretty much always the way to go now. Here is an example of what I've been using lately for smaller models and two CUDA GPUs: |
Beta Was this translation helpful? Give feedback.
-
|
Many thanks to all! I did re test and finally worked, after months haha. Finally could test R1 0525 IQ4_XS, from unsloth. Result is So it is surprisingly close to Q4_K_XL, but probably is slower for TG. Also both are really close to Q8 (3.2119), by 1-2%. Finally I will be able to test V3 0324 quants PPL, but I don't have the Q8 ppl sadly haha. |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
ik_llama.cppLast Updated: Tue May 13 03:52:20 PM EDT 2025 (still needs more updates, can't keep up, check through comments below)
NEW: Two new custom quants great for CPU+GPU or CPU only inferencing fitting 32k+ context in under 24GB VRAM here on huggingface ubergarm/DeepSeek-V3-0324-GGUF! or start out with the quant you already have to kick the tires on ik_llama.cpp.
tl;dr;
ik_llama.cppis a custom fork of llama.cpp introducing many interesting optimizations for MoE's like DeepSeek-R1 671B.The new SOTA quant types can repack your existing GGUFs on the fly or you can roll your own to maximize quality and speed for your exact system VRAM and RAM availability.
I highly recommend you give
ik_llama.cppa try especially for CUDA+CPU or pure CPU inferencing. All the very similar ergonmics as vanillallama-serverthat you already know and love.Install
Features
Quick Start
Existing DeepSeek-R1 671B GGUF
Get 64k context with a single 24GB VRAM GPU using your existing unsloth quants like unsloth/DeepSeek-R1-UD-Q2-K_XL!
Custom Quant
I rolled my own custom quant to improve quality while still fitting 32k context in under 24GB VRAM. No need to use
-rtras this quant is already repacked so you can still usemmap()allowing you to run on systems without enough RAM by paging the disk cache. This quant has lower perplexity thanUD-Q2_K_XLwhile only being slightly larger/slower. Good size for 256GB RAM systems whereQ4_K_Mdoesn't fit.Custom Quants
👇
Click here for how to make your own custom quants including repacking
☝️
Benchmarking
Test Rig
mlcmemory read bandwidthLinux TR24 6.13.0-061300-generic #202501302155 SMP PREEMPT_DYNAMIC Sat Feb 8 09:06:55 UTC 2025 x86_64 x86_64 x86_64 GNU/Linuxllama-bench
Note
ik_llama.cpp llama-benchdoesn't seem to iterate over all variables so fix these manually for test cases:-fmoe 0,1-rtr 0,1-otprobably, i didn't test this specifically as always usingexps=CPUfor this rig...It does seem to iterate over variables for
fa,mla, andamb.Perplexity
Even more perplexity logs
There is a lot going on here. There may be some issues with
nanand "numerical instability" depending on exact quants and llama.cpp forks in use. So this is still evolving.I made the above png graph using the first 35 chunks for easy comparison as generally
nandidn't appear too early for most quants.I also haven't compared perplexity across
ik_llama.cppwith different settings (e.g. mla etc) vs vanilla llama.cpp and CPU vs CUDA backends etc.The following exact detailed logs results are not included yet in the graph above.
Q8_0I ran the unsloth
Q8_0on that intel6980P CPU only backend with vanillallama.cpp/main@b1b132effor a baseline. Note there is no MLA etc yet in this case.ubergarm
Q2_K_R4This is a custom quant I rolled with
q8_0for all attention/shared experts/embeddings loaded on GPU. The rest of the MoE down exps areq3_k_r4and gate/up exps areq2_k_r4which gives fast speed quant that fits nicely into under 256GB RAM and 24GB VRAM with about 32k context without sacrificing much perplexity.This was run on
ik_llama.cpp@127c6ee6ubergarm
Q2_K_R4with various-ser N,1Testing same quant and config as above but with
-ser 4,1etc to get a feel for quality vs speed tradeoffs.These were run on
ik_llama.cpp@127c6ee6ubergarm
IQ2_BN_R4This is an experimental quant I rolled with
q8_0for all attention/shared experts/embeddings loaded on GPU. The rest of the MoE down exps areiq2_xs_r4and gate/up exps areiq2_bn_r4. However, perplexity looks pretty bad. So I'll likely aim for larger sized model with higher quality quants and make-up speed/accuracy trade off exploring-serinstead of going very small quants.Looking back on it with advise from the team, bitnet quants are very fast to compute, but only good quality for models trained specifically as a ternary bit-net. So this is not the correct use-case.
This was run on
ik_llama.cpp@127c6ee6ubergarm
IQ2_K_R4Another experimental quant with
q8_0for all GPU layers (with room for 32k context still) anddown=iq3_k_r4andgate/up=iq2_k_r4for-ot exps=CPUCPU offload.Debugging Crashes
Usually no need to do this, as any asserts will print the line number direclty.
TODO
References
Beta Was this translation helpful? Give feedback.
All reactions