Research: performance divergence

[VinnyG9]

Research Stage

• Background Research (Let's try to avoid reinventing the wheel)
• Hypothesis Formed (How do you think this will work and it's effect?)
• Strategy / Implementation Forming
• Analysis of results
• Debrief / Documentation (So people in the future can learn from us)

Previous existing literature and research

when i ran benches previously i got pretty good results on cpu inference like 30-40t/s on qwen3 30B, now i am trying to run the server for aider and the speed is less than half is it expected??

Hypothesis

No response

Implementation

No response

Analysis

No response

Relevant log output

[ikawrakow]

Please be specific in your issue. Provide quantization type used, system information, full command line to start the server and, ideally, last good/first bad commit where you observe the performance change. See #474 for an example.

[VinnyG9]

I've been testing ik_llama.cpp for about a month mostly benchmarks can't report any regression
running bare build time flags(NATIVE=1, CUDA=1, CUDA_ARCH) and runtime flags(rtr, fa, fmoe, numa)

latest ubuntu/mint

no matter the model i try, dense, MoE etc i get less than 50% performance than the benchmarks show, when running mainline the benchmark numbers are way lower but are consistent with performance numbers when running the server

[ikawrakow]

So, the issue is that the performance you observe when running llama-server is 2X lower than the performance you observe when running one of the benchmark tools?

Is the PP performance affected, or the TG performance, or both?

Generic statement will lead nowhere (other than the issue getting closed)

[VinnyG9]

So, the issue is that the performance you observe when running llama-server is 2X lower than the performance you observe when running one of the benchmark tools?

yes

Is the PP performance affected, or the TG performance, or both?

both, literally a bit less than half PP/TG. think it could be a numa issue? i tried with stock bios settings but got worse results albeit closer bench/serve numbers

[saood06]

both, literally a bit less than half PP/TG. think it could be a numa issue? i tried with stock bios settings but got worse results albeit closer bench/serve numbers

Please provide the exact commands used for benching and for server.

I brought llama-sweep-bench over to this repo and use it regularly because in my experience it does accurately reflect server performance (including how it changes across different depths), to the point where I run it to validate that the model has warmed up and loaded into RAM correctly (as my server is very sensitive to memory placement and the model is stored on HDDs so performance is unusable until the model is warmed up).

[Ph0rk0z]

Its funny because I often get slightly better speeds on server than the sweep bench. Nowhere near half so something is wrong.

Only numa thing that helps is adding interleave=all to the command line you run. Setting load balancing in the kernel to 0 doesn't do move the needle one way or another despite the warning.

One thing I did notice is that the bench can be a little irregular at times by a t/s here or there. May I also suggest getting ccmake when compiling so you can set your flags and forget it.

edit:

So playing with 4096 batches showed me something. In server, prompt speed on smaller prompts prints as half speed or less.I was getting 110 max and on a 2k token prompt would hit 60-70 reported. A large enough prompt still returns the correct speed. Can't explain 1/2 TG tho.

[VinnyG9]

both, literally a bit less than half PP/TG. think it could be a numa issue? i tried with stock bios settings but got worse results albeit closer bench/serve numbers

Please provide the exact commands used for benching and for server.

I brought llama-sweep-bench over to this repo and use it regularly because in my experience it does accurately reflect server performance (including how it changes across different depths), to the point where I run it to validate that the model has warmed up and loaded into RAM correctly (as my server is very sensitive to memory placement and the model is stored on HDDs so performance is unusable until the model is warmed up).

server:

  "Qwen3-30B-MoE":
    env:
      - "CUDA_VISIBLE_DEVICES= "
    proxy: "http://192.168.15.101:9999"
    cmd: |
      /ssd/share/Software/backends/ik_llama.cpp/build/bin/llama-server
      --host localhost --port 9999 --flash-attn
      --cache-type-k f16 --cache-type-v f16
      --ctx-size 40960
      --samplers "top_k;top_p;min_p;temperature;typ_p;xtc"
      --temp 0.6 --repeat-penalty 1.0
      --min-p 0.01 --top-k 20 --top-p 0.95
      -ngl 0 -rtr -fmoe -ser 7,1 --threads 31 --numa distribute
      --model /models/gguf/MoE/Qwen3-30B-A3B-128K-UD-Q4_K_XL.gguf

output

INFO [   launch_slot_with_task] slot is processing task | tid="123321346306048" timestamp=1748881719 id_slot=0 id_task=399
INFO [            update_slots] kv cache rm [p0, end) | tid="123321346306048" timestamp=1748881719 id_slot=0 id_task=399 p0=450
INFO [           print_timings] prompt eval time     =    3133.99 ms /   388 tokens (    8.08 ms per token,   123.80 tokens per second) | tid="123321346306048" timestamp=1748881752 id_slot=0 id_task=399 t_prompt_processing=3133.991 n_prompt_tokens_processed=388 t_token=8.077296391752578 n_tokens_second=123.80380160632241
INFO [           print_timings] generation eval time =   29634.35 ms /   400 runs   (   74.09 ms per token,    13.50 tokens per second) | tid="123321346306048" timestamp=1748881752 id_slot=0 id_task=399 t_token_generation=29634.354 n_decoded=400 t_token=74.085885 n_tokens_second=13.497847801912604
INFO [           print_timings]           total time =   32768.35 ms | tid="123321346306048" timestamp=1748881752 id_slot=0 id_task=399 t_prompt_processing=3133.991 t_token_generation=29634.354 t_total=32768.345
INFO [            update_slots] slot released | tid="123321346306048" timestamp=1748881752 id_slot=0 id_task=399 n_ctx=40960 n_past=1237 n_system_tokens=0 n_cache_tokens=1237 truncated=false
INFO [            update_slots] all slots are idle | tid="123321346306048" timestamp=1748881752

sweep-bench:

CUDA_VISIBLE_DEVICES= numactl --interleave=all /ssd/share/Software/backends/ik_llama.cpp/build/bin/llama-sweep-bench -m /models/gguf/MoE/Qwen3-30B-A3B-128K-UD-Q4_K_XL.gguf -rtr -fa -fmoe --numa distribute -t 31 -c 8196 -b 2048 -ub 512

main: n_kv_max = 8448, n_batch = 2048, n_ubatch = 512, flash_attn = 1, n_gpu_layers = -1, n_threads = 31, n_threads_batch = 31

|    PP |     TG |   N_KV |   T_PP s | S_PP t/s |   T_TG s | S_TG t/s |
|-------|--------|--------|----------|----------|----------|----------|
|   512 |    128 |      0 |    1.981 |   258.43 |    4.757 |    26.91 |
|   512 |    128 |    512 |    2.157 |   237.35 |    4.642 |    27.57 |
|   512 |    128 |   1024 |    2.421 |   211.47 |    5.040 |    25.40 |
|   512 |    128 |   1536 |    2.844 |   180.04 |    4.951 |    25.85 |
|   512 |    128 |   2048 |    2.991 |   171.20 |    5.313 |    24.09 |
|   512 |    128 |   2560 |    3.222 |   158.89 |    5.136 |    24.92 |
|   512 |    128 |   3072 |    3.525 |   145.24 |    5.442 |    23.52 |
|   512 |    128 |   3584 |    3.758 |   136.25 |    5.559 |    23.03 |
|   512 |    128 |   4096 |    4.089 |   125.20 |    5.580 |    22.94 |
|   512 |    128 |   4608 |    4.262 |   120.14 |    5.563 |    23.01 |
|   512 |    128 |   5120 |    4.832 |   105.96 |    6.061 |    21.12 |
|   512 |    128 |   5632 |    4.954 |   103.36 |    6.060 |    21.12 |
|   512 |    128 |   6144 |    5.218 |    98.12 |    6.202 |    20.64 |
|   512 |    128 |   6656 |    5.664 |    90.40 |    6.193 |    20.67 |
|   512 |    128 |   7168 |    5.776 |    88.65 |    6.122 |    20.91 |
|   512 |    128 |   7680 |    6.135 |    83.46 |    7.535 |    16.99 |
failed to decode the batch, n_batch = 2048, ret = 1
main: llama_decode() failed

llama-bench

CUDA_VISIBLE_DEVICES= /ssd/share/Software/backends/ik_llama.cpp/build/bin/llama-bench -m /models/gguf/MoE/Qwen3-30B-A3B-128K-UD-Q4_K_XL.gguf -t 31 --numa distribute -rtr 1 -fa 1 -fmoe 1 -n 64,128,256,512 -p 512,1024,2048,4096
ggml_cuda_init: failed to initialize CUDA: no CUDA-capable device is detected
WARNING: /proc/sys/kernel/numa_balancing is enabled, this has been observed to impair performance

model	size	params	backend	ngl	threads	fa	rtr	fmoe	test	t/s
============ Repacked 337 tensors
qwen3moe ?B Q4_K - Medium	16.49 GiB	30.53 B	CUDA	99	31	1	1	1	pp512	254.35 ± 7.37
qwen3moe ?B Q4_K - Medium	16.49 GiB	30.53 B	CUDA	99	31	1	1	1	pp1024	226.91 ± 7.88
qwen3moe ?B Q4_K - Medium	16.49 GiB	30.53 B	CUDA	99	31	1	1	1	pp2048	206.85 ± 6.65
qwen3moe ?B Q4_K - Medium	16.49 GiB	30.53 B	CUDA	99	31	1	1	1	pp4096	163.43 ± 2.74
qwen3moe ?B Q4_K - Medium	16.49 GiB	30.53 B	CUDA	99	31	1	1	1	tg64	32.71 ± 0.89
qwen3moe ?B Q4_K - Medium	16.49 GiB	30.53 B	CUDA	99	31	1	1	1	tg128	33.11 ± 0.54
qwen3moe ?B Q4_K - Medium	16.49 GiB	30.53 B	CUDA	99	31	1	1	1	tg256	31.07 ± 1.80
qwen3moe ?B Q4_K - Medium	16.49 GiB	30.53 B	CUDA	99	31	1	1	1	tg512	27.44 ± 3.13

[VinnyG9]

and running on GPU

INFO [            update_slots] all slots are idle | tid="123510069428224" timestamp=1748883135
INFO [   launch_slot_with_task] slot is processing task | tid="123510069428224" timestamp=1748883146 id_slot=0 id_task=408
INFO [            update_slots] kv cache rm [p0, end) | tid="123510069428224" timestamp=1748883146 id_slot=0 id_task=408 p0=456
INFO [           print_timings] prompt eval time     =    1171.24 ms /   381 tokens (    3.07 ms per token,   325.30 tokens per second) | tid="123510069428224" timestamp=1748883151 id_slot=0 id_task=408 t_prompt_processing=1171.238 n_prompt_tokens_processed=381 t_token=3.0741154855643047 n_tokens_second=325.2968226782259
INFO [           print_timings] generation eval time =    3364.22 ms /   124 runs   (   27.13 ms per token,    36.86 tokens per second) | tid="123510069428224" timestamp=1748883151 id_slot=0 id_task=408 t_token_generation=3364.215 n_decoded=124 t_token=27.13076612903226 n_tokens_second=36.8585242025257
INFO [           print_timings]           total time =    4535.45 ms | tid="123510069428224" timestamp=1748883151 id_slot=0 id_task=408 t_prompt_processing=1171.238 t_token_generation=3364.215 t_total=4535.453
INFO [            update_slots] slot released | tid="123510069428224" timestamp=1748883151 id_slot=0 id_task=408 n_ctx=40960 n_past=960 n_system_tokens=0 n_cache_tokens=960 truncated=false

main: n_kv_max = 8448, n_batch = 2048, n_ubatch = 512, flash_attn = 1, n_gpu_layers = 99, n_threads = 31, n_threads_batch = 31

PP	TG	N_KV	T_PP s	S_PP t/s	T_TG s	S_TG t/s
512	128	0	1.241	412.49	2.392	53.51
512	128	512	1.186	431.64	2.512	50.96
512	128	1024	1.223	418.59	2.613	48.99
512	128	1536	1.232	415.57	2.713	47.18
512	128	2048	1.275	401.43	2.819	45.41
512	128	2560	1.267	403.95	2.933	43.64
512	128	3072	1.318	388.40	3.042	42.07
512	128	3584	1.332	384.52	3.146	40.68
512	128	4096	1.366	374.89	3.267	39.18
512	128	4608	1.377	371.86	3.386	37.80
512	128	5120	1.389	368.53	3.533	36.23
512	128	5632	1.409	363.27	3.633	35.23
512	128	6144	1.432	357.60	3.710	34.51
512	128	6656	1.458	351.20	3.796	33.72
512	128	7168	1.492	343.10	3.905	32.78
512	128	7680	1.493	342.86	4.017	31.86
failed to decode the batch, n_batch = 2048, ret = 1
main: llama_decode() failed

[saood06]

Is there any reason why you use 31 threads? I would say try using 32 threads and see if that helps your performance (but I don't think that is the reason for the gap in performance between server and sweep).

See this comment about why that might be a bad choice: #223 (comment)

[VinnyG9]

Is there any reason why you use 31 threads? I would say try using 32 threads and see if that helps your performance (but I don't think that is the reason for the gap in performance between server and sweep).

See this comment about why that might be a bad choice: #223 (comment)

yeah when i benched it performance improved with the number of(physical) threads up to 31-32, though only for the moe's

is it normal that during generation the model pauses on every comma? i find it funny

[nux]

Not sure if relevant here - the topic name seems so. Was looking into some performance issues and found this thread.

nux@red ~/dev/ik_llama.cpp $ ./build/bin/llama-bench -m /mnt/nvme/models/ubergarm/DeepSeek-V3-0324-GGUF/DeepSeek-V3-0324-IQ4_K_R4/DeepSeek-V3-0324-IQ4_K_R4-00001-of-00010.gguf -p 512 -t 32 -mla 2 -fa 1 -fmoe 1 -ngl 99 --override-tensor "exps=CPU" -amb 512
ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
Device 0: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes

model	size	params	backend	ngl	fa	mla	amb	fmoe	test	t/s
deepseek2 671B IQ4_K_R4 - 4.5 bpw	386.18 GiB	672.05 B	CUDA	99	1	2	512	1	pp512	28.36 ± 0.03
deepseek2 671B IQ4_K_R4 - 4.5 bpw	386.18 GiB	672.05 B	CUDA	99	1	2	512	1	tg128	4.80 ± 0.01

I had a .txt file of an older benchmark showing this:
nux@red ~/dev/ik_llama.cpp $ ./build/bin/llama-bench -m /mnt/nvme/models/ubergarm/DeepSeek-V3-0324-GGUF/DeepSeek-V3-0324-IQ4_K_R4/DeepSeek-V3-0324-IQ4_K_R4-00001-of-00010.gguf -p 512 -t 32 -mla 2 -fa 1 -fmoe 1 -ngl 99 --override-tensor "exps=CPU" -amb 512
ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
Device 0: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes

model	size	params	backend	ngl	fa	mla	amb	fmoe	test	t/s
deepseek2 671B IQ4_K_R4 - 4.5 bpw	386.18 GiB	672.05 B	CUDA	99	1	2	512	1	pp512	76.13 ± 2.43
deepseek2 671B IQ4_K_R4 - 4.5 bpw	386.18 GiB	672.05 B	CUDA	99	1	2	512	1	tg128	9.79 ± 0.08

build: 1ea1df4 (3659)

I checked out the same checkout and ran it again and results were the same.

Am I missing anything obvious here? Something change that I need to adjust for?

Building like this:
cmake -B build -DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON -DLLAMA_CURL=ON
cmake --build build --config Release -j --clean-first

Running on 2x9115, 768gb ram, 3090 gpu

I ended up troubleshooting the performance I am seeing now when trying to figure out why ubergarm/DeepSeek-R1-0528-GGUF/IQ4_KS_R4 was running slowly.

If you think it makes sense for me to open a new issue I can, as my tg/pp have both slowed down, unlike what I'm reading about above

Thanks

[ikawrakow]

@Fuckingnameless

Your system seems to be one of those that are extremely finicky about tensor placement in RAM. Looking at the llama-bench vs llama-sweep-bench TG results I see a 20% difference in TG performance at zero context. There is an obvious difference between your server command and your benchmark runs: context is 41k tokens for the server and 8k or less in the benchmarks. This potentially changes where things go in RAM. Also, seeing numactl and --numa involved immediately rises red flags. Do you have a dual socket system? (I don't remember the system configurations of all users, so adding the details of your system to the issue would be useful).

Having said all this, I still find a factor of 2 difference in CPU performance strange. The difference is much less on CUDA, so I would focus on trying to resolve the CPU performance first.

[ikawrakow]

@nux

There was PR #461 that added CUDA implementation for some of the row-interleaved quants. This results in a change in behavior for your IQ4_K_R4 quantized model: prior to PR #461 all matrix multiplications for X_R4 tensors had to be done on the CPU. After PR #461, for batch size >= 32 they get offloaded to the GPU to perform the matrix multiplications. If the PCI-E speed is low for some reason, this can make PP slower. You can try adding -op 26,0,27,0,29,0 to the command line to see what happens. This will disable the offload to the GPU.

I have no explanation for the 2X lower TG performance. Try using -mla 3, which has been supported on the GPU since PR #408/#413

[nux]

I will put together a script to go through commits and benchmark to figure out exactly when this started. I'm noticing right now is that while llama-bench is running, the GPU utilization drops to 38-39% for about 10 seconds and going back up to 99%. While llama-bench is running I see this pattern repeating with gpu usage %

I have been using mla 3 - but ran the benchmark above in mla 2 for comparison purposes. PCI-E is 16x. Will post when I figure out what commit performance went down

[nux]

Commit 0976467 is when the performance went down for me. Was running for i in cut -d " " -f1 commits.txt ;do git checkout $i;./cmd-build.sh ;./start-bench.sh >> results.txt;done

start-bench is: ./build/bin/llama-bench -m /mnt/nvme/models/ubergarm/DeepSeek-V3-0324-GGUF/DeepSeek-V3-0324-IQ4_K_R4/DeepSeek-V3-0324-IQ4_K_R4-00001-of-00010.gguf -p 512 -t 32 -mla 2 -fa 1 -fmoe 1 -ngl 99 --override-tensor "exps=CPU" -amb 512

build: ccd6d9c (3716)

model	size	params	backend	ngl	fa	mla	amb	fmoe	test	t/s
deepseek2 671B IQ4_K_R4 - 4.5 bpw	386.18 GiB	672.05 B	CUDA	99	1	2	512	1	pp512	26.74 ± 0.05
deepseek2 671B IQ4_K_R4 - 4.5 bpw	386.18 GiB	672.05 B	CUDA	99	1	2	512	1	tg128	4.80 ± 0.00

build: 0976467 (3715)

model	size	params	backend	ngl	fa	mla	amb	fmoe	test	t/s
deepseek2 671B IQ4_K_R4 - 4.5 bpw	386.18 GiB	672.05 B	CUDA	99	1	2	512	1	pp512	26.75 ± 0.04
deepseek2 671B IQ4_K_R4 - 4.5 bpw	386.18 GiB	672.05 B	CUDA	99	1	2	512	1	tg128	4.81 ± 0.00

build: 1429291 (3714)

model	size	params	backend	ngl	fa	mla	amb	fmoe	test	t/s
deepseek2 671B IQ4_K_R4 - 4.5 bpw	386.18 GiB	672.05 B	CUDA	99	1	2	512	1	pp512	76.24 ± 1.44
deepseek2 671B IQ4_K_R4 - 4.5 bpw	386.18 GiB	672.05 B	CUDA	99	1	2	512	1	tg128	10.08 ± 0.06

build: 24c010b (3713)

model	size	params	backend	ngl	fa	mla	amb	fmoe	test	t/s
deepseek2 671B IQ4_K_R4 - 4.5 bpw	386.18 GiB	672.05 B	CUDA	99	1	2	512	1	pp512	77.25 ± 0.70
deepseek2 671B IQ4_K_R4 - 4.5 bpw	386.18 GiB	672.05 B	CUDA	99	1	2	512	1	tg128	10.07 ± 0.06