Performance profiling suggestions

[wilderfield]

Does anyone have any recommended tools for profiling llama.cpp on Windows?
Is there any trace / profiling capability in llama.cpp?

I want to get a flame graph showing the call stack and the duration of various calls.

I might just use Visual Studio.
https://learn.microsoft.com/en-us/visualstudio/profiling/beginners-guide-to-performance-profiling?view=vs-2022

Thanks

[TD-Sky]

Same question. I wonder the recommended methods to profile llama.cpp applications on Linux.

[mariamrakka]

Hello, I have the same question. I tried to explicitly set DGGML_PERF=1 when I compiled because I could not find LLAMA_PERF in the Makefile, but I did not get the results as reported in the Tips and Tricks. To get results as reported in (https://github.com/ggerganov/llama.cpp/wiki/GGML-Tips-&-Tricks), what can we do? Thanks.

[compilade]

GGML_PERF seems to have been removed in #8017.

On Linux, as a workaround, I've been using perf as a sampling profiler.

$ perf --call-graph=lbr -- ./bin/llama-bench -m ../models/something.gguf
...
$ perf report --hierarchy -M intel

It's not really equivalent to GGML_PERF (much less granular as to which matmuls are expensive), but it kind of works to figure out what kind of operator takes most of the cpu time (and also which instructions, in the disassembly view).

[ruanych]

perf doesn't seem to be a universal solution, as it is not supported in some Linux systems customised for development boards.

May I ask why llama.cpp removed support for performance counting? Is it possible to provide some guide programme to restore it?

Thanks.

[ggerganov]

It's now recommended to use make -j tests && ./tests/test-backend-ops perf for performance measurements. This way, you can measure the performance not just on CPU, but also on GPU and other backends

[mjkpolo]

Is there a way to measure row parallel performance? Trying to build an analytical model and I seem to overestimate the bandwidth of loading weights from GPU memory in row parallel mode (assuming it's gpu mem bandwidth * ngpu since they should all load a shard of the weight in parallel for each layer)