Bug: on AMD gpu, it offloads all the work to the CPU unless you specify --n-gpu-layers on the llama-cli command line

[differentprogramming]

What happened?

I spent days trying to figure out why it running a llama 3 instruct model was going super slow (about 3 tokens per second on fp16 and 5.6 on 8 bit) on an AMD MI50 32GB using rocBLAS for ROCm 6.1.2, using 0% GPU and 100% cpu even while using some vram. I'm currently using release b3246

Finally I noticed that (for 8 bit) it said:

llm_load_tensors: offloading 0 repeating layers to GPU
llm_load_tensors: offloaded 0/33 layers to GPU
llm_load_tensors:        CPU buffer size =  8137.64 MiB

adding something to the command line like "--n-gpu-layers 100" changed it to

llm_load_tensors: offloading 32 repeating layers to GPU
llm_load_tensors: offloading non-repeating layers to GPU
llm_load_tensors: offloaded 33/33 layers to GPU
llm_load_tensors:      ROCm0 buffer size =  7605.33 MiB
llm_load_tensors:        CPU buffer size =   532.31 MiB

and it jumped from

llama_print_timings:        load time =    1955.34 ms
llama_print_timings:      sample time =      15.28 ms /   128 runs   (    0.12 ms per token,  8378.61 tokens per second)
llama_print_timings: prompt eval time =       0.00 ms /     0 tokens (    -nan ms per token,     -nan tokens per second)
llama_print_timings:        eval time =   23640.83 ms /   128 runs   (  184.69 ms per token,     5.41 tokens per second)
llama_print_timings:       total time =   23754.79 ms /   128 tokens

to

llama_print_timings:        load time =    2824.15 ms
llama_print_timings:      sample time =      12.57 ms /   128 runs   (    0.10 ms per token, 10182.98 tokens per second)
llama_print_timings: prompt eval time =       0.00 ms /     0 tokens (    -nan ms per token,     -nan tokens per second)
llama_print_timings:        eval time =    2469.71 ms /   128 runs   (   19.29 ms per token,    51.83 tokens per second)
llama_print_timings:       total time =    2566.10 ms /   128 tokens

The CPU usage went down from maxing out a bunch of cores to only using one.
The GPU usage went from 0% up to the 99%

16 bit didn't improve as much
it went from

llama_print_timings:        load time =    2497.93 ms
llama_print_timings:      sample time =      15.75 ms /   128 runs   (    0.12 ms per token,  8126.47 tokens per second)
llama_print_timings: prompt eval time =       0.00 ms /     0 tokens (    -nan ms per token,     -nan tokens per second)
llama_print_timings:        eval time =   43228.83 ms /   128 runs   (  337.73 ms per token,     2.96 tokens per second)
llama_print_timings:       total time =   43344.71 ms /   128 tokens

to

llama_print_timings:        load time =    3937.74 ms
llama_print_timings:      sample time =      13.31 ms /   128 runs   (    0.10 ms per token,  9616.11 tokens per second)
llama_print_timings: prompt eval time =       0.00 ms /     0 tokens (    -nan ms per token,     -nan tokens per second)
llama_print_timings:        eval time =    8066.54 ms /   128 runs   (   63.02 ms per token,    15.87 tokens per second)
llama_print_timings:       total time =    8166.01 ms /   128 tokens

This is the command line I'm testing with

build/bin/llama-cli -m ./models/llama3-8b-instruct/ggml-model-q8_0.gguf -n 128 --threads 12 --n-gpu-layers 100

after building with

HIPCXX="/opt/rocm-6.1.2/llvm/bin/clang" HIP_PATH="/opt/rocm-6.1.2" cmake -S . -B build -DGGML_HIPBLAS=ON -DAMDGPU_TARGETS=gfx906 -DCMAKE_BUILD_TYPE=Release && cmake --build build --config Release -- -j 12

Yes I put the paths in manually. It was just part of how I was messing around to see what's wrong

Name and Version

version: 0 (unknown)
built with cc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0 for x86_64-linux-gnu

The version string seems to be another bug, but I'm using the source from release b3246

What operating system are you seeing the problem on?

Linux

Relevant log output

llm_load_tensors: offloading 0 repeating layers to GPU
llm_load_tensors: offloaded 0/33 layers to GPU
llm_load_tensors:        CPU buffer size =  8137.64 MiB

[SteelPh0enix]

isn't this just intended behavior? --gpu-layers is just 0 by default.

[shibe2]

Not all the work is done on CPU. If you give it long prompt, it should use GPU.

[differentprogramming]

None of the work is being done by the gpu, it runs exactly as fast as CPU build and gpu use stays at 0%.

The instructions should warn you that you have to tell it to put layers into the gpu to use the gpu.

[differentprogramming]

Or just maybe, if you compile it to use a GPU the default should be to use it.

[differentprogramming]

Note that there are other issues where people complained about these speeds and never figured out why.
I think they eventually got closed because people stopped talking.

[SteelPh0enix]

I don't think enabling GPU offload by default is a good idea - partial offload can hurt performance too (we should not assume that the user will be able to automatically offload any model to GPU in full), and it can lead to more unexpected/unstable behaviors, leaving the CPU inference as default is a reasonable choice.

One could say that discovering the GPU offload is "off" by default is a "rite of passage" for a new llama.cpp user. While i agree that there should maybe be a clear information that by default GPU offloading is off (i haven't seen one yet, i'm talking about something like a h1 banner in readme), i think that reading the docs is probably a better solution than turning this on by default

[shibe2]

None of the work is being done by the gpu, it runs exactly as fast as CPU build and gpu use stays at 0%.

How long is your prompt?

[differentprogramming]

Another question, isn't 100% gpu usage for a 5 times uplift vs. a 6 core skylake cpu a bit low?
I can't really know because there's no matrix of uplift per common GPUs, different ways of using them (BLAS, CUDA, Vulkan etc.) and different models and format.
While I realize that would be a combinatorial mess, there should be an entry with some example tokens per second for an example model, a few variations on it, and common GPUs. And some idea whether CUDA is for instance the fastest and how BLAS or rocBLAS or Vulkan compares.

[differentprogramming]

For instance I have no idea if I would get any uplift from spending a few weeks Hipifying the CUDA build!

[differentprogramming]

None of the work is being done by the gpu, it runs exactly as fast as CPU build and gpu use stays at 0%.

How long is your prompt?

I wasn't testing with interactive mode, so it was using built in prompts at random.
I notice that changing the length of the result doesn't change the per token speed reported.

[dspasyuk]

@differentprogramming It is the intended behavior, you can always default to CPU when everything fails. As for the difference between CUDA and cuBLAS, I do not see much difference on GTX1060/P4/A100/A4500 GPUs ~about 20 tokens per second on GTX1060 using Llama-instruct-Q4 Here is my default setup: https://github.com/dspasyuk/llama.cui

[differentprogramming]

I deleted a reply because I hadn't noticed that dspasyuk had mentioned lots of gpus other than his 1060.

[dspasyuk]

@differentprogramming No worries, GTX1060 is about 10 times slower than A4500 (~200 tokens/s) and A100 (PCIE card) is about 180 tokens per/sec using Llama3-instract-q4 and yesterday's version of llama.cpp with llama.cui