Linked CUDA but cannot use GPU

[TD-Sky]

Environment

• OS: WSL2 Archlinux
• Toolchains (running with nix-shell): gcc11, cuda11.8
• GPU: NVIDIA GeForce RTX 4060 Ti, driver 552.44

Steps

1. Compile with CUDA stubs (I changed Makefile)
2. Prepare a chat LLM in models/llama-7b, rename to ggml-model-q4_0.gguf.
3. Run ./examples/chat.sh

Expected and actual

It should utilize GPU, but the CUDA statistics in tasks manager is 0%.

I also have ollama-cuda which linked CUDA statically. It will utilize GPU.

I have done the same thing in Linux, it doesn't work as well.

[RhinoDevel]

Do you have an output log from the beginning of llama-cli run?

[TD-Sky]

I found ggml_cuda_init: failed to initialize CUDA: CUDA driver is a stub library. But compilation would fail when linking CUDA directly.

[TD-Sky]

Updated to the latest commit and compiled successfully. But there are still CUDA driver is a stub library errors.

[RhinoDevel]

Seems as if the "real" CUDA library cannot be found. E.g. ChatGPT gives some detailed hints, if you just enter "CUDA driver is a stub library". :-)

[dspasyuk]

You need to specify -n-gpu-layers 35 for example for a typical &b model. Something like this:

../llama.cpp/llama-cli --model models/Meta-Llama-3-8B-Instruct_Q5_K_S.gguf --n-gpu-layers 25 -cnv --interactive-first --simple-io -b 512 -n -1 --ctx_size 0 --temp 0.3 --top_k 10 --multiline-input --repeat_penalty 1.12 -t 6 -r "/n>" --log-disable

[TD-Sky]

It utilized GPU when running on Linux physical machine but not on WSL. The latter can't even compile as both of them have same environment.

[TD-Sky]

Is your system running in WSL2?

[TD-Sky]

with import <nixpkgs> {};
(mkShell.override { stdenv = gcc11Stdenv; }) {
  buildInputs = [
    cudaPackages_11.cudatoolkit
    cudaPackages_11.cudnn
  ];
  shellHook = ''
    export CUDA_PATH=${cudaPackages_11.cudatoolkit}
    export EXTRA_LDFLAGS="-L/lib"
    export EXTRA_CCFLAGS="-I/usr/include"
  '';
}