Why does the server-cuda container consume CPU time?

[wencan]

I'm running a model in the server-cuda container, and from the monitor, I can see that the model is loaded onto the GPU, but it's consuming CPU time during execution.

docker run --gpus all  --rm  --network host -v /home/wencan/Projects/models/:/models ghcr.io/ggerganov/llama.cpp:server-cuda -m /models/hfl/llama-3-chinese-8b-instruct-v2-gguf/ggml-model-q4_k.gguf --port 8000 --host 0.0.0.0 -n 512 --n-gpu-layers 33

system info

OS: Debian GNU/Linux 12 (bookworm) x86_64 
Kernel: 6.1.0-21-amd64 
Uptime: 1 hour, 44 mins 
Packages: 2170 (dpkg), 63 (flatpak), 3 (snap) 
Shell: bash 5.2.15 
Resolution: 2560x1440 
DE: GNOME 43.9 
WM: Mutter 
WM Theme: Adwaita 
Theme: Adwaita [GTK2/3] 
Icons: Adwaita [GTK2/3] 
Terminal: gnome-terminal 
CPU: AMD Ryzen 7 1800X (16) @ 3.600GHz 
GPU: NVIDIA GeForce GTX 1080 Ti 
Memory: 5795MiB / 15896MiB 

docker info

Client:
 Context:    default
 Debug Mode: false

Server:
 Containers: 12
  Running: 3
  Paused: 0
  Stopped: 9
 Images: 14
 Server Version: 20.10.24+dfsg1
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runtime.v1.linux nvidia runc io.containerd.runc.v2
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 1.6.20~ds1-1+b1
 runc version: 1.1.5+ds1-1+deb12u1
 init version: 
 Security Options:
  apparmor
  seccomp
   Profile: default
  cgroupns
 Kernel Version: 6.1.0-21-amd64
 Operating System: Debian GNU/Linux 12 (bookworm)
 OSType: linux
 Architecture: x86_64
 CPUs: 16
 Total Memory: 15.52GiB
 Name: debian12
 ID: Y6MZ:UI3N:Q3N5:F5F7:4UYJ:7RMA:47YH:YBBU:PKG2:LIZ5:6NMQ:PPKK
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Registry Mirrors:
  https://hub-mirror.c.163.com/
 Live Restore Enabled: false

docker image info:

REPOSITORY                      TAG           IMAGE ID       CREATED         SIZE
ghcr.io/ggerganov/llama.cpp     server-cuda   85e97d6afaf3   2 days ago      2.2GB

[JohannesGaessler]

You need to run CPU code in order to coordinate the GPU code so it's inevitable that the CUDA container will consume some CPU time.

[wencan]

@JohannesGaessler
I've considered that too. However, according to the monitor, it seems that a lot of CPU time is always being consumed during content generation, and no GPU has been consumed.