Proprietary & OSS Nvidia modules are not found

[kubebn]

lspci -nnkvs 0000:b3:00.0
b3:00.0 VGA compatible controller [0300]: NVIDIA Corporation GA106 [RTX A2000 12GB] [10de:2571] (rev a1) (prog-if 00 [VGA controller])
	Subsystem: NVIDIA Corporation GA106 [RTX A2000 12GB] [10de:1611]
	Flags: fast devsel, IRQ 309, NUMA node 0, IOMMU group 1
	Memory at fa000000 (32-bit, non-prefetchable) [size=16M]
	Memory at 38ffe0000000 (64-bit, prefetchable) [size=256M]
	Memory at 38fff0000000 (64-bit, prefetchable) [size=32M]
	I/O ports at f000 [size=128]
	Expansion ROM at fb000000 [disabled] [size=512K]
	Capabilities: [60] Power Management version 3
	Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
	Capabilities: [78] Express Legacy Endpoint, MSI 00
	Capabilities: [b4] Vendor Specific Information: Len=14 <?>
	Capabilities: [100] Virtual Channel
	Capabilities: [250] Latency Tolerance Reporting
	Capabilities: [258] L1 PM Substates
	Capabilities: [128] Power Budgeting <?>
	Capabilities: [420] Advanced Error Reporting
	Capabilities: [600] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?>
	Capabilities: [900] Secondary PCI Express
	Capabilities: [bb0] Physical Resizable BAR
	Capabilities: [c1c] Physical Layer 16.0 GT/s <?>
	Capabilities: [d00] Lane Margining at the Receiver <?>
	Capabilities: [e00] Data Link Feature <?>
	Kernel driver in use: vfio-pci
	Kernel modules: nvidiafb, nouveau

lspci -nnkvs 0000:b3:00.1
b3:00.1 Audio device [0403]: NVIDIA Corporation GA106 High Definition Audio Controller [10de:228e] (rev a1)
	Subsystem: NVIDIA Corporation GA106 High Definition Audio Controller [10de:1611]
	Flags: fast devsel, IRQ 310, NUMA node 0, IOMMU group 1
	Memory at fb080000 (32-bit, non-prefetchable) [size=16K]
	Capabilities: [60] Power Management version 3
	Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
	Capabilities: [78] Express Endpoint, MSI 00
	Capabilities: [100] Advanced Error Reporting
	Capabilities: [160] Data Link Feature <?>
	Kernel driver in use: vfio-pci
	Kernel modules: snd_hda_intel

Image built:

sudo docker run --rm -t -v $PWD/.build/:/out -v $PWD/./image-cache.oci:/image-cache.oci:ro -v /dev:/dev --privileged     ghcr.io/siderolabs/imager:v1.10.4 metal --image-cache /image-cache.oci --image-disk-size=2GiB --extra-kernel-arg=init_on_alloc=1 --extra-kernel-arg=slab_nomerge --extra-kernel-arg=pti=on --extra-kernel-arg=consoleblank=0 --extra-kernel-arg=nvme_core.io_timeout=4294967295 --extra-kernel-arg=printk.devkmsg=on --extra-kernel-arg=ima_template=ima-ng --extra-kernel-arg=ima_appraise=fix --extra-kernel-arg=ima_hash=sha512 --extra-kernel-arg=console=ttyS0 --extra-kernel-arg=reboot=k --extra-kernel-arg=panic=1 --extra-kernel-arg=talos.shutdown=halt --extra-kernel-arg=talos.platform=metal --extra-kernel-arg=talos.config=metal-iso --system-extension-image ghcr.io/siderolabs/nvidia-container-toolkit-production:570.148.08-v1.17.8@sha256:bb79e8822115fe97fd2ed3683c228cd1a299023dc37d005aec4fbde1b9e623ad --system-extension-image ghcr.io/siderolabs/nvidia-open-gpu-kernel-modules-production:570.148.08-v1.10.4@sha256:6ec24cbf40392afc58303279e66fbecb9c3fd31be8c2137f214d63ffe0586516
skipped pulling overlay (no overlay)
profile ready:
arch: amd64
platform: metal
secureboot: false
version: v1.10.4
customization:
  extraKernelArgs:
    - init_on_alloc=1
    - slab_nomerge
    - pti=on
    - consoleblank=0
    - nvme_core.io_timeout=4294967295
    - printk.devkmsg=on
    - ima_template=ima-ng
    - ima_appraise=fix
    - ima_hash=sha512
    - console=ttyS0
    - reboot=k
    - panic=1
    - talos.shutdown=halt
    - talos.platform=metal
    - talos.config=metal-iso
input:
  kernel:
    path: /usr/install/amd64/vmlinuz
  initramfs:
    path: /usr/install/amd64/initramfs.xz
  sdStub:
    path: /usr/install/amd64/systemd-stub.efi
  sdBoot:
    path: /usr/install/amd64/systemd-boot.efi
  baseInstaller:
    imageRef: ghcr.io/siderolabs/installer-base:v1.10.4
  imageCache:
    imageRef: ""
    ociPath: /image-cache.oci
  systemExtensions:
    - imageRef: ghcr.io/siderolabs/nvidia-container-toolkit-production:570.148.08-v1.17.8@sha256:bb79e8822115fe97fd2ed3683c228cd1a299023dc37d005aec4fbde1b9e623ad
    - imageRef: ghcr.io/siderolabs/nvidia-open-gpu-kernel-modules-production:570.148.08-v1.10.4@sha256:6ec24cbf40392afc58303279e66fbecb9c3fd31be8c2137f214d63ffe0586516
output:
  kind: image
  imageOptions:
    diskSize: 3197108224
    diskFormat: raw
    bootloader: dual-boot
  outFormat: .zst
initramfs ready
kernel command line: talos.platform=metal console=ttyS0 init_on_alloc=1 init_on_alloc=1 slab_nomerge slab_nomerge pti=on pti=on consoleblank=0 consoleblank=0 nvme_core.io_timeout=4294967295 nvme_core.io_timeout=4294967295 printk.devkmsg=on printk.devkmsg=on ima_template=ima-ng ima_template=ima-ng ima_appraise=fix ima_appraise=fix ima_hash=sha512 ima_hash=sha512 selinux=1 reboot=k panic=1 talos.shutdown=halt talos.config=metal-iso
UKI ready
disk image ready
output asset path: /out/metal-amd64.raw
compression done: /out/metal-amd64.raw.zst

QEMU:

ps aux | grep qemu
root       13379  238  0.0 190146160 75284 ?     SLl  14:27   0:27 qemu-system-x86_64 -name talos-worker,process=talos-worker -m 178G -smp cpus=36 -cpu host,kvm=off,pdpe1gb=on -machine q35,accel=kvm,smm=on,mem-merge=off -enable-kvm -display none -daemonize -pidfile /root/gpu-passthrough-scripts/.worker/qemu-worker-vm.pid -serial file:/root/gpu-passthrough-scripts/.worker/qemu-worker-vm-console.log -cdrom /root/gpu-passthrough-scripts/.worker/config.iso -device virtio-net-pci,netdev=workertap,mac=02:cf:46:08:42:16 -netdev tap,id=workertap,ifname=workertap,script=no,downscript=no -device virtio-rng-pci -monitor unix:/root/gpu-passthrough-scripts/.worker/worker.monitor,server,nowait -boot order=cn,reboot-timeout=5000 -chardev socket,path=/root/gpu-passthrough-scripts/.worker/worker.sock,server=on,wait=off,id=qga0 -device virtio-serial -device virtserialport,chardev=qga0,name=org.qemu.guest_agent.0 -device i6300esb,id=watchdog0 -watchdog-action pause -drive file=/root/gpu-passthrough-scripts/.worker/worker-main.qcow2,if=virtio,media=disk,index=0,cache=writeback,discard=ignore,format=qcow2 -mem-path /dev/hugepages -mem-prealloc -drive if=pflash,format=raw,readonly=on,file=/root/gpu-passthrough-scripts/.worker/worker-flash.img -drive if=pflash,format=raw,file=/root/gpu-passthrough-scripts/.worker/efi-vars.raw -device pcie-root-port,id=pcie.1,port=0x11,chassis=2,slot=2,bus=pcie.0 -device vfio-pci,host=0000:b3:00.0,bus=pcie.1,multifunction=on,rombar=0 -device pcie-root-port,id=pcie.2,port=0x12,chassis=3,slot=3,bus=pcie.0 -device vfio-pci,host=0000:b3:00.1,bus=pcie.2,rombar=0

MC:

    kubelet:
        image: ghcr.io/siderolabs/kubelet:v1.33.1
        extraArgs:
            cloud-provider: external
            rotate-server-certificates: "true"
        extraConfig:
            allowedUnsafeSysctls:
                - net.core.somaxconn
            cpuManagerPolicy: static
            imageGCHighThresholdPercent: 70
            imageGCLowThresholdPercent: 50
            maxPods: 512
            shutdownGracePeriod: 60s
            topologyManagerPolicy: best-effort
            topologyManagerScope: container
        defaultRuntimeSeccompProfileEnabled: true
        disableManifestsDirectory: true
    network:
        nameservers:
            - 1.1.1.1
            - 1.0.0.1
        kubespan:
            enabled: true
            filters:
                endpoints:
                    - 0.0.0.0/0
                    - ::/0
                    - '!10.0.0.0/8'
                    - '!172.16.0.0/12'
                    - '!192.168.0.0/16'
    install:
        disk: /dev/vda
        extraKernelArgs:
            - cpufreq.default_governor=performance
            - amd_pstate=active
            - intel_idle.max_cstate=0
            - spec_rstack_overflow=microcode
            - mitigations=off
        image: ghcr.io/siderolabs/installer:v1.11.0-alpha.0
        wipe: true
    files:
        - content: "[plugins]\n  [plugins.\"io.containerd.cri.v1.runtime\"]\n    [plugins.\"io.containerd.cri.v1.runtime\".containerd]\n      default_runtime_name = \"nvidia\"        \n"
          permissions: 0o0
          path: /etc/cri/conf.d/20-customization.part
          op: create
    time:
        servers:
            - time.cloudflare.com
    sysctls:
        fs.inotify.max_user_instances: "8192"
        fs.inotify.max_user_watches: "1048576"
        net.core.bpf_jit_harden: "1"
        net.core.netdev_max_backlog: "4096"
        net.core.rmem_max: "67108864"
        net.core.somaxconn: "65535"
        net.core.wmem_max: "67108864"
        net.ipv4.tcp_fin_timeout: "10"
        net.ipv4.tcp_keepalive_intvl: "60"
        net.ipv4.tcp_keepalive_time: "600"
        net.ipv4.tcp_tw_reuse: "1"
        vm.max_map_count: "128000"
    registries:
...
    features:
        rbac: true
        stableHostname: true
        apidCheckExtKeyUsage: true
        diskQuotaSupport: true
        kubePrism:
            enabled: true
            port: 7445
        hostDNS:
            enabled: true
            forwardKubeDNSToHost: false
        imageCache:
            localEnabled: true
    kernel:
        modules:
            - name: nvidia
            - name: nvidia_uvm
            - name: nvidia_drm
            - name: nvidia_modeset

DMESG is attached.

dmesg.log

I tried both Proprietary & OSS NVIDIA extensions and getting the same "Nvidia modules are not found" error in QEMU machine. Could it be related to the GPU model? Similar setup is actually working fine on GPU H100 machines.

[kubebn]

Tested it with the same setup for ubuntu, same qemu flags etc:

sudo apt install ubuntu-drivers-common -y
sudo ubuntu-drivers install --gpgpu nvidia:570-server
sudo apt install nvidia-utils-570-server -y

&&

nvidia-drm.ko: OK
nvidia-modeset.ko: OK
nvidia-peermem.ko: OK
nvidia-uvm.ko: OK
nvidia.ko: OK
---
root@io-gpu-0vk8serzcakum46o:~# nvidia-smi
Tue Jun 17 04:28:08 2025
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 570.133.20             Driver Version: 570.133.20     CUDA Version: 12.8     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA RTX A2000 12GB          Off |   00000000:01:00.0 Off |                  Off |
| 30%   48C    P0             24W /   70W |       0MiB /  12282MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+

[kubebn]

Embarrassingly, it turned out to be a typo in the image path. NVIDIA extensions are working just fine.