GPU-operator on Talos

[kubebn]

Hi, I’m trying to give the GPU operator another try on Talos, but it seems to be failing due to validator init containers. I noticed that the folder can now be specified in the Helm chart here: https://github.com/NVIDIA/gpu-operator/blob/main/deployments/gpu-operator/values.yaml#L33.

What would be the correct path for the NVIDIA drivers on Talos? Would it be /proc/driver/nvidia/ ?

cat values-talos.yaml
sandboxWorkloads:
  enabled: true
driver:
  enabled: false
devicePlugin:
  enabled: false
toolkit:
  enabled: false
---
k get po
NAME                                                          READY   STATUS     RESTARTS   AGE
gpu-feature-discovery-n7kth                                   0/1     Init:0/1   0          5m33s
gpu-operator-666bbffcd-hnv5d                                  1/1     Running    0          6m11s
gpu-operator-node-feature-discovery-gc-7c7f68d5f4-h9f52       1/1     Running    0          6m11s
gpu-operator-node-feature-discovery-master-58588c6967-49p9w   1/1     Running    0          6m11s
gpu-operator-node-feature-discovery-worker-fmv87              1/1     Running    0          6m12s
gpu-operator-node-feature-discovery-worker-lglwz              1/1     Running    0          6m12s
gpu-operator-node-feature-discovery-worker-qgq4v              1/1     Running    0          6m12s
gpu-operator-node-feature-discovery-worker-v29d7              1/1     Running    0          6m12s
gpu-operator-node-feature-discovery-worker-z22dr              1/1     Running    0          6m12s
nvidia-dcgm-exporter-gc2st                                    0/1     Init:0/1   0          5m33s
nvidia-operator-validator-rdqs8                               0/1     Init:0/3   0          5m34s
---
k logs nvidia-operator-validator-rdqs8 -c driver-validation
time="2025-06-04T10:17:21Z" level=info msg="version: b5479aaa-amd64, commit: b5479aa"
time="2025-06-04T10:17:21Z" level=info msg="Attempting to validate a pre-installed driver on the host"
time="2025-06-04T10:17:22Z" level=info msg="Attempting to validate a driver container installation"
time="2025-06-04T10:17:22Z" level=warning msg="failed to validate the driver, retrying after 5 seconds\n"
time="2025-06-04T10:17:27Z" level=info msg="Attempting to validate a driver container installation"
time="2025-06-04T10:17:27Z" level=warning msg="failed to validate the driver, retrying after 5 seconds\n"
time="2025-06-04T10:17:32Z" level=info msg="Attempting to validate a driver container installation"
time="2025-06-04T10:17:32Z" level=warning msg="failed to validate the driver, retrying after 5 seconds\n"
time="2025-06-04T10:17:37Z" level=info msg="Attempting to validate a driver container installation"
time="2025-06-04T10:17:37Z" level=warning msg="failed to validate the driver, retrying after 5 seconds\n"
---
talosctl get extensions -n 10.0.2.15
NODE        NAMESPACE   TYPE              ID            VERSION   NAME                                  VERSION
10.0.2.15   runtime     ExtensionStatus   0             1         nvidia-container-toolkit-production   570.124.06-v1.17.5
10.0.2.15   runtime     ExtensionStatus   1             1         nonfree-kmod-nvidia-production        570.124.06-v1.10.0
10.0.2.15   runtime     ExtensionStatus   modules.dep   1         modules.dep                           6.12.25-talos
---
talosctl read /proc/modules -n 10.0.2.15
nvidia_uvm 2220032 0 - Live 0xffffffffc64e4000 (PO)
nvidia_drm 147456 0 - Live 0xffffffffc6464000 (PO)
nvidia_modeset 1798144 1 nvidia_drm, Live 0xffffffffc6280000 (PO)
drm_ttm_helper 16384 1 nvidia_drm, Live 0xffffffffc64dd000
ttm 106496 1 drm_ttm_helper, Live 0xffffffffc64b5000
nvidia 96849920 7 nvidia_uvm,nvidia_modeset, Live 0xffffffffc061f000 (PO)
e1000 155648 0 - Live 0xffffffffc05f5000
---
talosctl read /proc/driver/nvidia/version -n 10.0.2.15
NVRM version: NVIDIA UNIX x86_64 Kernel Module  570.124.06  Wed Feb 26 02:12:04 UTC 2025
GCC version:  gcc version 14.2.0 (GCC)

[smira]

The best source of current effort is siderolabs/extensions#476, and this is done by awesome @jfroy.

[jfroy]

I have not tried the GPU Operator on Talos with the SideroLabs extensions in some time. As you have done, you need to disable driver and toolkit installation/management.

You can try specifying "/usr/local/glibc" as the driver path (this is where the SideroLabs extensions installs the userspace components), but if I remember correctly this will still fail.

[kubebn]

I have not tried the GPU Operator on Talos with the SideroLabs extensions in some time. As you have done, you need to disable driver and toolkit installation/management.

You can try specifying "/usr/local/glibc" as the driver path (this is where the SideroLabs extensions installs the userspace components), but if I remember correctly this will still fail.

Yeah, it still fails.

cat values-talos.yaml
driver:
  enabled: false
devicePlugin:
  enabled: true
toolkit:
  enabled: false
hostPaths:
  rootFS: "/"
  driverInstallDir: "/usr/local/glibc"

k get po
k lNAME                                                          READY   STATUS     RESTARTS   AGE
gpu-feature-discovery-mz6z6                                   0/1     Init:0/1   0          3m16s
gpu-operator-666bbffcd-4n7cl                                  1/1     Running    0          3m40s
gpu-operator-node-feature-discovery-gc-7c7f68d5f4-7cns2       1/1     Running    0          3m40s
gpu-operator-node-feature-discovery-master-58588c6967-8hp4t   1/1     Running    0          3m40s
gpu-operator-node-feature-discovery-worker-27gk5              1/1     Running    0          3m40s
gpu-operator-node-feature-discovery-worker-5lbxm              1/1     Running    0          3m40s
gpu-operator-node-feature-discovery-worker-864zr              1/1     Running    0          3m40s
gpu-operator-node-feature-discovery-worker-8xhrt              1/1     Running    0          3m40s
gpu-operator-node-feature-discovery-worker-t7s4t              1/1     Running    0          3m40s
nvidia-dcgm-exporter-d4dl8                                    0/1     Init:0/1   0          3m17s
nvidia-device-plugin-daemonset-w9lkz                          0/1     Init:0/1   0          3m17s
nvidia-operator-validator-5krjf                               0/1     Init:0/4   0          3m18s
---
k logs nvidia-operator-validator-5krjf -c driver-validation
time="2025-06-09T10:11:59Z" level=info msg="version: b5479aaa-amd64, commit: b5479aa"
time="2025-06-09T10:11:59Z" level=info msg="Attempting to validate a pre-installed driver on the host"
time="2025-06-09T10:12:00Z" level=info msg="Attempting to validate a driver container installation"
time="2025-06-09T10:12:00Z" level=warning msg="failed to validate the driver, retrying after 5 seconds\n"
time="2025-06-09T10:12:05Z" level=info msg="Attempting to validate a driver container installation"
time="2025-06-09T10:12:05Z" level=warning msg="failed to validate the driver, retrying after 5 seconds\n"
time="2025-06-09T10:12:10Z" level=info msg="Attempting to validate a driver container installation"
time="2025-06-09T10:12:10Z" level=warning msg="failed to validate the driver, retrying after 5 seconds\n"
time="2025-06-09T10:12:15Z" level=info msg="Attempting to validate a driver container installation"