OCP 4.18.22: nvidia-operator-validator pod in Init:CreateContainerError - error executing hook `/usr/local/nvidia/toolkit/nvidia-container-runtime-hook` (exit code: 1)

[wabouhamad]

Important Note: NVIDIA AI Enterprise customers can get support from NVIDIA Enterprise support. Please open a case here.

Describe the bug
When deploying the GPU operator v25.3.2 on OpenShift Container Plartform version 4.18.22, deploying the default clusterpolicy fails (notReady). The driver daemoset container completes buidling the drivers but the nvidia-operator-validator pod is in Init:CreateContainerError. oc describe pod on on the validator shows:

kubelet            Started container toolkit-validation
  Warning  Failed          4m49s (x4 over 5m59s)   kubelet            Error: container create failed: error executing hook `/usr/local/nvidia/toolkit/nvidia-container-runtime-hook` (exit code: 1)
  Normal   Pulled          3m31s (x7 over 6m13s)   kubelet            Container image "nvcr.io/nvidia/cloud-native/gpu-operator-validator@sha256:e183dc07e5889bd9e269c320ffad7f61df655f57ecc3aa158c4929e74528420a" already present on machine
  Warning  BackOff         2m12s (x14 over 6m11s)  kubelet            Back-off restarting failed container toolkit-validation in pod nvidia-operator-validator-kkpjf_nvidia-gpu-operator(3492f33c-e20f-47ea-9693-dc3304fefd84)

To Reproduce

• Deploy Single Node OCP 4.18.22 on AWS.
• Create a g4dn.xlarge (T4 GPU) machineset to add a worker node to this single node
• Deploy NFD operator and nodefeaturediscovery CR instance to label the worker nodes
• Deploy GPU Operator lasted v25.3.2
• Deploy clusterpolicy from CSV alam-examples
• after 10 mins check if GPU stack was deployed successfully, run cmd: oc get pods -n nvidia-gpu-operator

Expected behavior
GPU operator stack successfully deployed in nvidia-gpu-operator namespace such as:

$ oc get pods -n nvidia-gpu-operator
NAME                                                  READY   STATUS      RESTARTS   AGE
gpu-feature-discovery-ssx5r                           1/1     Running     0          29h
gpu-operator-5fcc456c94-wdq8x                         1/1     Running     0          38h
nvidia-container-toolkit-daemonset-tc69k              1/1     Running     0          38h
nvidia-cuda-validator-kzfjn                           0/1     Completed   0          54m
nvidia-dcgm-exporter-d2qrj                            1/1     Running     0          38h
nvidia-dcgm-x6b6v                                     1/1     Running     0          38h
nvidia-device-plugin-daemonset-92hr4                  1/1     Running     0          38h
nvidia-driver-daemonset-418.94.202508060022-0-wnrz2   2/2     Running     0          38h
nvidia-node-status-exporter-pr2zj                     1/1     Running     0          38h
nvidia-operator-validator-v4j75                       1/1     Running     0          29h

Environment (please provide the following information):

• GPU Operator Version: v25.3.2
• OS: RHCOS 418.94.202508060022-0 - based on RHEL 9.4
• Kernel Version: 5.14.0-427.81.1.el9_4.x86_64
• Container Runtime Version: crio version 1.31.11-2.rhaos4.18.git65ec77a.el9
• Kubernetes Distro and Version: OpenShift running kubernetes v1.31.11

Information to attach (optional if deemed irrelevant)

must-gather logs:
https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/test-platform-results/pr-logs/pull/rh-ecosystem-edge_nvidia-ci/268/pull-ci-rh-ecosystem-edge-nvidia-ci-main-4.18-stable-nvidia-gpu-operator-e2e-25-3-x/1954619565199593472/artifacts/nvidia-gpu-operator-e2e-25-3-x/gpu-operator-e2e/artifacts/gpu-operator-tests-must-gather/gpu-must-gather/

Nvidia-smi output form driver container daemonset:

sh-5.1# nvidia-smi
Tue Aug 12 03:27:29 2025       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 570.148.08             Driver Version: 570.148.08     CUDA Version: 12.8     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  Tesla T4                       On  |   00000000:00:1E.0 Off |                    0 |
| N/A   32C    P8             13W /   70W |       0MiB /  15360MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
                                                                                         
+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+
sh-5.1# 

Additional notes:

Workaround that resolved this issue:

containers/podman#16101

Added no-cgroups = true to the GPU worker node /var/usrlocal/nvidia/toolkit/.config/nvidia-container-runtime/config.toml file, and the GPU stack is now deployed successfully on OCP 4.18.22.

[nvidia-container-cli]
  environment = []
  ldconfig = "@/run/nvidia/driver/sbin/ldconfig"
  load-kmods = true
  path = "/usr/local/nvidia/toolkit/nvidia-container-cli"
  root = "/run/nvidia/driver"
  no-cgroups = true

[kannon92]

The main difference from 4.18.21 to 4.18.22 was a crun bump from 1.21 to 1.23.

I was able to dig a bit more into this (ref: https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/test-platform-results/pr-logs/pull/rh-ecosystem-edge_nvidia-ci/268/pull-ci-rh-ecosystem-edge-nvidia-ci-main-4.18-stable-nvidia-gpu-operator-e2e-25-3-x/1954619565199593472/artifacts/nvidia-gpu-operator-e2e-25-3-x/).

Pulling the worker journal logs:

Aug 10 20:19:32.641438 ip-10-0-33-221 systemd[1]: Started crio-conmon-6ba7d8637fb000e431e1a922daad7228cc45474709761a0e6d721db9a38a44e8.scope.
Aug 10 20:19:32.649487 ip-10-0-33-221 systemd[1]: crio-6ba7d8637fb000e431e1a922daad7228cc45474709761a0e6d721db9a38a44e8.scope: bpf-foreign: Failed to create foreign BPF program: Permission denied
Aug 10 20:19:32.649496 ip-10-0-33-221 systemd[1]: crio-6ba7d8637fb000e431e1a922daad7228cc45474709761a0e6d721db9a38a44e8.scope: bpf-foreign: Failed to prepare foreign BPF hashmap: Permission denied
Aug 10 20:19:32.657397 ip-10-0-33-221 systemd[1]: Started libcrun container.
Aug 10 20:19:32.711941 ip-10-0-33-221 systemd[1]: crio-6ba7d8637fb000e431e1a922daad7228cc45474709761a0e6d721db9a38a44e8.scope: Deactivated successfully.
Aug 10 20:19:32.713855 ip-10-0-33-221 conmon[40135]: conmon 6ba7d8637fb000e431e1 <nwarn>: runtime stderr: error executing hook `/usr/local/nvidia/toolkit/nvidia-container-runtime-hook` (exit code: 1)
Aug 10 20:19:32.713878 ip-10-0-33-221 conmon[40135]: conmon 6ba7d8637fb000e431e1 <error>: Failed to create container: exit status 255
Aug 10 20:19:32.714409 ip-10-0-33-221 crio[2502]: time="2025-08-10 20:19:32.713964541Z" level=error msg="Container creation error: error executing hook `/usr/local/nvidia/toolkit/nvidia-container-runtime-hook` (exit code: 1)\n" id=8c3bb895-1c3a-4a6f-a8b3-823950419b98 name=/runtime.v1.RuntimeService/CreateContainer
Aug 10 20:19:32.714594 ip-10-0-33-221 systemd[1]: crio-conmon-6ba7d8637fb000e431e1a922daad7228cc45474709761a0e6d721db9a38a44e8.scope: Deactivated successfully.
Aug 10 20:19:32.716368 ip-10-0-33-221 crio[2502]: time="2025-08-10 20:19:32.716328833Z" level=info msg="createCtr: deleting container ID 6ba7d8637fb000e431e1a922daad7228cc45474709761a0e6d721db9a38a44e8 from idIndex" id=8c3bb895-1c3a-4a6f-a8b3-823950419b98 name=/runtime.v1.RuntimeService/CreateContainer
Aug 10 20:19:32.716515 ip-10-0-33-221 crio[2502]: time="2025-08-10 20:19:32.716374881Z" level=info msg="createCtr: removing container 6ba7d8637fb000e431e1a922daad7228cc45474709761a0e6d721db9a38a44e8" id=8c3bb895-1c3a-4a6f-a8b3-823950419b98 name=/runtime.v1.RuntimeService/CreateContainer
Aug 10 20:19:32.716515 ip-10-0-33-221 crio[2502]: time="2025-08-10 20:19:32.716411999Z" level=info msg="createCtr: deleting container 6ba7d8637fb000e431e1a922daad7228cc45474709761a0e6d721db9a38a44e8 from storage" id=8c3bb895-1c3a-4a6f-a8b3-823950419b98 name=/runtime.v1.RuntimeService/CreateContainer
Aug 10 20:19:32.730717 ip-10-0-33-221 crio[2502]: time="2025-08-10 20:19:32.730676776Z" level=info msg="createCtr: releasing container name k8s_toolkit-validation_nvidia-operator-validator-kkpjf_nvidia-gpu-operator_3492f33c-e20f-47ea-9693-dc3304fefd84_2" id=8c3bb895-1c3a-4a6f-a8b3-823950419b98 name=/runtime.v1.RuntimeService/CreateContainer
Aug 10 20:19:32.730990 ip-10-0-33-221 kubenswrapper[2561]: E0810 20:19:32.730956    2561 log.go:32] "CreateContainer in sandbox from runtime service failed" err=<
Aug 10 20:19:32.730990 ip-10-0-33-221 kubenswrapper[2561]:         rpc error: code = Unknown desc = container create failed: error executing hook `/usr/local/nvidia/toolkit/nvidia-container-runtime-hook` (exit code: 1)
Aug 10 20:19:32.730990 ip-10-0-33-221 kubenswrapper[2561]:  > podSandboxID="3f467232cfe54b8232419b143ba0f856120633bc841405d8847e64cff7d8b7f2"
Aug 10 20:19:32.731462 ip-10-0-33-221 kubenswrapper[2561]: E0810 20:19:32.731082    2561 kuberuntime_manager.go:1274] "Unhandled Error" err=<
Aug 10 20:19:32.731462 ip-10-0-33-221 kubenswrapper[2561]:         init container &Container{Name:toolkit-validation,Image:nvcr.io/nvidia/cloud-native/gpu-operator-validator@sha256:e183dc07e5889bd9e269c320ffad7f61df655f57ecc3aa158c4929e74528420a,Command:[sh -c],Args:[nvidia-validator],WorkingDir:,Ports:[]ContainerPort{},Env:[]EnvVar{EnvVar{Name:NVIDIA_VISIBLE_DEVICES,Value:all,ValueFrom:nil,},EnvVar{Name:WITH_WAIT,Value:false,ValueFrom:nil,},EnvVar{Name:COMPONENT,Value:toolkit,ValueFrom:nil,},},Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{},Claims:[]ResourceClaim{},},VolumeMounts:[]VolumeMount{VolumeMount{Name:run-nvidia-validations,ReadOnly:false,MountPath:/run/nvidia/validations,SubPath:,MountPropagation:*Bidirectional,SubPathExpr:,RecursiveReadOnly:nil,},VolumeMount{Name:kube-api-access-rqmsz,ReadOnly:true,MountPath:/var/run/secrets/kubernetes.io/serviceaccount,SubPath:,MountPropagation:nil,SubPathExpr:,RecursiveReadOnly:nil,},},LivenessProbe:nil,ReadinessProbe:nil,Lifecycle:nil,TerminationMessagePath:/dev/termination-log,ImagePullPolicy:IfNotPresent,SecurityContext:&SecurityContext{Capabilities:nil,Privileged:*true,SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,ReadOnlyRootFilesystem:nil,AllowPrivilegeEscalation:nil,RunAsGroup:nil,ProcMount:nil,WindowsOptions:nil,SeccompProfile:nil,AppArmorProfile:nil,},Stdin:false,StdinOnce:false,TTY:false,EnvFrom:[]EnvFromSource{},TerminationMessagePolicy:File,VolumeDevices:[]VolumeDevice{},StartupProbe:nil,ResizePolicy:[]ContainerResizePolicy{},RestartPolicy:nil,} start failed in pod nvidia-operator-validator-kkpjf_nvidia-gpu-operator(3492f33c-e20f-47ea-9693-dc3304fefd84): CreateContainerError: container create failed: error executing hook `/usr/local/nvidia/toolkit/nvidia-container-runtime-hook` (exit code: 1)
Aug 10 20:19:32.731462 ip-10-0-33-221 kubenswrapper[2561]:  > logger="UnhandledError"

[tariq1890]

thank you @kannon92 ! @wabouhamad Can we install crun 1.21 on the OCP 4.18.22 environment and re-run GPU Operator? I'd like to confirm that it is the crun bump that is leading to this failure

[kannon92]

I did have a coworker who was able to confirm that if they switch the OCI runtime to runc, everything works fine.

I don't think we have yet a RCA on crun.

[averyngo34]

I can confirm that 4.18.22 is also affected by this and the workaround in containers/podman#16101 works

[azzid]

I can confirm that 4.18.21 is also affected by this and the workaround in containers/podman#16101 works

Are you referring to the set no-cgroups = true workaround? Where you able to make it persist through node reboots?

[averyngo34]

Are you referring to the set no-cgroups = true workaround?

Yes.

Where you able to make it persist through node reboots?

This can be done using deamonset but this exec into kernel is not a safe practice

[evansl-uk]

This has impacted some workloads running in Red Hat OpenShift on IBM Cloud too.

[omaslara]

Thank you @wabouhamad!

I'm experiencing the same issue, and my environment is similar:

• Single Node OCP 4.18.22 on AWS.
• EC2 instance type g5.2xlarge (1 x A10G GPU)
• NFD operator and nodefeaturediscovery CR instance to label the worker nodes
• NVIDIA GPU Operator v25.3.2

% oc get pods
NAME                                                  READY   STATUS                  RESTARTS       AGE
gpu-feature-discovery-vvqn6                           0/1     Init:0/1                0              7m26s
gpu-operator-6b767d6dd8-xftp5                         1/1     Running                 0              133m
nvidia-container-toolkit-daemonset-66rx9              1/1     Running                 0              7m26s
nvidia-dcgm-5q6xw                                     0/1     Init:0/1                0              7m26s
nvidia-dcgm-exporter-qwjn9                            0/1     Init:0/2                0              7m26s
nvidia-device-plugin-daemonset-kztqd                  0/1     Init:0/1                0              7m26s
nvidia-driver-daemonset-418.94.202508060022-0-5tzxg   2/2     Running                 0              8m7s
nvidia-node-status-exporter-k8fnq                     1/1     Running                 0              8m3s
nvidia-operator-validator-zgmq8                       0/1     Init:CrashLoopBackOff   2 (4m1s ago)   7m26s

Events:

5m25s       Warning   Failed                pod/nvidia-operator-validator-zgmq8                       Error: container create failed: error executing hook `/usr/local/nvidia/toolkit/nvidia-container-runtime-hook` (exit code: 1

I will test the suggested workaround.

[pdfruth]

We are impacted by this as well.
We have an OpenShift v4.18.22 ROKS cluster running on IBM Cloud.

How are you able to work around this?
As mentioned in the OP, the suggested workaround is;

Added no-cgroups = true to the GPU worker node /var/usrlocal/nvidia/toolkit/.config/nvidia-container-runtime/config.toml file, and the GPU stack is now deployed successfully on OCP 4.18.22.

We've edited the config.toml file... but the operator pods are still stuck in Init: status.
Is there anything else that must be done, after editing the file, to enable the operator (and the associated cluster policy) to pick up the changes?

Anyone have any idea when a more permanent solution will come?

[GRuuuuu]

@pdfruth Try delete gpu validator pod

[pdfruth]

@GRuuuuu Thank you for the suggestion. Deleting the nvidia-operator-validator pod, after editing the config.toml file, appears to have worked.

Until such time as someone can get to the root cause, and provide a more durable resolution... How do we protect against the edit of config.toml being reversed after a node reboot (or an openshift update).

[tariq1890]

Please update your OCP 4.18 cluster to version 4.18.24. The root cause of this issue was a change in crun that was packaged in Openshift 4.18.22. A new version of crun has been released and packaged into OCP 4.18.24