CUDA error = 999

[delenius]

I am following the instructions for running nvblox via isaac-ros-common (in the docker environment).

I get this error:

$ ros2 launch nvblox_examples_bringup isaac_sim_example.launch.py \
rosbag:=${ISAAC_ROS_WS}/isaac_ros_assets/isaac_ros_nvblox/quickstart \
navigation:=False
[INFO] [launch]: All log files can be found below /home/admin/.ros/log/2025-03-20-15-18-21-661341-dl-1269
[INFO] [launch]: Default logging verbosity is set to INFO
Using container type: component_container_isolated, with arguments: ['--ros-args', '--log-level', <launch.substitutions.launch_configuration.LaunchConfiguration object at 0x7f01e1e485b0>]
[INFO] [launch.user]: Adding nodes [ /nvblox_node ] to container nvblox_container.
[INFO] [launch.user]: Starting nvblox with the 'isaac_sim' camera in 'static' mode.
[INFO] [launch.user]: Now starting playback of rosbag "/workspaces/isaac_ros-dev/isaac_ros_assets/isaac_ros_nvblox/quickstart" with command: ros2bagplay/workspaces/isaac_ros-dev/isaac_ros_assets/isaac_ros_nvblox/quickstart--clock
[INFO] [component_container_isolated-1]: process started with pid [1281]
[INFO] [ros2-2]: process started with pid [1283]
[INFO] [rviz2-3]: process started with pid [1285]
[rviz2-3] QStandardPaths: XDG_RUNTIME_DIR not set, defaulting to '/tmp/runtime-admin'
[ros2-2] stdin is not a terminal device. Keyboard handling disabled.[INFO] [1742509101.817913998] [rosbag2_storage]: Opened database '/workspaces/isaac_ros-dev/isaac_ros_assets/isaac_ros_nvblox/quickstart/rosbag2_2024_04_04-15_44_33_0.db3' for READ_ONLY.
[ros2-2] [INFO] [1742509101.817942257] [rosbag2_player]: Set rate to 1
[ros2-2] [INFO] [1742509101.821607188] [rosbag2_player]: Adding keyboard callbacks.
[ros2-2] [INFO] [1742509101.821624033] [rosbag2_player]: Press SPACE for Pause/Resume
[ros2-2] [INFO] [1742509101.821627971] [rosbag2_player]: Press CURSOR_RIGHT for Play Next Message
[ros2-2] [INFO] [1742509101.821630767] [rosbag2_player]: Press CURSOR_UP for Increase Rate 10%
[ros2-2] [INFO] [1742509101.821633042] [rosbag2_player]: Press CURSOR_DOWN for Decrease Rate 10%
[ros2-2] [INFO] [1742509101.821849245] [rosbag2_storage]: Opened database '/workspaces/isaac_ros-dev/isaac_ros_assets/isaac_ros_nvblox/quickstart/rosbag2_2024_04_04-15_44_33_0.db3' for READ_ONLY.
[rviz2-3] [INFO] [1742509101.927754477] [rviz2]: Stereo is NOT SUPPORTED
[rviz2-3] [INFO] [1742509101.927814022] [rviz2]: OpenGl version: 4.6 (GLSL 4.6)
[component_container_isolated-1] [INFO] [1742509101.958086526] [nvblox_container]: Load Library: /opt/ros/humble/lib/libnvblox_ros_lib.so
[component_container_isolated-1] [INFO] [1742509101.971499767] [nvblox_container]: Found class: rclcpp_components::NodeFactoryTemplate<nvblox::NvbloxNode>
[component_container_isolated-1] [INFO] [1742509101.971530862] [nvblox_container]: Instantiate class: rclcpp_components::NodeFactoryTemplate<nvblox::NvbloxNode>
[component_container_isolated-1] CUDA error = 999 at ./nvblox_core/nvblox/src/core/cuda_stream.cpp:36 'cudaStreamCreateWithFlags(&stream_, flags)'. Error string: unknown error.
[rviz2-3] [INFO] [1742509101.988399215] [rviz2]: Stereo is NOT SUPPORTED
[rviz2-3] [INFO] [1742509102.044893815] [rviz2]: Stereo is NOT SUPPORTED
[ERROR] [component_container_isolated-1]: process has died [pid 1281, exit code 99, cmd '/opt/ros/humble/lib/rclcpp_components/component_container_isolated --ros-args --log-level info --ros-args -r __node:=nvblox_container -p use_sim_time:=True'].
[INFO] [launch]: process[component_container_isolated-1] was required: shutting down launched system
[INFO] [rviz2-3]: sending signal 'SIGINT' to process[rviz2-3]
[INFO] [ros2-2]: sending signal 'SIGINT' to process[ros2-2]
[rviz2-3] [INFO] [1742509102.077320483] [rviz2]: Stereo is NOT SUPPORTED
[rviz2-3] [INFO] [1742509102.702857125] [rclcpp]: signal_handler(signum=2)
[ros2-2] [INFO] [1742509102.703029567] [rclcpp]: signal_handler(signum=2)
[INFO] [ros2-2]: process has finished cleanly [pid 1283]
[WARNING] [launch_ros.actions.load_composable_nodes]: Abandoning wait for the 'nvblox_container/_container/load_node' service response, due to shutdown.
[ERROR] [rviz2-3]: process has died [pid 1285, exit code -11, cmd '/opt/ros/humble/lib/rviz2/rviz2 -d /opt/ros/humble/share/nvblox_examples_bringup/config/visualization/isaac_sim_example.rviz --ros-args -p use_sim_time:=True'].
admin@dl:/workspaces/isaac_ros-dev$ 
exit

nvidia toolkit seems to work correctly, e.g. below is the output of nvidia-smi inside the docker container, and some environment variables:

$ nvidia-smi 
Thu Mar 20 15:25:14 2025       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 570.86.16              Driver Version: 570.86.16      CUDA Version: 12.8     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce RTX 5090        Off |   00000000:01:00.0  On |                  N/A |
|  0%   39C    P8             24W /  575W |    5581MiB /  32607MiB |      4%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
                                                                                         
+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+

$ env|grep NVIDIA
NVIDIA_VISIBLE_DEVICES=all
NVIDIA_REQUIRE_CUDA=cuda>=9.0
NVIDIA_DRIVER_CAPABILITIES=all
NVIDIA_PRODUCT_NAME=Triton Server
NVIDIA_TRITON_SERVER_VERSION=24.08
NVIDIA_BUILD_ID=107631419
NVIDIA_REQUIRE_JETPACK_HOST_MOUNTS=

$ env|grep CUDA
NVIDIA_REQUIRE_CUDA=cuda>=9.0
CUDA_CACHE_DISABLE=1
_CUDA_COMPAT_STATUS=System has unsupported display driver / cuda driver combination (CUDA_ERROR_SYSTEM_DRIVER_MISMATCH) cuInit()=803
CUDA_VERSION=12.6.0.022
CUDA_DRIVER_VERSION=560.35.03
_CUDA_COMPAT_PATH=/usr/local/cuda/compat

[jaiveersinghNV]

The 999 error code is a catch-all for a lot of potential CUDA system configuration errors, so it's hard to give you a precise answer about what's going wrong.

I notice that there seems to be a mismatch between the driver version reported with nvidia-smi (570) and the value of your CUDA_DRIVER_VERSION environment variable (560). You might be in a bad driver installation state where your machine has both modules loaded into the kernel, for some reason.

Have you already tried:

• restarting the machine
• reinstalling drivers
• on a separate computer/with a separate GPU
• disabling your CPU's integrated graphics, if present