CUDA Memory error during RGB_depth frame alignment on Jetson Orin Nano

[Metazet255]

Hello everyone,

I have a computer vision segmentation setup with 3 Realsense cameras connected to a Jetson Orin Nano, which acts as a device to a Windows Host laptop (through SSH). I have simultaneously initilaised 3 Realsense camera streams which fetch the RGB & Depth images & send them to the ORT inferencer when a UDP request is sent, say for a certain camera (like cam ID 1).

I am getting a CUDA 700 illegal memory access error intermittently when switching between cameras & I noticed that the depth image was always black just before the failure. I also observed that when I removed the line which aligns the Depth resolution to fit the RGB resolution (aligned = self.aligns[cam_id].process(frames) ), the CUDA error vanishes. However my predicted calculations regarding distance & angle are wrong because of the misalignment between the RGB & Depth image. I have tried increasing the frame queue size of the Realsense cameras (currently it is 1) as well as reducing the frame rate to 6 fps & adding delays within teh pipeline but the error still occurs. I have updated the CUDA driver & well as the Jetpack & teh Realsense firmware to the most recent version, so I know the problem is not there.

Has anyone else faced this issue & if so, how did you deal with it ?

Camera Specs: 2 Intel RealSense D435I cameras
L4T: Linux4Tegra: GNU/Linux 5.15.148-tegra aarch64
Ubuntu Version: Ubuntu 22.04.5 LTS
Jetpack version: R36 (release), REVISION: 4.4

[MartyG-RealSense]

Hi @Metazet255 As you are using R36 revision 4.4, the JetPack version for that release should be JP 6.2.1. Is that what you have on your Orin Nano, please?

There was a past case in the year 2020 at #7415 where a RealSense user experienced the 700 CUDA error when changing resolution. It occurred intermittently for them too. In their particular situation, they reported at #7415 (comment) that the error never occurred if they disabled the camera's IR emitter.

Ultimately, they resorted to using CPU processing (CUDA = false).

[Metazet255]

Hello @MartyG-RealSense ,

I checked that I am using Jetpack 6.2.1, which is currently installed on my Jetson Orin Nano development kit. I tried disablibg the IR emitter, but teh CUDA 700 error still appears.

Can you tell me more about disabling CUDA & using CPU processing ? I have set teh BUILD_WITH_CUDA as false in teh cmake file, but teh CUDA error still appears.

BUILD_ASAN:BOOL=OFF
BUILD_CSHARP_BINDINGS:BOOL=OFF
BUILD_CV_EXAMPLES:BOOL=OFF
BUILD_CV_KINFU_EXAMPLE:BOOL=OFF
BUILD_DLIB_EXAMPLES:BOOL=OFF
BUILD_EASYLOGGINGPP:BOOL=ON
BUILD_EXAMPLES:BOOL=ON
BUILD_GLSL_EXTENSIONS:BOOL=ON
BUILD_GRAPHICAL_EXAMPLES:BOOL=ON
BUILD_LEGACY_PYBACKEND:BOOL=OFF
BUILD_MATLAB_BINDINGS:BOOL=OFF
BUILD_OPEN3D_EXAMPLES:BOOL=OFF
BUILD_OPENNI2_BINDINGS:BOOL=OFF
BUILD_OPENVINO_EXAMPLES:BOOL=OFF
BUILD_PCL_EXAMPLES:BOOL=OFF
BUILD_PC_STITCHING:BOOL=OFF
BUILD_PYTHON_BINDINGS:BOOL=OFF
BUILD_PYTHON_DOCS:BOOL=OFF
BUILD_RS2_ALL:BOOL=ON
BUILD_SHARED_LIBS:BOOL=ON
BUILD_TOOLS:BOOL=ON
BUILD_UNITY_BINDINGS:BOOL=OFF
BUILD_UNIT_TESTS:BOOL=OFF
BUILD_WITH_CPU_EXTENSIONS:BOOL=ON
BUILD_WITH_CUDA:BOOL=OFF
BUILD_WITH_DDS:BOOL=OFF
BUILD_WITH_OPENMP:BOOL=OFF
BUILD_WITH_STATIC_CRT:BOOL=ON
ENABLE_CCACHE:BOOL=ON
ENABLE_EASYLOGGINGPP_ASYNC:BOOL=ON
ENABLE_SECURITY_FLAGS:BOOL=OFF
USE_EXTERNAL_LZ4:BOOL=OFF

[MartyG-RealSense]

If BUILD_WITH_CUDA is set to false then the librealsense SDK will not have CUDA support enabled and will use the CPU for processing of depth-color alignment and pointclouds instead. So if the error occurs even with CUDA support set to false then it suggests that the error when aligning is not caused by the CUDA support.

If you disable the IR emitter and use the Infrared image (which is like a monochrome RGB image) instead of RGB then the IR image will be perfectly aligned by default with the depth image without having to use alignment, because both depth and Infrared originate from the same sensor.

[Metazet255]

Hello @MartyG-RealSense ,

The model has been trained on bgr images only, so I can't use the IR images in place of teh bgr images unfortunately.

How can I figure out whether teh pointcloud operations use GPU instead of CPU ?

[MartyG-RealSense]

The only way to be certain that the CPU is being used to generate a pointcloud is by setting DBUILD_WITH_CUDA to false.

As you have installed the graphical examples by setting the BUILD_GRAPHICAL_EXAMPLES flag to On, you could try running the rs-pointcloud example pointcloud program (which maps depth and RGB together into a combined image) to see whether the error occurs if you have not tried this program already.

[Metazet255]

Hello @MartyG-RealSense,

The segmentation is running with a single camera & with a batch of images. The 700 CUDA error always happens intermittently when switching between cameras, which leads me to believe that teh relasense SDK & how it acceses the camera streams is the cause.

This is error:
2025-09-29 19:47:38.430437536 [E:onnxruntime:Default, cuda_call.cc:123 CudaCall] CUDA failure 700: an illegal memory access was encountered ; GPU=0 ; hostname=metazet-desktop ; file=/opt/onnxruntime/onnxruntime/core/providers/cuda/gpu_data_transfer.cc ; line=65 ; expr=cudaMemcpyAsync(dst_data, src_data, bytes, cudaMemcpyHostToDevice, static_cast<cudaStream_t>(stream.GetHandle()));
2025-09-29 19:47:38.430547104 [E:onnxruntime:Default, cuda_call.cc:123 CudaCall] CUDA failure 700: an illegal memory access was encountered ; GPU=0 ; hostname=metazet-desktop ; file=/opt/onnxruntime/onnxruntime/core/providers/cuda/cuda_execution_provider.cc ; line=446 ; expr=cudaStreamSynchronize(static_cast<cudaStream_t>(stream_));
[run_model] Error during model processing: [ONNXRuntimeError] : 1 : FAIL : CUDA failure 700: an illegal memory access was encountered ; GPU=0 ; hostname=metazet-desktop ; file=/opt/onnxruntime/onnxruntime/core/providers/cuda/gpu_data_transfer.cc ; line=65 ; expr=cudaMemcpyAsync(dst_data, src_data, bytes, cudaMemcpyHostToDevice, static_cast<cudaStream_t>(stream.GetHandle()));

Do you have any other suggestions I could try out ? Is there a way to use jtops to visualise GPU usage when the program runs ?

[MartyG-RealSense]

When you are switching cameras, do you mean that you are unplugging one camera and plugging another camera in whilst the program is running, or do you have all the cameras attached to the computer at the same time and have a method of switching between which camera is currently active, please?

Once a camera is unplugged, the librealsense SDK only allows a 5 second period to reconnect a camera before the pipeline will time-out.

If you are unplugging and inserting different cameras, it may be a good idea to use multiple camera code like that of the rs-multicam SDK example program if you are not using multicam code already, so that the program can automatically add and remove cameras from its list of active cameras as they are inserted and unplugged.

If I have misunderstood how your camera switching mechanism works then I apologize and ask that you please provide further information about how the switch takes place. Thanks!

[Metazet255]

Hello @MartyG-RealSense

By switching cameras, I mean UDP requests from a different camera, & not physically disconnecting & reconnecting cameras. I run 4 camera pipleines in parallel & process images from whichever cam ID is requested in teh UDP request. This goes on in an infinite loop with a new random UDP request being generated every 5 seconds until I exit the program with Ctrl + Z.

[MartyG-RealSense]

I am slightly familiar with UDP requests but my knowledge of them is limited, unfortunately.

Running in an infinite loop sounds as though it could possibly lead to a memory leak, causing the program to become more unstable as the computer's available memory is consumed over time. You could check for a memory leak in htop by observing the available memory and see if it is falling whilst the program runs.

[Metazet255]

The inifnite loop simulates real world conditions where the robot is expected to detect tuberails in a greenhouse autonomously & perform row shifts continuously. Analysis with teh help of chatGPT tells me to use teh CPU for depth-color alignment & the GPU for running teh inference. I am able to avoid teh CUDA errors when I use teh CPU for both alignment & inference (this takes longer time), but I do not know how to hide CUDA from teh realsense functions (including align.process(frames)) only. Do you know how it is possible to do so ?

As to your response about the memory leak, if this was the case, then the program would always fail after a certain number of image requests. But this is not the case, as the program sometimes runs succesfully for a long time, but abruptly ends sometimes as well.

[MartyG-RealSense]

If librealsense is built from source code on Jetson with CMake without the flag DBUILD_WITH_CUDA then RealSense should completely ignore the presence of CUDA and use the CPU for alignment and point clouds.

If you would prefer to have GPU assistance for faster processing then if you program in the C++ language there would be the option of using GLSL to GPU-accelerate point clouds and alignment instead of CUDA. #7611 (comment) has more information about this.