IRuntime::deserializeCudaEngine Error [arm64]

[rt-2pm2]

Dear All,

I noticed that when trying to run Autoware on Arm architecture some components are failing. Precisely, it seems related to tensor-rt failing to load the neural network models downloaded with the artifacts.

Here one of the extract from the autoware log:

[component_container_mt-62] [I] [TRT] [MemUsageChange] Init CUDA: CPU +0, GPU +0, now: CPU 41, GPU 263 (MiB)
 [component_container_mt-62] [I] [TRT] [MemUsageChange] Init builder kernel library: CPU +2195, GPU +1669, now: CPU 2392, GPU 1932 (MiB)
 [component_container_mt-62] [I] [TRT] ----------------------------------------------------------------
 [component_container_mt-62] [I] [TRT] Input filename:   /home/adlink/autoware_data/traffic_light_classifier/traffic_light_classifier_mobilenetv2_batch_6.onnx
 [component_container_mt-62] [I] [TRT] ONNX IR version:  0.0.8
 [component_container_mt-62] [I] [TRT] Opset version:    11
 [component_container_mt-62] [I] [TRT] Producer name:    pytorch
 [component_container_mt-62] [I] [TRT] Producer version: 1.13.1
 [component_container_mt-62] [I] [TRT] Domain:
 [component_container_mt-62] [I] [TRT] Model version:    0
 [component_container_mt-62] [I] [TRT] Doc string:
 [component_container_mt-62] [I] [TRT] ----------------------------------------------------------------
 [component_container_mt-62] [W] [TRT] Engine is not initialized. Retrieving data from network
 [component_container_mt-62] [W] [TRT] Engine is not initialized. Retrieving data from network
 [component_container_mt-62] [I] [TRT] Setting optimization profile for tensor: input {min [6, 3, 224, 224], opt [6, 3, 224, 224], max [6, 3, 224, 224]}
 [component_container_mt-62] [I] [TRT] Loading engine
 [component_container_mt-62] [I] [TRT] Loaded engine size: 7 MiB
 [component_container_mt-62] [E] [TRT] IRuntime::deserializeCudaEngine: Error Code 1: Serialization (Serialization assertion stdVersionRead == kSERIALIZATION_VERSION failed.Version tag does not match. Note: Current Version: 239, Serialized Engine Version: 236)

I don't see those errors when I run on x86 and I noticed that in the configuration files (arm64.env and amd64.env), the tensor-rt version is different between arm and x86 architectures.

My machine is an Ampere Altra Development Platform with Ubuntu 22.04 and ROS-Humble. I put here some extra info about the nvidia-related libraries installed on the machine:

libnvidia-cfg1-570/unknown,now 570.133.20-0ubuntu1 arm64 [installed,automatic]
libnvidia-common-570/unknown,now 570.133.20-0ubuntu1 all [installed,automatic]
libnvidia-compute-570/unknown,now 570.133.20-0ubuntu1 arm64 [installed,automatic]
libnvidia-decode-570/unknown,now 570.133.20-0ubuntu1 arm64 [installed,automatic]
libnvidia-encode-570/unknown,now 570.133.20-0ubuntu1 arm64 [installed,automatic]
libnvidia-extra-570/unknown,now 570.133.20-0ubuntu1 arm64 [installed,automatic]
libnvidia-fbc1-570/unknown,now 570.133.20-0ubuntu1 arm64 [installed,automatic]
libnvidia-gl-570/unknown,now 570.133.20-0ubuntu1 arm64 [installed,automatic]
nvidia-compute-utils-570/unknown,now 570.133.20-0ubuntu1 arm64 [installed,automatic]
nvidia-dkms-570-open/unknown,now 570.133.20-0ubuntu1 arm64 [installed,automatic]
nvidia-driver-570-open/unknown,now 570.133.20-0ubuntu1 arm64 [installed]
nvidia-firmware-570-570.124.06/unknown,now 570.124.06-0ubuntu1 arm64 [installed,auto-removable]
nvidia-firmware-570-570.133.20/unknown,now 570.133.20-0ubuntu1 arm64 [installed,automatic]
nvidia-kernel-common-570/unknown,now 570.133.20-0ubuntu1 arm64 [installed,automatic]
nvidia-kernel-source-570-open/unknown,now 570.133.20-0ubuntu1 arm64 [installed,automatic]
nvidia-modprobe/unknown,now 570.133.20-0ubuntu1 arm64 [installed,automatic]
nvidia-settings/unknown,now 570.133.20-0ubuntu1 arm64 [installed,automatic]
nvidia-utils-570/unknown,now 570.133.20-0ubuntu1 arm64 [installed,automatic]

libcudnn8-dev/unknown,now 8.9.7.29-1+cuda12.2 arm64 [installed]
libcudnn8/unknown,now 8.9.7.29-1+cuda12.2 arm64 [installed]
libnvinfer-dev/unknown,now 10.3.0.26-1+cuda12.5 arm64 [installed]
libnvinfer-headers-dev/unknown,now 10.3.0.26-1+cuda12.5 arm64 [installed]
libnvinfer-headers-plugin-dev/unknown,now 10.3.0.26-1+cuda12.5 arm64 [installed]
libnvinfer-plugin-dev/unknown,now 10.3.0.26-1+cuda12.5 arm64 [installed]
libnvinfer-plugin10/unknown,now 10.3.0.26-1+cuda12.5 arm64 [installed]
libnvinfer10/unknown,now 10.3.0.26-1+cuda12.5 arm64 [installed]
libnvonnxparsers-dev/unknown,now 10.3.0.26-1+cuda12.5 arm64 [installed]
libnvonnxparsers10/unknown,now 10.3.0.26-1+cuda12.5 arm64 [installed]

Any ideas?

Thanks

[amadeuszsz]

Hi @rt-2pm2

From your logs, I see you're using the latest release of autoware_universe. We recently added a feature that validates the TensorRT engine. If the version doesn't match, the engine is automatically rebuilt.

To make this work, you can do one of the following:

• Use the Autoware nightly build.
• Wait for the next Autoware release (0.44).
• Manually remove the engine files from your autoware_data directory, then launch Autoware again and wait for the engine to be rebuilt.

[rt-2pm2]

Hi @rt-2pm2

From your logs, I see you're using the latest release of autoware_universe. We recently added a feature that validates the TensorRT engine. If the version doesn't match, the engine is automatically rebuilt.

To make this work, you can do one of the following:

• Use the Autoware nightly build.
• Wait for the next Autoware release (0.44).
• Manually remove the engine files from your autoware_data directory, then launch Autoware again and wait for the engine to be rebuilt.

Thanks @amadeuszsz !

I followed your instructions, precisely, I deleted the files *.engine, and then, after the regeneration, I didn't get that error anymore.