Performance Regression (CPU & Latency) After Upgrading Fast DDS from 2.6.x to 2.14.5 with ROS 2 Humble

[yglsaltfish]

Hi Fast DDS Team and Community,

We are currently using ROS 2 Humble and have recently upgraded our RMW implementation's Fast DDS version from 2.6.x to 2.14.5.

After the upgrade, we conducted some performance benchmarks and observed a noticeable regression in terms of CPU utilization and message latency, especially as the message size increases.

I have attached a set of performance graphs that illustrate this. This issue occurs in both cross-machine and local communication scenarios. The tests were run with a single topic at a frequency of 1 kHz. The key observations from the graphs are:

• Avg_Latency_Us / Max_Latency_Us: Both average and maximum latency are significantly higher with Fast DDS 2.14.5 compared to 2.6.x, particularly for message sizes above 16KB.
• Avg_Cpu_All_Runs (%): The CPU usage is consistently higher across all message sizes with the 2.14.5 version, for both Best-Effort and Reliable QoS settings.

The legend in the image details the specific configurations tested:

• Channel-Ros2-Besteffort-Fastdds2.6.X_Multi-Topic (Blue)
• Channel-Ros2-Reliable-Fastdds2.6.X_Multi-Topic (Orange)
• Channel-Ros2-Besteffort-Fastdds2.14.5_Multi-Topic (Green)
• Channel-Ros2-Reliable-Fastdds2.14.5_Multi-Topic (Red)

Given these results, we have a few questions:

1. Is this performance difference between versions 2.6.x and 2.14.5 an expected behavior?
2. Are there any known issues or significant changes between these versions that could explain this increase in latency and CPU load?
3. Does eProsima provide official benchmark results for different Fast DDS versions? Or is there a recommended benchmarking suite we can use to validate our findings against a standard?

Any insights or suggestions would be greatly appreciated. Thank you