Running Autoware and AWSim Lab on different machines

[rt-2pm2]

I would like to ask whether someone is able to successfully run Autoware and AWSim Lab on two separate machines.

I followed the documentation for configuring DDS for multiple computers.

My configuration is composed as follow:

• Simulation Machine (AWSim Lab) with i7-4790K (4core, 8 threads), 32GB RAM, and a 4060Ti
• Autoware Machine (Ampere Altra Development Platform) with the 64 Core Arm CPU and a 4060Ti

The Simulation machine is able to run the AWSim Lab simulator 60fps when no-one is subscribing to its topics. However, launching Autoware on the other machine determines the simulator to nearly freeze (0-1 fps).

I tried to just subscribe to the image_raw topic from the camera with a simple ros2 topic hz /sensing/camera/traffic_light/image_raw from the Autoware Machine and I notice a visible drop in the frame rate ( < 10 fps ).

Do you have some hints about what could cause the issue? I guess it's a networking problem, since subscribing on the same machine does not impact the framerate. The bandwidth occupied by the camera topic is about half of the 1Gbps LAN.
I saw that someone performed tests in a distributed scenario but they were employing 10Gbps adapter. I was wondering whether that's a requirement for running the simulator on a separate machine.

Thanks in advance for any suggestions.

[liuXinGangChina]

hi @rt-2pm2

Don't worry ， it is easy to run in 2 pcs。
Just make sure the interface name is the eth port that you used to connect 2 pc rather than lo，
Lo is used for single pc scenario

Happy hacking

心刚

[rt-2pm2]

Hi @liuXinGangChina,
thank you very much for your reply. Did you experience any issue of FPS drop, like the one I was mentioning in the post?

I followed the documentation for setting up the two-machine scenario, and I indeed had the cyclone xml configuration files specifying the device used for inter-machine communication. As I reported above, the communication itself was possible, the only problem was the near-freeze of the AWSim simulator when another machine subscribes to its topics.

I was looking for some feedback, hoping to spot some stupid mistake I could have made, before going down the rabbit hole inspecting the AWSim source code.

What type of network adapter are you using in your setup? Are you able to make it work, maintaining the simulation smooth, with standard 1Gbps Ethernet?

Thanks again for your time

[liuXinGangChina]

Regarding hardware config

Better try replace sim-machine ‘s cpu with higher end like 11500 or more，because simulator itself not only use graphic card but also use cpu to calculate sensor data

Best regards

心刚

[rt-2pm2]

Regarding hardware config

Better try replace sim-machine ‘s cpu with higher end like 11500 or more，because simulator itself not only use graphic card but also use cpu to calculate sensor data

Best regards

心刚

Hi I see that indeed the CPU could be not optimal, however the issue I am experiencing seems related to networking. For this reason,
I was curious to understand what type of network hardware is used for this "distributed" simulation.

The reason why I think it is a network issue is due to the tests I made. The slow down in the AWSimLabs frame-rate is experienced only when I subscribe to its topic from another machine. It is not the case when I subscribe to the topics from the same machine. So, I guess that streaming the camera topic on the network creates a bottleneck in the simulation pipeline.

If there are other users running the AWSim simulator on a machine and the Autoware Framework on another machine, linking them with 1Gbps LAN, and they don't have the problem I posted here, then I should have made some mistake in the network configuration.

[liuXinGangChina]

on my 2 local machines, i use 1000m lan adapter and the fps is around 25 fps
truly it is not 60 fps but not similar to your status which is below 10 fps

did you connect 2 pc directly or using a switch?

better try directly

happy hacking

心刚

[rt-2pm2]

on my 2 local machines, i use 1000m lan adapter and the fps is around 25 fps truly it is not 60 fps but not similar to your status which is below 10 fps

did you connect 2 pc directly or using a switch?

better try directly

happy hacking

心刚

Hi, thanks for your feedback and collaboration.

I have the two machines connected using a switch.

I tried to change the old machine, with a slightly more capable one having a 6 cores cpu (Intel(R) Core(TM) i7-8700 ). Alas, I have the same behavior: as soon as the Autoware framework starts fetching data from the simulator....the latter freezes to nearly 0-1 fps.

With the better cpu, I see 25-30 fps with just the ros2 topic hz /sensing/camera/traffic_light/image_raw. It's better than the old one, but yet it is not able to bear the streaming workload towards the Autoware machine.

I also noticed something interesting: when it freezes, the network bandwidth is not saturated.
I see just 25Mbits/s when it's freezing, whilst I see the ~500Mbits/s when I am listening to the camera topic.
Also the CPU utilization of the machine is low, so I suppose there is some issue with the communication, maybe a blocking call or similar...but you say that you have a working setup...so I don't know where to look for.

By the way, @liuXinGangChina are you using the AWSim-Labs Demo Binary for your simulations?

[rt-2pm2]

I think I found the Autoware component that is causing the AWSim to freeze. I tried to disable the "perception" part and the simulator was able to run at 50 fps. I then tried to re-enable the perception, while tweaking the module: it seems that disabling the "traffic light module" allows the simulator to run at 40-50 fps without freezing.

I am still investigating the issue, since it's not clear to me how a module on the Autoware side could cause the freeze of the AWSim.

[liuXinGangChina]

1. Yes we use awsim-lab binary file for autoware testing
2. Traffic light module use image-raw from awsim-lab simulator for traffic light inference pipeline, if your test env don't involve traffic light things you can disable that output in the panel in awsim-lab simulator。 Indeed the image raw from the simulator use great bandwidth between 2 pc

Happy hacking

心刚

[rt-2pm2]

Thanks for the advice.

Yet, I will try to understand why the freeze. Disabling the inference, but subscribing to the topic does not freeze the simulator, so the bandwidth is not the only problem.

[liuXinGangChina]

Wierd， on my side ros2 topic echo on pc2 will cause fps decrease on pc1

[liuXinGangChina]

By the way ， awsim team is working on this issue now ， you may try new release in the future

[rt-2pm2]

Thanks for the information!

Wierd， on my side ros2 topic echo on pc2 will cause fps decrease on pc1

I get a decrease on the FPS too, but not a complete freeze (0-1 FPS). The streaming of the data from the AWSim to another machine determines a ~10/15 FPS drop, but it would be manageable.

By the way ， awsim team is working on this issue now ， you may try new release in the future

May I ask you where I can find the details about the work they are doing? Is there another topic where they were discussing this problem? As far as I know, you are the only one aware of my issue :-D.

[rt-2pm2]

I made several tests and it seems that the problem is not the "different" machines, but is dependent on the interface used to connect the nodes.
Indeed, I was able to reproduce this issue also running everything on the same machine.

As I said in the problem statement, I was able to run the AWSim-Lab and the Autoware framework on the same x86 machine. However, I was doing this selecting the "loopback" interface, as instructed in the documentation.
I tried to use the Ethernet interface as device and I started seeing the "freezing" problem.
Maybe, is the problem related to the limited mtu of the Ethernet interface? Or maybe the queuing discipline?
These values are different on the two interfaces:

1: lo: <LOOPBACK,MULTICAST,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: enp7s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
    link/ether 18:31:bf:b6:4a:42 brd ff:ff:ff:ff:ff:ff

I want to point out that the DDS is working: I see the topics and I can run the ddsperf and test the pings on the same machine when I use the physical Ethernet interface in the cyclonedds.xml configuration file.

I tried another simpler demo that required the GPU and it fails when I run it on the physical Ethernet interface. I am referring to the "rosbag playback" demo.

@liuXinGangChina Have you ever tried running demos, meant to be used with the "loopback", selecting the physical Ethernet interface on the same machine?

@xmfcx I saw that you made tests on a distributed configuration, do you have any comments on the issue I am seeing here?

[rt-2pm2]

I want to give a last minute update:

To give some context, after the first failed attempts weeks ago where I had the two machines on the same switch, but for convenience I recently moved the simulation machine in another part of the same room, yet connected to the same LAN with a speed of 1Gbps (verified with the iperf). At that time I didn't try the mtu setting on the interfaces.
After the good partial results I was getting yesterday, by playing with the mtu settings, I moved the machine back on the same local switch, where the Autoware machine is connected, and I noticed an improvement. I could suspect that some of the switches in the rack, connecting the different part of the room, does not support the jumbo frames with big mtu and that was hindering the performance (just a supposition).

Now, with mtu at ~9000 and the two machines directly attached to the same switch a few meters apart I start seeing something moving.
Yet, it seems that the point cloud from AWSim-Lab arrives a bit staggered and the autonomous driving is not perfect, but I can run the demo.
I also tried AWSim (the tier4 version) which has a vehicle with a single lidar and I got better results than with the AWSim-Lab.

Any comment on the mtu? It was not reported as something critical in the documentation, but the tests I made when running local demos, supposed to be run on the loopback, on the Ethernet interface seem to point out that the default 1500 was not enough.

[rt-2pm2]

Thanks a lot @xmfcx for sharing your experience and providing your support.

First question is that you are testing "Rosbag replay simulation" within single machine for both machines right? You are not attempting to run rosbag in one and the autoware on the other. (Just to clarify.)

Yes, I was playing with the MTU/interface setting while running the rosbag and the autoware framework on the same machine.

As stated above, the "local" demo worked on the loopback but was failing when using the Ethernet interface in the dds settings. Increasing the mtu seemed to fix the problem on the x86 machine.

[rt-2pm2]

If these data are going through the router, it will make things and the entire network slower.

Can you connect the both machines directly with a cable without any switch or router involved? And redo the autoware tests?

You are right that the network could be a limiting factor. I will make more tests: first by directly connecting the machines and then upgrading to 10Gbps.

[xmfcx]

When I decided to run Autoware and AWSIM on different machines, I started from getting both machines a 10GbE interface and a good cable.

And I reported all that I changed here: https://github.com/orgs/autowarefoundation/discussions/3813

I don't have access to those machines anymore so I cannot check their MTU settings. But I don't remember having to modify those.

Bandwidth tests

With the default configuration in the loopback, the bandwidth usage by AWSIM Labs (on real time speed) is about 350 Mbps.

I had to get the PID of AWSIM Labs first, then run sudo lsof -Panp 493236 -i. It returned:

COMMAND      PID USER   FD   TYPE  DEVICE SIZE/OFF NODE NAME
awsim_lab 493236  mfc   50u  IPv4 1427891      0t0  UDP *:51467 
awsim_lab 493236  mfc   51u  IPv4 1427892      0t0  UDP *:7400 
awsim_lab 493236  mfc   52u  IPv4 1427893      0t0  UDP *:7401 
awsim_lab 493236  mfc   53u  IPv4 1427894      0t0  UDP 127.0.0.1:44691 

So it is using 44691 UDP port for the loopback traffic.

Then ran wireshark on lo with sudo and Menu Bar -> Statistics -> I/O Graph

This shows the traffic used by AWSIM Labs (which is about 350 Mbps).

This means, if you have a similar config, you are not bottle-necked by the bandwidth.

[rt-2pm2]

Thanks for the information.

Yes, indeed I can confirm that the 1Gbps bandwidth itself should be sufficient. I had a glance at iftop to get an idea during some of the tests, and the numbers I saw were not far away from what you recorded. When I observe staggering and lag there is no bandwidth saturation. It was curious to see that when I had the freezing problem, the bandwidth, CPU and GPU utilization was low.