Synchronizing Clients

[icolwell-as]

I noticed the Unreal client runs on its own clock, and updates only if there is a new server tick.
I was thinking it would be nice to be able to guarantee the server and client are synchronized and operating in lock-step.
Also, synchronizing clients would allow faster than real time simulation without worrying about missing ticks.

A few ideas:

1. Use a counting semaphore, the server increases it to some value based on how many clients are expected, then each client decrements the value when it reads. The server only continues to the next tick once the value reaches 0.
2. Keep using the existing binary semaphore, but have a separate semaphore for each expected client (server sets to 1 and waits until client sets to zero, client does opposite).
3. Use the client shared memory. Each client could write the latest tick they received to the client shared memory (or maybe some new shared memory space). The server only continues to the next tick if it sees tick updates from all expected clients.
(EDIT: See ping-pong semaphores in following post)

Anyway, just curious if there were any ideas already floating around, or if there are any preferences in implementation. I'm leaning towards 1 or 2 because then it would be easy to implement using existing semaphore wait/blocking calls.

This isn't something we need right away, but I can see us implementing it later.

[icolwell-as]

After reading a bit more on semaphores, we probably want a set of "ping-pong" semaphores for each client we wish to operate in lock step.

Ping pong semaphores described here:
https://pages.mtu.edu/~shene/NSF-3/e-Book/SEMA/sema-techniques.html

[rodrigoqueiroz]

I had a sync version long time ago with Carla. It worked like this:

Carla has a asynchronous mode where the simulation will not run a tick until you call it. After sending the Traffic State to the client, I call carla_world.wait_for_tick(). That function blocks until the client runs a tick.
The time for the tick is any arbitrary time you are trying to simulate and it will run the physics accordingly. Then I use this fixed time as my delta time and new sim_time in the Server.

Issues: Not everything in Unreal will run correctly as the Engine itself keeps running in real-time. The main process from GeoScenario (SimTraffic and TVs) worked well in this sync mode, but all the SDV Vehicles are running in separate processes and I had to write a signal in the shared memory for them to resume/sleep and wait until all of them finished at least one planning cycle.

Conclusion: The synchronization had too many separate parts to sync in the Server, and not only Server/Client. I had to implement a better synchronization process and even considered putting all SDV instances as ROS nodes and synchronize from there.

In your use case, all vehicles will be running in the main process as PVs/TVs and the only synchronization you need is with the clients. The ping pong semaphore seems to be the best approach.

[icolwell-as]

Thanks @rodrigoqueiroz! I had forgot about SDVs, I'll keep that in mind in case we plan to use SDVs in the future.