High Latency & VR Instability with Quest 3 in Isaac Sim

[Mzkzzzz]

Description

I'm using a Meta Quest 3 headset with a simulation setup based on the unitree_sim_isaaclab repo and Isaac Sim 4.5.0. Two major issues occur:

1. High Latency

There's noticeable delay between Quest 3 controller (or hand tracking) input and the robot's response.

Latency exists even when running on a powerful machine (RTX 4090 Laptop) with CUDA enabled.

Particularly problematic for controlling end-effectors.

2. Robot Falls Over When Head Tracking is Enabled

When I wear the Quest 3 and rotate my head, the robot in simulation falls over or destabilizes.

This only happens when full 6-DoF VR tracking is active.

If I keep my head still (or only use the controller), the robot stays balanced.

Launch Isaac Sim 4.5.0 with the following command:
(isaaclab) frank@zhekai:~/project_202508/unitree_sim_isaaclab$ python sim_main.py --device cpu --enable_cameras --task Isaac-Move-Cylinder-G129-Inspire-Wholebody --enable_inspire_dds --robot_type g129

Launch xr teleoperate with the following command:(tv) frank@zhekai:~/xr_teleoperate/teleop$ python teleop_hand_and_arm.py --ee=inspire1 --sim

Main issues:
**Observe latency while moving hand/controller.

Put on headset and rotate head → robot destabilizes and collapses.**

I would appreciate if you can offer me some advice!!!

[silencht]

All the recent latency reports have come from Quest 3 users; while it’s unlikely, we can’t rule out a Quest-specific issue.

Please restart every device—including your router (this is critical). Then verify that your Quest 3 is connected to the 5 GHz Wi-Fi band. After that, check whether the latency persists.

[Mzkzzzz]

Thank you for your reply. I will try vision pro then.

Sorry to bother you again. Which mode can I use if I want to simulate the behavior of walk? --sim or --motion? Any advice on the sudden collapse when I was trying to move my head (with quest3)?

Have a good day!!!

[le-wei]

@Mzkzzzz
If you are using the built-in low-level control model provided by the project, please refer to the example send_commands_8bit.py for sending commands. If you are issuing commands via a VR device, make sure to map the commands to the corresponding ranges.

[Mzkzzzz]

Hi, I would like to know if it's possible to control the walking of the robot in simulation mode using Quest 3 only, instead of just controlling the hand/arm.

If this is not currently supported, and code modification is required, which file should I modify to implement VR-based walking control? Should I create a new script similar to send_commands_8bit.py, or integrate it into the existing VR pipeline?

Thanks!

[silencht]

All the recent latency reports have come from Quest 3 users; while it’s unlikely, we can’t rule out a Quest-specific issue.

Please restart every device—including your router (this is critical). Then verify that your Quest 3 is connected to the 5 GHz Wi-Fi band. After that, check whether the latency persists.

@Mzkzzzz
Another potential fix for latency and disconnections is to switch your Quest 3 from Wi-Fi to ADB port forwarding. You can refer to #32 for guidance.

[Mzkzzzz]

Thanks for the suggestion! However, my main question is not about latency, but rather whether VR-based walking control in simulation mode is supported at all using Quest 3, or if I should modify the codebase (e.g., send_commands_8bit.py or another file) to implement it.

[le-wei]

@Mzkzzzz
We have not implemented an example for lower-limb control using VR. You will need to implement it yourself to control lower-limb motion through Quest 3. The main idea is to obtain joystick input from the Quest 3 and map it to [forward/backward linear velocity, lateral velocity, rotational velocity, and crouch speed]. Then, follow the structure in send_commands_8bit.py to organize the DDS message and publish it to the corresponding DDS topic.