GLFW MUJOCO_GL option for Darwin (M-series) instead of CGL

[hello-robot-shehab]

Intro

Hello!

We are using MoJoCo to simulate Stretch from Hello Robot, to allow researchers and professors at universities to use Stretch in a simulated environment.

My setup

mujoco==3.2.7, Python, ARM (M3 chip), macOS Sequoia 15.1.1

My question

On MacOS, I would like to create multiple renderers alongside the main mujoco.viewer.launch() window, e.g. scene1 = mujoco.Renderer(mjmodel, height=480, width=640) scene2 = mujoco.Renderer(mjmodel, height=480, width=640)

However, I notice that CGL has a mutex lock on GL Context when the main viewer is in view.

This prevent me from being able to call scene1.render() -> which causes the main thread to hang as it waits, and if I move it to a thread, that thread just sleeps forever because it can never acquire the lock.

I'd like to use GLFW instead of CGL for rendering to support rendering multiple Renderers.

On my local machine, I commented out the darwin check in mujoco/gl_context.py to force it to use GLFW, and I can confirm that I can now perform headless rendering.

However, gl_context.py ignores the MUJOCO_GL=glfw environment variable, and forces Darwin users to CGL.

My question is, why is this done? GLFW seems to work just fine on my local macOS.

Thank you!

Minimal model and/or code that explain my question

Model: Any, doesn't matter, but this is the one we're using: https://github.com/hello-robot/stretch_mujoco/blob/main/stretch_mujoco/models/scene.xml Thank you.

Code:

import mujoco
import mujoco._functions
import mujoco._callbacks
import mujoco._render
import mujoco.viewer
from mujoco._structs import MjData, MjModel

scene_xml_path = "./models/scene.xml"
mjmodel = MjModel.from_xml_path(scene_xml_path)
mjdata = MjData(mjmodel)

rgb_renderer = mujoco.Renderer(mjmodel, height=480, width=640)

def ctrl_callback( model: MjModel, data: MjData):
    # This is simplified -> we'd have this on a thread. If there is a better option, please let us know!
    rgb_renderer.update_scene(data, "d405_rgb")
    rgb_frame = rgb_renderer.render() # This fails on CGL with darwin, works fine on linux

mujoco._callbacks.set_mjcb_control(ctrl_callback)

mujoco.viewer.launch(mjmodel)

Confirmations

• I searched the latest documentation thoroughly before posting.
• I searched previous Issues and Discussions, I am certain this has not been raised before.

[yuvaltassa]

@saran-t, any thoughts?

[saran-t]

I think I'm missing something, but why do you need a separate physics thread at all?

[hello-robot-shehab]

Hey @saran-t, without the physics thread, we're seeing a viscous motion in the Passive Viewer; it looks like the robot is moving through molasses. This is not seen on the managed viewer.

I am using three devices for my testing: a slow intel NUC, a faster NUC and a M3 Pro. The behavior is less noticeable on the faster devices, but it still does not look as smooth as the managed viewer - even though the same _ctrl_callback is running.

After I added an FPS counter to the UI (passive) thread, I noticed that the .sync() call was very slow, so the mj_step() call was missing the timestep.

why do you need a separate physics thread at all?

When I moved the mj_step() call to a different Physics thread, and load the UI thread so it's not sitting on the sync lock, I'm seeing the Physics thread cycle at the appropriate timestep.

[hello-robot-shehab]

Hey @saran-t, why do you say we should not need a physics thread?

[hello-robot-shehab]

Hey @saran-t, just updating you.

For the off-screen rendering of 5 cameras on CPU, we found the best simulation stability and performance with running:

• viewer.sync() and Renderer.render() (inside a viewer.lock() context) on the main loop capped at 30Hz (sleeping this thread makes the lock available for longer on the physics thread)
• and mj_step() (with the viewer.lock() context) on the spawned physics thread.

To note: a combination of not using the appropriate locks during rendering the off-screen renderers and stepping the simulation yielded weird instabilities in simulation.

It would be great to not have to use the lock to render the cameras (because acquiring the viewer lock blocks the physics thread), but not using them resulted in errors like: WARNING: Nan, Inf or huge value in QPOS at DOF 0. The simulation is unstable. that do not appear when the locks are used when rendering offscreen renderers.

To answer the original question, we found that even though enabling GLFW on macs allows us to do the rendering on the physics thread, doing this ultimately yielded about the same results as using the Passive Viewer with the configuration I mentioned above. So, taking your advice of avoiding undefined behavior, the passive viewer implementation seems to work.

Thank you for your suggestions and comments, @saran-t .