Sim2sim for PandaPickCubeCartesian

[lvjonok]

Hello! First of all, thank you for the great library for robot learning!

I am exploring PandaPickCubeCartesian environment in particular for vision-based scenario. Agent trains well and the results are really fast in comparison to other frameworks. However, I face some issues in sim2sim transfer.

Before attempting to transfer the policy for real robot I wanted to deploy it in MuJoCo C simulator. However, the rendering is really different, it was expected as Madrona is used for rendering, but I wanted to hear your thoughts on what can be a pipeline, maybe there are already some existing approaches how to update MjModel from MjxModel and tune rendering for this tranfer.

Render from MuJoCo

Render from Madrona

Moreover, given that the team is actively developing MujocoWarp, I had a question whether the support for Madrona is long-term one because I saw that Warp itself has rendering capabilities.

Thank you in advance! I hope to have a productive discussion here!

[btaba]

@StafaH and @Andrew-Luo1 for madrona rendering settings. There are early efforts to add rendering natively in warp, or to keep supporting madrona, but this is a big TBD

[StafaH]

Hey There,

The easiest way to do sim2sim would be to change the MjModel to match more what the madrona renderer looks like. The default tiled floor in MuJoCo is not supported on the madrona side. If you go into your MjModel and change the color of the floor, colors of the geometry, and mess with the camera fov, you should be able to get a similar domain and the sim2sim should be successful.

If you think there are any issues on the madrona side in terms of things not looking how they are supposed to look just let me know and I can investigate.

[lvjonok]

Thank you for the response, I think something similar to mjx.get_data_into, but for the model will make sense. I will consider your suggestions and keep this thread updated with the results!

[lvjonok]

Indeed, taking care of transferring

'geom_rgba',
'geom_matid',
'geom_size',
'cam_pos',
'cam_quat',
'light_pos',
'light_dir',
'light_directional',
'light_castshadow',
'light_cutoff',

fields from mjx.Model to mujoco.MjModel made the difference.

Madrona Render

MuJoCo Render

One can still say the difference, but I am not aware how to make them look even more similar.

I am yet to sim2sim, but if there are some more fields that can be transferred (looks like FOV is different, although randomize_vision.py does not change it) and it will make quality more comparable, I am open for suggestions.

[StafaH]

It also looks like your ground is not being rendered on the madrona side, so you can either enable it in madrona or disable it in mujoco side. You can adjust the fov to get close to what madrona looks like, but both use different rendering implementations so some discrepancies will still be there. (In MuJoCo setting plane size to 0 0 will create an infinite plane, in madrona setting it to 0 0 will make the geom not renderered).