How to simultaneously visualize and render offscreen?

[aseembits93]

(Sorry for cross posting from Issues!)

Referring to https://roboti.us/forum/index.php?threads/how-to-simultaneously-visualize-and-render-offscreen.3517/ which was not resolved completely.
Thank you for opensourcing mujoco, it is integral to my research. I have a scene with a robot and I need to render two viewpoints, a third person free moving view of the robot(onscreen) and a view from a fixed camera attached to the robot's body(offscreen).
Could someone guide me on how to do this? I define one mjmodel, mjdata and two GLFWwindow, mjvCamera, mjvOption, mjvScene, mjrContext.
Guidance would be very much appreciated. Thanks!

[yuvaltassa]

Hello, did you resolve this?

Rendering two camera views should not be a problem, though indeed with glfw you might need to open a second window, I don't remember the details of glfw's support for windowless rendering.

[aseembits93]

Hi @yuvaltassa, I was able to get dual windowed rendering to work but seem to be having trouble implementing offscreen and windowed rendering. Haven't been able to pinpoint the exact cause. I'm suspecting it could be related to context switching. Hope to hear back soon.

[aseembits93]

@yuvaltassa
Here's the code (you could copy it in the sample/ folder)->
https://github.com/aseembits93/dualrendermujoco/blob/main/dualrender.cpp

You can compile by doing g++ -O2 -I../include -L../bin -std=c++11 -mavx -pthread -Wl,-rpath,'$ORIGIN' dualrender.cpp -lmujoco200 -lGL -lglew ../bin/libglfw.so.3 -o ../bin/dualrender

and run by doing ./dualrender ../model/humanoid.xml 30 1 abc.out

Any guidance on modifying this for windowed + offscreen rendering would be very very helpful. Thank you in advance!

[Domattee]

If you can render to two windows with GLFW, to my knowledge the only thing you still need is to start the second window invisible with glfwWindowHint(GLFW_VISIBLE, 0) as in record.cc.

If I understand the documentation correctly there is another way for offscreen rendering already implemented which skips GLFW.
In the model xml you can define offheight and offwidth for the visual element, which determines the size of the offscreen buffer.
After rendering your main window you mjr_setBuffer to OFFSCREEN, render again, and mjr_readPixels, before setting the buffer back again. This way you do not have to open two GLFW windows or contexts.

[aseembits93]

Thanks @Domattee!