Solving PandaPickCubeCartesian in full cartesian or joint space

[azimi99]

The paper briefly mentions this, but doesn't provide further details. I'm wondering if the environment script for this task can be provided that utilizes additional camera angles.

[StafaH]

We were able to train the full cartesian/joint space under certain conditions. The most straightforward is to add a second camera that is able to view the scene from the side. Adding a camera attached to the hand itself can also help in solving the 3D version of the task. Depending on the pose of the main camera, the task could be solvable but it's tricky from a single fixed camera formulation.

We don't have any environment scripts. @Andrew-Luo1 might be able to provide some code for this, he was the one to crack the full 3D version of the task.

The Brax vision PPO already supports multi-image, so adding a second camera to the scene xml and then loading both images into the observation dict should be sufficient to get you started.