Any examples of multiple cameras capturing depth information?

[yizhangdanikor]

I used two cameras to collect RGB and depth information in Omnibizon. I set the camera position and orientation using cam_1. set_position orientation (cam_1-position, quat1), but I found that when the two cameras were using their own set positions and quaternion poses for coordinate conversion, the point clouds could not align. Can anyone give me any suggestions for checking? In addition, after setting it up, I also checked the settings using cam_1. get_position _orientation (), and the obtained parameters were consistent with the set parameters. Is the coordinate system of the camera in Omnibizon equivalent to the depth acquisition reference plane?

[yizhangdanikor]

I am using OmniGibson v1.1.1

[hang-yin]

I vaguely remember @cremebrule had experience setting up point cloud observation. Do you mind taking a look at this?

[hang-yin]

Hi @yizhangdanikor , here's a response from @cremebrule :

in order to convert from image -> world frame you have to actually do image frame -> camera frame -> world frame where the first transform is 180 degrees about the x axis;
I'm guessing it's that middle step that is missing.

[yizhangdanikor]

Hi @yizhangdanikor , here's a response from @cremebrule :

in order to convert from image -> world frame you have to actually do image frame -> camera frame -> world frame where the first transform is 180 degrees about the x axis; I'm guessing it's that middle step that is missing.

Hi, thank you for your reply. I tried your method and it seems to be correct. By the way, I also want to ask if the normal in Omnigibson is represented by the x, y, z in the obtained quaternion [x, y, z, w]? And this normal is in the world coordinate system, not the camera coordinate system.

[hang-yin]

Hi @yizhangdanikor glad it works!

1. For normal, you potentially do something like:

import omnigibson.utils.transform_utils as T
z_up = th.zeros((3, 1))
z_up[-1] = 1.0
normal = (T.quat2mat(quaternion_orientation) @ z_up).flatten()

2. When you call get_position_orientation, you can pass in a frame argument ["world", "scene", "parent"] to specify in which frame you want the pose to be in.

[yizhangdanikor]

1. For normal, you potentially do something like:

import omnigibson.utils.transform_utils as T
z_up = th.zeros((3, 1))
z_up[-1] = 1.0
normal = (T.quat2mat(quaternion_orientation) @ z_up).flatten()

2. When you call get_position_orientation, you can pass in a frame argument ["world", "scene", "parent"] to specify in which frame you want the pose to be in

Thank you for your quick response. However, what I obtained is the normal of the object surface in the image currently captured by the camera, which is used as the ground truth for normal estimation. So, the API I used is norm=cam.get_obs()[0]["normal"].cpu().data.numpy(). This API does not seem to have parameters for specifying the coordinate system. Moreover, the normals I using obtained this method are quaternions. Currently, I am taking the first three elements of [x,y,z,w] and normalizing them. The resulting normals are in the world coordinate system. The visualized normals seem to be correct, but I am not sure how to confirm their accuracy. You mentioned the method of converting quaternions to normals. I tried it before, but the visualization of the normals drawn from the object surface was not as good as the method I mentioned.