How to train on RGB-D dataset?

[JimXu1989]

🚀 Feature

I want to train Instance segment on rgbd dataset

Motivation & Examples

I want to use a rgbd camera to capture the data, when I segment the object, I can also get the 3D information

[GitHub-Crow]

Make a normal detection dataset and a depth info annotation may be a simple way to apply detectron2 to implement this.

[JimXu1989]

Make a normal detection dataset and a depth info annotation may be a simple way to apply detectron2 to implement this.

May you discribe more details about the method?
Thanks a lot

[andreaceruti]

Hi Jim! I am interested in this application too. Have you done some progress so far?

[gfiameni]

@gfiameni I have also done it in your same way, but the performance are better if I use only RGB image. I was thinking of trying to fuse RGB and depth directly using the maskrcnn repo to check the results, but I haven't done it yet. Looking some papers It seems that the standard way of fusing depth and rgb channels is not by stacking the channels together, instead we should build 2 different backbone, one of them extracts RGB features and the other one extract depth features. At the end with some pooling layer we should join the different features. Up to now I have not tried this since it will be very time consuming, but if one of us will ever do it we can use this thread to share our thoughts

Yes, you are right. Fusing unrelated channels it is not expected to provide any benefits. Actually my attempt was to fuse a depth image (greyscale palette) but performance is still an issue.

[andreaceruti]

@gfiameni yes, the same for me. I add grayscale depth images to my dataset like this one

Actually I get 0.79 AP from evaluation of the RGB dataset, instead just 0.67 from the RGBD dataset, but It could be also because my dataset is small (only 49 images).
Have you seen this wiki shared from official maskrcnn implementation? I also start training from scratch (Model.weights = ""), so I think I respect all the suggestions found in detectron2 and maskrcnn repo.
Anyway If I will ever try to fuse 2 different backbones with detectron I will share the model here

[andreaceruti]

@gfiameni Another useful thing to do could be to enhance the deth image as done in this paper before feeding them to the network

[gfiameni]

@gfiameni yes, the same for me. I add grayscale depth images to my dataset like this one Actually I get 0.79 AP from evaluation of the RGB dataset, instead just 0.67 from the RGBD dataset, but It could be also because my dataset is small (only 49 images). Have you seen this wiki shared from official maskrcnn implementation? I also start training from scratch (Model.weights = ""), so I think I respect all the suggestions found in detectron2 and maskrcnn repo. Anyway If I will ever try to fuse 2 different backbones with detectron I will share the model here

Have you normalized the depth before transforming it into greyscale? I am working with a small dataset too but fine-tuning FPN with RN50 brings me to max 0.59 of AP.

[andreaceruti]

@gfiameni Yes I normalize It since in my case, have a uint16 depth format that represents the real dinstance from 0 to 6 meters, so my maximum uint16 value is 6000, that represents 6 meters (D435i realsense camera).
I normalize it in uint8 with the meaning that 6 meters are now represented as 255, 0 instead are the invalid pixels, probably due to the light reflex. The depth has to be normalized if I am not wrong because some layer of the network does not accept as input uint16 arrays, I do not remember which exactly but I can try to reproduce the problem.

[redsphinx]

I am trying to add do pantoptic segmentation on RGB-D images. Did anyone figure out how to add the depth information to panoptic_fpn_R_50 ?