Why In Instance Segmentation, the Predicted Mask Is NOT precise?

[innat]

Hello. I've tried Detectron2 and also MMDetection to verify the predicted instance segmentation mask on known classes (the classes that belong to the coco data set). Let's say, car image. Here are the results of the /mask_rcnn_x101_64x4d_fpn_mstrain-poly_3x_coco.py config.

(left: original input from web, right: mask-rcnn-x101 output).

Please see the mask prediction below, it's not precise in a sense.

Now, the thing is, I only did inference with the pre-trained weight, which is optimized on the coco dataset which also has the car (index 2). For that, I assumed the model should give a more precise mask prediction on a very simple car image. Some query:

• Is it simply the capacity of the model itself, and I should try more sota models with high mask AP?
• Or, I better retrain my dataset, even though the pre-trained weight comes cocoa-like dataset. For example: let's say, my dataset contains: airplane, human, car; three classes, and I should not rely on the pre-trained weights only but need to fine-tune, even though the pre-trained weights have the information of these 3 classes?
• Or, Should I tweak some other parameters, like, image size, threshold, bgr/rgb, etc.?

---

Tested the QueryInst, highest mask ap, but the results are the same overall.

[zachcoleman]

• It is the current capacity of the model. For Mask-RCNN, the default output of the mask head is either 14x14 or 28x28 and the mask must be resized and fit to the predicted bbox for the object. In your example, the car is much larger than 28x28 and so the upsampling is causing this lack of "precision". Mask-RCNN is sensitive to scale, anchor configuration, and mask configuration. Your example is an extreme case of the scale of an object in the image.
• Needing to retrain and/or fine-tune highly depends on your input images vs COCO (are you out-of-domain?)
• In mmdet, you may want to tweak the config for larger mini mask sizes