AttributeError: Cannot find field 'gt_masks' in the given Instances! for mask2former with coco-format dataset

[jetsonwork]

Hello,
I want to fine tune a swin-based mask2former model, and I register my coco format dataset as below:

from detectron2.data.datasets import register_coco_instances
register_coco_instances("my_dataset_train", {}, "/content/drive/MyDrive/ColabNotebooks/data/3D-EM-Platelet/train/3D-EM-platelet-train.json", "/content/drive/MyDrive/ColabNotebooks/data/3D-EM-Platelet/train")
register_coco_instances("my_dataset_val", {}, "/content/drive/MyDrive/ColabNotebooks/data/3D-EM-Platelet/val/3D-EM-platelet-val.json", "/content/drive/MyDrive/ColabNotebooks/data/3D-EM-Platelet/val")

train_metadata = MetadataCatalog.get("my_dataset_train")
train_dataset_dicts = DatasetCatalog.get("my_dataset_train")

val_metadata = MetadataCatalog.get("my_dataset_val")
val_dataset_dicts = DatasetCatalog.get("my_dataset_val")

but when I want to train the model I am facing to the below error. Could you please advise how can I solve the issue?

AttributeError: Cannot find field 'gt_masks' in the given Instances!

[DimbyTa]

Hello @jetsonwork,

Your issue relates not to how you register your dataset but to the data itself or the model you use. Indeed, after reading the docs: https://detectron2.readthedocs.io/en/latest/_modules/detectron2/structures/instances.html?highlight=AttributeError#, you can see that the AttributeError is raised when: name == "_fields" or name not in self._fields, as stated in the docs of Instances.

If it was related to how you registered your dataset, the error would have stated that it could not find my_dataset_train or my_dataset_val.

If you read the docs on gt_masks, you can see that it is related to Model Input (https://detectron2.readthedocs.io/en/latest/tutorials/models.html?highlight=gt_masks#model-input-format), the docs talk about custom models and what they expect as input and the correct format.

As stated in https://detectron2.readthedocs.io/en/latest/tutorials/models.html?highlight=gt_masks#use-a-model, each model takes as inputs a list[dict] with each dict corresponding to one image. It is also stated that the required keys depend on the model.

That means the mask2former model probably requires the gt_masks field in the inputs (e.g.: [{"image": Tensor, "height": some_value, "width": some_value, ..., "gt_masks": PolygonMasks or BitMasks},{"image": Tensor, "height": some_value, "width": some_value, ..., "gt_masks": PolygonMasks or BitMasks},... ]).

Can you confirm that you do have gt_masks in the inputs? Try: print(train_dataset_dicts[0].keys())

For more information on Instances: https://detectron2.readthedocs.io/en/latest/modules/structures.html#detectron2.structures.Instances

These might help you but I'm not sure:
https://colab.research.google.com/github/NielsRogge/Transformers-Tutorials/blob/master/MaskFormer/Fine-tuning/Fine_tuning_MaskFormerForInstanceSegmentation_on_semantic_sidewalk.ipynb

https://detectron2.readthedocs.io/en/latest/tutorials/datasets.html

[jetsonwork]

Thank you @DimbyTa for your advice. when I print(train_dataset_dicts[0].keys()) the result is as below:

and the below is the print(train_dataset_dicts[0]):

{'file_name': '/mnt/c/Users/.../Downloads/small_ds/1.jpg',
'height': 1008,
'width': 756,
'image_id': 1,
'annotations': [{'iscrowd': 0,
'bbox': [320.9885931558934,
532.8285667052404,
18.330302529343726,
19.99669366837486],
'category_id': 0,
'segmentation': [[323.48817986444027,
532.8285667052404,
320.9885931558934,
547.8260869565215,
333.48652669862776,
552.8252603736153,
339.3188956852371,
551.9920648040996]],
'bbox_mode': <BoxMode.XYWH_ABS: 1>},
{'iscrowd': 0,
'bbox': [295.99272607042474,
529.4957844271779,
14.164324681765606,
7.498760125640501],
'category_id': 0,
'segmentation': [[304.324681765581,
529.4957844271779,
295.99272607042474,
531.1621755662092,
295.99272607042474,
536.9945445528184,
302.65829062654973,
536.9945445528184,
310.15705075219034,
536.9945445528184]],
'bbox_mode': <BoxMode.XYWH_ABS: 1>},
{'iscrowd': 0,
'bbox': [216.00595139692504,
651.9755331459743,
33.32782278062487,
20.829889237890598],
'category_id': 0,
'segmentation': [[230.17027607869062,
651.9755331459743,
216.00595139692504,
651.9755331459743,
216.00595139692504,
666.1398578277399,
231.00347164820624,
671.9722268143493,
247.66738303851866,
672.8054223838649,
247.66738303851866,
664.4734666887086,
249.3337741775499,
657.8079021325836]],
'bbox_mode': <BoxMode.XYWH_ABS: 1>},
{'iscrowd': 0,
'bbox': [187.6773020333939,
674.4718135228961,
17.497106959828074,
14.99752025128123],
'category_id': 0,
'segmentation': [[195.17606215903447,
689.4693337741774,
187.6773020333939,
684.4701603570836,
188.5104976029095,
674.4718135228961,
196.0092577285501,
674.4718135228961,
205.17440899322196,
674.4718135228961]],
'bbox_mode': <BoxMode.XYWH_ABS: 1>},
{'iscrowd': 0,
'bbox': [305.1578773350966,
651.1423375764587,
10.83154240370311,
13.331129112249982],
'category_id': 0,
'segmentation': [[315.1562241692841,
664.4734666887086,
305.1578773350966,
664.4734666887086,
305.1578773350966,
657.8079021325836,
310.99024632170597,
651.1423375764587,
315.9894197387997,
657.8079021325836]],
'bbox_mode': <BoxMode.XYWH_ABS: 1>},
{'iscrowd': 0,
'bbox': [297.659117209456,
629.4792527690526,
15.830715820796854,
16.663911390312364],
'category_id': 0,
'segmentation': [[313.48983303025284,
634.4784261861463,
302.65829062654973,
629.4792527690526,
297.659117209456,
636.1448173251775,
297.659117209456,
640.3107951727555,
300.9918994875185,
646.1431641593649,
307.6574640436435,
646.1431641593649,
312.6566374607372,
644.4767730203337]],
'bbox_mode': <BoxMode.XYWH_ABS: 1>},
{'iscrowd': 0,
'bbox': [260.165316581253,
653.6419242850055,
19.163498098859293,
39.16019176723432],
'category_id': 0,
'segmentation': [[274.3296412630186,
692.8021160522399,
266.83088113737796,
692.8021160522399,
266.83088113737796,
682.8037692180524,
260.165316581253,
676.1382046619274,
261.83170772028427,
662.8070755496774,
265.9976855678624,
655.3083154240368,
272.6632501239873,
653.6419242850055,
275.1628368325342,
666.1398578277399,
273.49644569350295,
672.8054223838649,
279.3288146801123,
680.3041825095055]],
'bbox_mode': <BoxMode.XYWH_ABS: 1>},
{'iscrowd': 0,
'bbox': [280.99520581914356,
671.1390312448336,
14.16432468176555,
18.330302529343726],
'category_id': 0,
'segmentation': [[295.1595305009091,
671.1390312448336,
283.49479252769044,
672.8054223838649,
282.6615969581748,
676.971400231443,
280.99520581914356,
683.636964787568,
282.6615969581748,
687.8029426351461,
291.8267482228467,
689.4693337741774,
294.3263349313935,
681.1373780790211]],
'bbox_mode': <BoxMode.XYWH_ABS: 1>},
{'iscrowd': 0,
'bbox': [317.65581087783096,
702.8004628864272,
16.66391139031242,
20.829889237890598],
'category_id': 0,
'segmentation': [[328.487353281534,
702.8004628864272,
317.65581087783096,
715.2983964291616,
320.1553975863778,
723.6303521243178,
328.487353281534,
723.6303521243178,
333.48652669862776,
719.4643742767397,
334.3197222681434,
713.6320052901303]],
'bbox_mode': <BoxMode.XYWH_ABS: 1>},
{'iscrowd': 0,
'bbox': [282.6615969581748,
720.2975698462553,
15.830715820796797,
19.996693668374974],
'category_id': 0,
'segmentation': [[290.1603570838154,
720.2975698462553,
282.6615969581748,
727.796329971896,
282.6615969581748,
734.4618945280209,
290.99355265333105,
740.2942635146303,
298.4923127789716,
731.9623078194741,
298.4923127789716,
726.1299388328647]],
'bbox_mode': <BoxMode.XYWH_ABS: 1>},
{'iscrowd': 0,
'bbox': [320.1553975863778,
776.9548685733176,
20.829889237890598,
27.495453794015475],
'category_id': 0,
'segmentation': [[333.48652669862776,
776.9548685733176,
323.48817986444027,
781.1208464208958,
320.1553975863778,
793.6187799636301,
320.1553975863778,
798.6179533807239,
328.487353281534,
804.4503223673331,
338.4857001157215,
801.9507356587862,
340.98528682426837,
794.4519755331457,
340.98528682426837,
786.9532154075051]],
'bbox_mode': <BoxMode.XYWH_ABS: 1>},
{'iscrowd': 0,
'bbox': [302.65829062654973,
796.9515622416926,
13.331129112249982,
13.331129112249869],
'category_id': 0,
'segmentation': [[306.82426847412785,
797.7847578112082,
302.65829062654973,
802.7839312283019,
304.324681765581,
808.6163002149112,
309.3238551826747,
810.2826913539425,
313.48983303025284,
807.7831046453956,
315.9894197387997,
803.6171267978175,
315.1562241692841,
798.6179533807239,
310.99024632170597,
796.9515622416926]],
'bbox_mode': <BoxMode.XYWH_ABS: 1>},
{'iscrowd': 0,
'bbox': [326.82096214250276,
831.1125805918331,
17.497106959828102,
14.99752025128123],
'category_id': 0,
'segmentation': [[330.9869399900809,
846.1101008431143,
326.82096214250276,
833.61216730038,
333.48652669862776,
831.1125805918331,
340.15209125475275,
833.61216730038,
343.48487353281524,
840.2777318565049,
344.31806910233087,
844.4437097040831]],
'bbox_mode': <BoxMode.XYWH_ABS: 1>},
{'iscrowd': 0,
'bbox': [293.49313936187787,
773.6220862952551,
17.497106959828102,
16.663911390312478],
'category_id': 0,
'segmentation': [[298.4923127789716,
773.6220862952551,
293.49313936187787,
781.9540419904114,
293.49313936187787,
781.9540419904114,
297.659117209456,
790.2859976855676,
308.4906596131591,
789.452802116052,
310.99024632170597,
780.2876508513801,
310.99024632170597,
775.2884774342864]],
'bbox_mode': <BoxMode.XYWH_ABS: 1>},
{'iscrowd': 0,
'bbox': [555.1165481897832,
221.21342370639766,
21.66308480740622,
22.496280376921817],
'category_id': 0,
'segmentation': [[576.7796329971894,
226.2125971234914,
572.6136551496113,
243.70970408331948,
555.9497437592988,
234.54455281864762,
555.1165481897832,
225.37940155397578,
565.9480905934863,
221.21342370639766]],
'bbox_mode': <BoxMode.XYWH_ABS: 1>}]}