-
Notifications
You must be signed in to change notification settings - Fork 0
Open
Description
Currently, the COCO metrics are a bit different than what we get with the official cocoapi
. While the matching algorithm and other calculations should be the same, there are some differences in our implementation:
- We do not do anything special with
"iscrowd":1
ground truths - We do not use the
area
attribute from COCO json ground truths (which is obtained from the segmentation mask): instead, we multiply the width and height of the bounding box to obtain the area - I do not use
maxDets
- that is, I do not limit the maximum number of detections per image. This limiting is usually done during inference (usually "fused" with NMS), or I might additionally add it to IO constructors forDetections
- but not as part of the evaluation
However, even if remove these three things (as is done in our integration example), the results still differ. So the best thing would be to narrow the differences down:
- take only 1 size range, one IoU threshold and one class, and make sure we still have the difference.
- Then, try to get the difference down to one image (it should be possible)
- For this image, evaluate what went wrong - manually inspect the matches that were made
Hopefully this reveals some un-noticed difference in our matching algorithm, as compared to COCO, and then this issue will be fixed
Metadata
Metadata
Assignees
Labels
No labels