Skip to content

Make COCO metrics replicate official cocoapi better #37

@tadejsv

Description

@tadejsv

Currently, the COCO metrics are a bit different than what we get with the official cocoapi. While the matching algorithm and other calculations should be the same, there are some differences in our implementation:

  • We do not do anything special with "iscrowd":1 ground truths
  • We do not use the area attribute from COCO json ground truths (which is obtained from the segmentation mask): instead, we multiply the width and height of the bounding box to obtain the area
  • I do not use maxDets - that is, I do not limit the maximum number of detections per image. This limiting is usually done during inference (usually "fused" with NMS), or I might additionally add it to IO constructors for Detections - but not as part of the evaluation

However, even if remove these three things (as is done in our integration example), the results still differ. So the best thing would be to narrow the differences down:

  • take only 1 size range, one IoU threshold and one class, and make sure we still have the difference.
  • Then, try to get the difference down to one image (it should be possible)
  • For this image, evaluate what went wrong - manually inspect the matches that were made

Hopefully this reveals some un-noticed difference in our matching algorithm, as compared to COCO, and then this issue will be fixed

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions