Different performance in training-test and testing

[HuangJunJie2017]

Checklist

1. The bug has not been fixed in the latest version.

Describe the bug
Testing the trained models with tools/test.py will generate a result which is different from the testing result during training process.
For example, the testing AP after 210 epoch is 73.0 (reported in the training log during the training process). However, if i test the saved checkpoint of 210 epoch with tools/test.py, the testing AP is 73.2.
It seem that the different is become bigger gradually during the training process.

[innerlee]

Is this a val vs test difference?

[HuangJunJie2017]

At the beginning of the training process, the performance between them is exactly the same. (1st epoch)
but at the end(210epochs), they are different. It is very strange. I wonder if there is something wrong with the BN?

[innerlee]

I don't know the answer but it sounds possible. You can try syncbn to help diagnose. ref https://github.com/open-mmlab/mmaction2/blob/a33a188003ee87a852326b803188954eb6b5545e/configs/recognition/tsn/tsn_r50_1x1x8_50e_sthv1_rgb.py#L9

[HuangJunJie2017]

Emmm, may be this is the point. From another perspective, I test the models trained with only one gpu, and the results are exactly the same.