Changing ViTDet patch size hurt RPN cls loss

[austinmw]

Hi, I tried changing the ViTDet patch size from 16 to 8, but this hurts the RPN cls loss significantly. Is there another configuration I also need to change when changing the patch size?

[HosseinSheikhi]

The FPN implementation is kinda ad hoc to patch_size = 16. Lets assume an 1024x1024 image. also assume FPN scales = [4,2,1,0.5]. patch size = 16, leads to pyramids of resolutions [256,128,64,32]. (There is also one more top_block which adds resolution 16, lets ignore). If patch size = 8, having same scales leads to pyramids of resolutions [512,256,128,64]. so maybe you would better to change scales to [2,1,0.5,0.25].