lowest reasonable learning rate for mask r-cnn

[amida47]

I am training a mask rcnn model from scratch using the config at Miscs/mask_rcnn_R_50_FPN_3x_gn.yaml , I started the training with lr set to 0.02 and trained until it hit a plateau, I dropped it then to 2e-3 , continued the training until again it hit a plateau, I tried after this dropping lr to 2e-4, 2e-5 and 2e-7 but the model didn't show any sign of improvement it just stayed at the same values, so do you think I should keep dropping the lr or just stop there ?

ps: I am using a custom medical dataset, the input images have 4 channels