Custom keypoint segmentation - optimization problems

[ChristianPritz]

Hi all,

I am using detectron2's R101-FPN for custom keypoint segmentation. I am trying to fit 30 keypoints (midline of a deformable tube) on a training dataset that consists of several 10Ks of instances. I am getting quite acceptable results for the fit of the midline , however I can't get the training the loss lower than 0.6. I would need to improve the overall accuracy of the fit.

Here is how I do the training:

`num_keypoints = len(keypoint_names) #30
output_path = '/home/workstation/Documents/trained models/keypoint_30/'

cfg = get_cfg()
cfg.MODEL.DEVICE = "cuda"
cfg.merge_from_file(model_zoo.get_config_file("COCO-Keypoints/keypoint_rcnn_R_101_FPN_3x.yaml"))
cfg.OUTPUT_DIR = output_path
cfg.MODEL.WEIGHTS = model_zoo.get_checkpoint_url("COCO-Keypoints/keypoint_rcnn_R_101_FPN_3x.yaml")

cfg.MODEL.WEIGHTS = model_path
cfg.DATASETS.TRAIN = ("keypoint_train",)
cfg.DATASETS.TEST = ("keypoint_test",)
cfg.DATALOADER.NUM_WORKERS = 2
cfg.SOLVER.IMS_PER_BATCH = 10
cfg.SOLVER.BASE_LR = 0.00025
cfg.SOLVER.MAX_ITER = 40000
cfg.MODEL.ROI_HEADS.BATCH_SIZE_PER_IMAGE = 512
cfg.MODEL.ROI_HEADS.NUM_CLASSES = 1
cfg.MODEL.RETINANET.NUM_CLASSES = 1
cfg.MODEL.ROI_KEYPOINT_HEAD.NUM_KEYPOINTS = num_keypoints
cfg.SOLVER.STEPS = (int(cfg.SOLVER.MAX_ITER0.5),int(cfg.SOLVER.MAX_ITER0.7),int(cfg.SOLVER.MAX_ITER*0.9) )
cfg.TEST.KEYPOINT_OKS_SIGMAS = np.ones((num_keypoints, 1), dtype=float).tolist()
os.makedirs(cfg.OUTPUT_DIR, exist_ok=True)
trainer = DefaultTrainer(cfg)
trainer.resume_or_load(resume=False)
trainer.train()`

If you have any hints (code snippets) on or can point me towards good template of custom keypoint segmentation that addresses the following issues I would be very happy:

• Proper validation keypoints (still failing at it)
• hyper parameter optimization
• dataset augmentation

Any hint is greatly appreciated.

Cheers

Christian