Add validation accuracy during training

[fgraffitti-cyberhawk]

I'm trying to add some information on the validation dataset during training.
While adding losses is quite straightforward (running the model in training mode on the items of the validation data loader returns a dictionary with the losses), I am finding quite tricky to calculate the validation accuracy.

For maskRCNN for example, it seems to me that the training accuracy is computed in BaseMaskRCNNHead.forward using the mask_rcnn_loss function, but I can't find a way to use this (or a similar) approach for calculating the validation accuracy.

Anyone has an idea on how to approach this problem?

[aymanaboghonim]

you can add a val hook like this code
`class ValidationLoss(HookBase):
def init(self, cfg):
super().init()
self.cfg = cfg.clone()
self.cfg.DATASETS.TRAIN = cfg.DATASETS.TEST
self._loader = iter(build_detection_train_loader(self.cfg))

def after_step(self):
    data = next(self._loader)
    with torch.no_grad():
        loss_dict = self.trainer.model(data)
        
        losses = sum(loss_dict.values())
        assert torch.isfinite(losses).all(), loss_dict

        loss_dict_reduced = {"val_" + k: v.item() for k, v in 
                             comm.reduce_dict(loss_dict).items()}
        losses_reduced = sum(loss for loss in loss_dict_reduced.values())
        if comm.is_main_process():
            self.trainer.storage.put_scalars(total_val_loss=losses_reduced, 
                                            **loss_dict_reduced)

val_loss = ValidationLoss(cfg) `

then register the hook like this
trainer.register_hooks([val_loss]) trainer._hooks = trainer._hooks[:-2] + trainer._hooks[-2:][::-1] trainer.resume_or_load(resume=False) trainer.train() to start training and get val metrics

[fgraffitti-cyberhawk]

hey @aymanaboghonim , thanks for your reply, I'll give it a try later. I have a question regarding your code though.
It looks like your code allows you to calculate the validation losses but not the validation accuracy, is thins correct?
I already have implemented validation losses in my code, and now I was looking at implementing the accuracy on the val set as well.

[aymanaboghonim]

sorry for replaying late. yes, it is about loss not accuracy.
have you coded the val accuracy or not yet ??

[fgraffitti-cyberhawk]

no, I haven't written any code for accuracy yet, got busy working on other stuff :/ I'm planning to do that at some point in the future but haven't had a chance yet :)