How to resume training?

[mmoollllee]

Is your feature request related to a problem? Please describe.
After training was interrupted I want to pick up the process and continue training where it stopped.

Describe the solution you'd like
Some parameter to define the correct folder where progress was saved to.

[ambroiseb]

Hi @mmoollllee.
Stricly resuming a training is not implemented for most of our training algorithms.
But you can restart a new training (from epoch 0) with as pretrained model the one from interrupted training. Indeed, if you have left the algorithm running long enough to make a validation pass, you should find in your "output_folder" (a parameter of the training algorithm) the weights and config file from the best model.
To do so, you need to set some parameters:

• "model_weight_file" = path_to_your_weight_file_from_interrupted_training.pth
• "config_file" = path_to_your_config_file_from_interrupted_training (.yaml, .py... depending on the algorithm)
• "use_custom_model" = "True"
  Hoping it helps, let me know if something is unclear.
  Best regards

[imtheclown]

Hi @ambroiseb, I am new to Ikomia and I am not familiar with the technicalities. I would like to ask how could I use the answer above to restart training in train_yolov7? I've read the documentation of train_yoloV7 and I did not found the parameters stated above. Thank you.

[ambroiseb]

Hi @imtheclown
For now resuming a training is not possible with train_yolo_v7.
I'll let you know when it's fixed.
Best regards