How to gracefully stop Trials?

[utahman]

I'm running AutoModel in Jupyter notebook. Since I'm not sure how many trials I want to wait (depends on the observed progress) I set number of trials to big number. But if I just click stop after say Trial 123 and call export_model() it fails on exception:

OSError: SavedModel file does not exist at: .\TestName3\best_model/{saved_model.pbtxt|saved_model.pb}

Is there a way to stop trials and be able to export current best model?

[haifeng-jin]

I don't have a solution for this yet since AutoKeras would do a final training of the best model after all the trials. If you don't finish all the trials, the final training cannot be done.
Therefore, you cannot export the best model.

[psaks]

Hi @haifeng-jin ,
I am not quite sure this answer addresses the question. Until you have performed the final training, the best model would be considered the one with the lowest val_loss. It would be tremendously helpful to persist that model across trials after every epoch where a new best have been found. I would think of it as a generalisation of the ModelCheckpoint callback to AutoKeras.
After trials have been stopped, the model with best val_loss could be trained further. This would also mitigate the issue related to #920
Cheers,