PBT with biome.text

[dcfidalgo]

Copied this over from an issue

This notebook is a guide on how to use ray tune's PBT with biome.text.

It took me quite some time to figure out the details and inner workings, but i think i understand it now and i am confident that the notebook does the right thing:
https://colab.research.google.com/drive/1vmt3D_2kq6VxE2-nlwsZl0YrN3nAhHp0?usp=sharing

Any comments on the notebook are more than appreciated!

dvsrepo frascuchon Do we still want to take the HPO stuff from gitlab to github? I would wait for this before trying to implement PBT in it, but there are also some points i still have to figure out ...

dvsrepo

Looks good!

Yes, we will release all HPO stuff here.

Please work with frascuchon to figure out how to integrate this into the HPOExperiment class and related methods. We should allow to "tune" anything related to the TrainerConfiguration with PBT.

Aside note: I've seen this also in hf tune examples, but I still don't see why would anyone want to tune num_epochs when you have early stopping at different levels (trainer early stopping and hpo process).

dcfidalgo

Update: After talking to frascuchon he thinks it would be better to first explore the possibility of creating a more suited trainer for our HPO stuff (keeping PBT in mind), then trying to implement the PBT stuff from above in our current HPOExperiment.

I will have a look at this in the coming weeks.

dvsrepo

Great. Given that we use ray tune and will be using more and more ray,
please consider raysgd with @frascuchon as an option to build on top for
the base trainer.