Tensorboard

[Zaker237]

Hallo @ALL

i'm right now using Haystack DensePassageRetreival to train a DPR model, i will like to see the evolution during the training to really know if my model is learning something. i tryed to bind a tensorboard to it but, i did see a way to do it so i will like to know if it's possible and if yes how can i do it. Thank you in advance.

[mayankjobanputra]

Hey @Zaker237,

I don't think we support any kind of training tracking today, but if you saw a way, maybe you can share your insights. I will open an issue for tracking support and will discuss it with the team. I will post a link to that issue here and maybe you can share your thoughts directly on that issue. If you also have a generic solution to support all kinds of trackers (i.e. tensorboard, wandb, etc), that would also be great to hear :)

[Zaker237]

Hey @mayankjobanputra

Thank for your reply. I think it will really be a good idea. Since tracking how the training is going on can sometime help.
As idea i think the first think to do ist to give to the user the ability to give a logs folders when the model i initialised or when he starts the trainning. and then writte the logs in that folders during the training in the right format so that tensorboard can just be start in that folder.

[mayankjobanputra]

Hi @Zaker237,

I had opened an issue regarding tracking... It turns out that Haystack does support tracking via MLFlow. There are more details in the issue, please feel free to check it out :)

[Zaker237]

@mayankjobanputra
i saw that haystack has a tracker (haystack.utils.experiment_tracking.Tracker), i wanted to ask if there is a way to use it to have some plot of the taining? plot of the behavior of the loss during a DPR training

[mayankjobanputra]

@Zaker237 I am afraid I don't see the option to do that (at least with the latest Haystack version) for DPR training, but I will talk to the team and get back to you.