Possible to convert NeMo T5 back to Huggingface format?

[Lauler]

Hi

An important consideration for choosing a framework to train on can sometimes include whether it can be converted to a format that makes it shareable, accessible and usable for a larger proportion of users. With this I'm of course implying converting to a Huggingface compatible model once the training is finished.

For some model implementations in Megatron (see this issue with BERT) it has not been straightforward to perform this conversion due to rearrangements in the transformer layers.

My question has to do with your T5 LLM training implementation. It has a script to convert from Huggingface to NeMo. Does this mean that we can also do the reverse conversion back to Huggingface, once we have trained a model using NeMo? Do you have any examples of this being done?