How to use exported NeMo model?

[averkij]

Hello, engineers.

I've trained and exported NeMo's PunctuationCapitalizationModel into the TorchScript format. I have used the export() method to do that, because this model extends Exportable class. Besides the forward method PunctuationCapitalizationModel implements others, — like the method which do all the stuff to transform the logits into probs, capitalize tokens, add punctuations and give the final result (strings with punctuation marks).

On the other hand, the exported model only has the forward method which gives me the logits. So the question is — how to use this exported model? Do I need to add all the transformation work by myself after the inference or I can add somehow these additional methods into the exported model?

I am planning to use this model in the Triton server using TorchScript backend.

[titu1994]

@borisfom any advice? As far as I know, we only export the models, not their surrounding infrastructure (so model, but not convenience methods like described above). So the user will have to implement the rest of the infrastructure to properly use the model in deployed environment right ?

[averkij]

@titu1994 @borisfom thanks for your involvement.

My plan for using this model is to tokenize the data, then send it to the exported model, and finally post-process the logits into the resulting data. Looks not the best way to do that (especially in Triton, where will be different backends for this single task, — TorchScript for inference and Python for post-processing), but maybe I'm wrong.

Would appreciate any advice.