Are ASR models trained using mixed precision also suitable for half precision inference?

[whrichd]

I trained a model using bf16. But after I see that it's still saved it's saved in fp32. Will this model still be able to provide computation performance improvements in inference? Does it need to be converted back to bf16?

I see in the documentation and other sources that mixed precision is only mentioned for training speed boost.

[titu1994]

Yes, models trained with fp16 / bf 16 can do inference in fp32 or the original dtype.

There are some cases (rare), where a model trained in bf16 can't do inference in fp16 or vice versa, but that can be fixed with a little finetuning on the required dtype.

Some super large models may also have difficulty if they are trained on fp32 and tried to do inference in fp16, but bf16 will work most of the time

[whrichd]

Thanks. How do I export the trained model (saved in fp32) back to bf16? So from what I've learned I can do inference with it using autocast, but what to do to change the dtype of the model itself? Then to export to ONNX format for example? Tried model.bfloat16() but the resulting dtype is still fp32.

[titu1994]

Model.to(torch.bfloat16) is not recommended, that will convert even weights that aren't meant to support bfloat to that type (batch norm counters etc)

Use The export script in NeMos script dir to export model, that handles dtype for you

[whrichd]

Great! Thanks.