Large Number of Deletions

[picheny-nyu]

I have some noisy low quality data; I put the data through a few Nemo speech recognition models and see a lot of deletions (relative to, say, wav2vec and whisper). Is there some setting I can adjust to reduce the number of deletions? I am using the models out of the box without any fine tuning.

Thanks
Michael Picheny

[titu1994]

Which model are you using from NeMo? You can try to use Conformer or Fast Conformer transducer or hybrid model to get a baseline

[titu1994]

Whisper has been trained on a lot of private data with music and noise, so it is more robust to severely degraded audio quality. It seems CTC models are close to the whisper model, and to improve their wer you can try using beam search with them.

Finetuning tutorial is available here - https://colab.research.google.com/github/NVIDIA/NeMo/blob/stable/tutorials/asr/ASR_CTC_Language_Finetuning.ipynb

You can write a short script based on that to do the finetuning with your requirements.

[titu1994]

Also, fast conformer hybrid can be run in CTC mode - you can use the transcribe speech.py in examples with decoder_type='ctc' here - https://github.com/NVIDIA/NeMo/blob/main/examples/asr/transcribe_speech.py#L157

Or manually load the model and call model.change_decoding_strategy(None, decoder_type="ctc")

[picheny-nyu]

Thanks so much. I suspect that rnnt fine tuning would also help a lot and I would be happy to try and report back, would you have a similar pointer? Thanks again so much for your help!

[titu1994]

Right RNNT might do fine with some finetuning.

RNNT finetuning, and hybrid model finetuning is almost exactly following the same recipe as the CTC finetuning. You should be able to follow those steps for all 3. If you need any help, you can ask here or in a issue and we will help.

You'll have to call model.change_vocabulary() with the new tokenizer (if you use a new Tokenizer, otherwise it's better to continue to use the inbuilt one).

[picheny-nyu]

The fine-tuning tutorial you point me to is for Japanese. Since my data is English I don't need most of the steps. Would you happened to have a pointer to script that just fine tunes an English model from English data? It would save me some time filtering out the unnecessary steps. If not, I can struggle through it. Thanks again!