Citrinet model with LM to reduce the WER for microphone recorded audio

[kruthikakr]

Hi
I am using stt_en_citrinet_1024 model and able to get good transcript , I am using the recorded audios with microphone and WER is varying from 3.5% to 15%. This has names of person and place, how to include the words in the model.

any suggestions with following aspects

1. Preprocessing the audio
2. using LM model for decoding ( for citrinet is there any implementation in nemo)
3. Post processing steps say spell correction

looking for inputs .

[titu1994]

You could finetune Citrinet using the same Tokenizer on the specific domain (if there is sufficient data).

If you have some noise files, noise robust training, the same method as QuartzNet can be applied to Citrinet.

For preprocessing, there should be monochannel 16Khz wav files. We find that attempting signal denoising before inference will generally not do much better, and sometimes will do worse based on the artifacts introduced.

For language modelling with Citrinet (and BPE models in general), we plan to release code snippets to build custom KenLM model and run beam search through similar steps as the offline asr notebook. However there are some significant differences and we have not compiled a clean script for such a task yet. I will try to prioritize that in the coming weeks.

There is also Transformer based rescoring that can further boost offline WER reduction, though that pipeline is not ready yet.

@AlexGrinch is there any ETA (within some months?) that you expect to have the pipeline for transformer based rescoring?

[kruthikakr]

Thanks you for response.
We would try to write the script with LM for BPE models . Any inputs or leads are much appreciated.

[titu1994]

@VahidooX If you have a rough draft, could you create a gist and share here when it's ready ? We can clean it up in the actual PR

[kruthikakr]

Can someone please share some details on this ? waiting for response.

[VahidooX]

Created a PR for adding the feature of training and evaluating n-gram KenLM on top of BPE-base ASR models. It still needs the documentations. #2066

[VahidooX]

The PR to support N-gram LM for ASR models is merged : #2066
It can do grid search for beam search decoder's hyperparameters to fine-tune them. The scripts support both character-level and BPE-level models. You may read more here: https://github.com/NVIDIA/NeMo/blob/main/docs/source/asr/asr_language_modelling.rst

You need to install the beam search decoders and KenLM to use this feature.

[kruthikakr]

Thank you very much.

[kruthikakr]

Hi.. We are able to successfully implement LM model with Citrinet . Now to add custom words/vocabulary , we don't have sufficient training data , Can we add custom tokens with the KenLM process ?

[titu1994]

You can train your KenLM model with any kind of data. Just note that the model will predict UNK for such tokens so you might need to prepare your own script to generate the dataset for KenLM (based on the current script with modifications for custom words).

We don't support this in our scripts currently

[kruthikakr]

What are the other various ways we can try for adding the custom token to ASR .
Now the citrinet with LM model is still giving errors for few words , i think more traning is needed. But apart from traning , how to improvise the WER for online ASR with adding custom tokens.

Also Transformer based rescoring will be better than KenLM ?

[titu1994]

Post training the only way to add custom words is with a language model which has those tokens.

[kruthikakr]

Hi..after fine tuning the Citirnet LM ( 40M lines of LM librispeech corpus) we observe more deletions at the output , For the same audio with just citrinet without LM these deletions were not there.

What could be step to solve this , We are trying to achieve less WER for user audio samples.which is the way to proceed ?

1. By finetuning the Citrinet with more various types of data ( other than the 6 datasets ie 7000hr data ) ? Or
2. By finetuning the Citinet LM model with more data ?

How can we improvise the model for user audios ?

[titu1994]

It depends on the domain of text you are trying to transcribe - if it's significantly different from the academic datasets on which the model was trained on, it might be worth it to finetuned the ASR model.

If only text jargon is different, then getting more text data from the domain required in order to build a domain focussed language model would be easier.

[lucasjinreal]

Hi, all ,how do I download Citrinet and training on Chinese?

[titu1994]

We have a config for QuartzNet 15x5 trained on Mandarin as well as pretrained models of Citrinet trained on Mandarin. You can use from_pretrained() to get those Citrinet models and fine time then (they have an inbuilt vocabulary of 5200~ tokens from Mandarin).

[titu1994]

Please refer to the ASR Nemo documentation page for how to use pretrained models https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en/main/asr/results.html

You can get the config via
cfg = ASRModel.from_pretrained("xyz", return_config=True)
OmegaConf.save(cfg, path)

[titu1994]

And I don't think it's easy to run RIVA without docker.

[lucasjinreal]

@titu1994 Does it need some specific dependencies except python?

[titu1994]

For Nemo version, You have to install the Nemo toolkit. Instructions to install the entire library are there in the documentation and readme page.

For Riva, refer to their install guide.

[neso613]

PLease help me to understand Citrinet model optimization in RIVA
@titu1994 @kruthikakr