Fastest way to infer BPE RNNT Conformer

[roman-vygon]

Hi!
I've trained a conformer transducer model using BPE with this config, now I would like to infer it on a large dataset (~1M of ~5s audios). Currently, I'm using this script for transcription, passing the dataset manifest to it. I've tried increasing batch size to 256 with no increase in speed.
It takes ~220hours to transcribe which is too much and GPU usage seems to be 0% all the time, while VRAM usage is approximately 50%.
I have 4GPUs that I can use.
Is there a way to (1) use all 4 gpus for transcription with this model and (2) increase gpu usage and speed of inference?

[VahidooX]

That script should use one GPU. Would you please check the value of device variable here:
https://github.com/NVIDIA/NeMo/blob/main/examples/asr/transcribe_speech.py#L90
It should something like cuda:0.

We are working on adding a multi-gpu multi-node transcribe method and would be added hopefully in the following weeks.
In the meantime, you may simply divide your manifest file into four files, run each one separately on a gpu by using CUDA_VISIBLE_DEVICES or by running "python transcribe_speech.py cuda=GPU_ID ...".

[roman-vygon]

Thank you for the response! I will do that to use all 4 GPUs.
However, I think there is a major problem with transcribe method right now cause it creates a dataloader with 1 worker only, that was the cause of my GPU usage being 0% almost always. For a quick hack, I added the "num_workers" parameter here, however, there should be a way to pass this as an argument of transcribe function.

[VahidooX]

The PR to add multi-gpu/multi-node inference for ASR models: #3017