CPU Memory leak in " speech_to_text_eval.py"

[mehadi92]

I try to run this script with 30G of RAM but memory increased over time and get Out of memory

python speech_to_text_eval.py \
    model_path=my_best_model.nemo \
    pretrained_name=null \
    dataset_manifest=/nemo_data/nemo_mnifest.json \
    output_filename=output.json \
    batch_size=32 \
    amp=False \
    use_cer=False \
    num_workers=1

[titu1994]

That script is going to wait until it completed all transcriptions before returning result. If you have set decoding strategy to return hypothesis or tensors then it will consume memory for each sample.

Easiest thing to do is split the manifest and call if in a loop

[mehadi92]

Is there any options to set return hypothesis=False from this script?

[titu1994]

Not really. I suppose you could modify the transcribe speech script itself but that would take a little effort

[mehadi92]

@titu1994 could you please guide me on where to change? I could not see any option to set hypothesis=False in the transcribe speech script - Thanks

[titu1994]

Set this line to false https://github.com/NVIDIA/NeMo/blob/main/examples/asr/transcribe_speech.py#L228

And return the transcriptions without writing them by setting this to true - https://github.com/NVIDIA/NeMo/blob/main/examples/asr/transcribe_speech.py#L344

Youll get back list of str that you'll need to manually write into manifest. There is 1:1 correspondence between manifest line number and transcript index.

You might want to test your code on just a few files before running it fully.

I still suggest not using a single large manifest like this and to just split the manifest