Inconsistent performance across runs?

[maxeduc]

I was trying out training with conformer_transducer_bpe_streaming.yaml on Librispeech. I'm running the default settings except with batch size 8 (setting accumulate_grad_batches to 2 to compensate) and fused_batch_size 4 to get it to not OOM. (seems weird that would be necessary on a 24GB card, but that's beside the point).

My first run failed after about 6k steps due to some CUDA kernel timeout error (which is probably another bug, but that's also beside the point). I was hitting about 2it/s (or 1it/s in terms of the effective batch). My second run has been ~1.24it/s (.62it/s effective batch) consistently. I changed nothing the second time around. Does anyone know why someone would see a sudden decrease in performance between runs with no changes to anything else, or external programs running, or anything?

Loss graph looks the same (ignore the exploding loss in the second run, that's another separate issue I guess):

Training time is distinctly different:

The hyperparameters and computer load are seriously exactly the same between runs.

[titu1994]

RNNT always uses quite a bit of memory, our large models take around 32 GB to train on V100s at BS 8 with fp32. Fp16 can increase the batch size slightly.

Never got a cuda kernel timeout so dunno how to fix that. Maybe a one off thing.

Now to solve the actual question the first thing to ask is are you using cudnn auto tuning ? If so, disable it. It exhausts more memory for asr models and can cause inconsistency in runs if different kernel is picked.

[titu1994]

@VahidooX if you hand some ideas, can you suggest ?

[maxeduc]

Thanks for the info. Makes sense that RNNT would be expensive.

As for cudnn autotuning, is torch.backends.cudnn.benchmark the setting you're referring to? If so, this:

trainer = pl.Trainer(**cfg.trainer)
exp_manager(trainer, cfg.get("exp_manager", None))
asr_model = EncDecRNNTBPEModel(cfg=cfg.model, trainer=trainer)

# Initialize the weights of the model from another model, if provided via config
asr_model.maybe_init_from_pretrained_checkpoint(cfg)

print("Autotuning on? ", torch.backends.cudnn.benchmark)

gives me

Autotuning on? False

I'm also getting NaN with these runs after a while, but the config documentation has some things to try tweaking for that. I think I also read FP16 can cause this? (which I'm running)

[titu1994]

Which version of Nemo are you using ? 1.11 is necessary for fp16 Conformer. The flag for cudnn benchmarking is set in the PTL Trainer flag as benchmark: false. It won't show up before you call trainer.fit().

[titu1994]

Since your global batch size is too small, try to use grad accumulation to get around batch size of 128 or 256

[maxeduc]

I installed from source on the main branch about a week ago, so it should be past 1.11. Printing out the flags being sent to the trainer I see benchmark: false.

For the record my runs have been pretty consistent (~1.25it/s) with my recent attempts and I haven't been able to replicate the faster run since. That's also the only run that's given me the CUDA kernel timeout, so maybe this was just a CUDA bug or some driver weirdness.

Will try increasing the grad accumulation as well. Thanks again for the fast responses!