Resuming from .ckpt insreases my val wer by alot

[jomzxc]

As stated by other discussions here, I used resume_if_exists to resume my previous training, however it resulted in a sudden spike of val_wer as you can see in the image.

For context here is the full command that i used:

/home/sabat/NeMo/examples/asr/asr_transducer/speech_to_text_rnnt_bpe.py --config-path=/home/sabat/NeMo/examples/asr/conf/contextnet_rnnt --config-name=config_rnnt_bpe model.train_ds.manifest_filepath=/home/sabat/NeMo/data/train_clean_100.json model.validation_ds.manifest_filepath=/home/sabat/NeMo/data/dev_clean.json model.test_ds.manifest_filepath=/home/sabat/NeMo/data/test_clean.json model.tokenizer.dir=/home/sabat/NeMo/tokenizers/tokenizer_spe_bpe_v70 model.tokenizer.type=bpe trainer.devices=-1 trainer.accelerator=gpu trainer.max_epochs=100 trainer.precision=16 exp_manager.create_wandb_logger=True exp_manager.wandb_logger_kwargs.name=rnnt_bpe_v70_clean100 exp_manager.wandb_logger_kwargs.project=Cleft2Speech ++exp_manager.resume_if_exists=True +exp_manager.explicit_log_dir=/home/sabat/NeMo/nemo_experiments/ConvRNNTBPE5x1/2025-03-18_07-06-32/ ++exp_manager.create_early_stopping_callback=True ++exp_manager.disable_validation_on_resume=False

[ashors1]

Hi, could you share some details about your environment? Are you using a NeMo container? If so, which one?