ASR Training loss and WER is very low but validation loss and WER is very high

[mehadi92]

Try to finetune stt_en_conformer_ctc_large_ls.nemo with custom data. But getting validation loss is very high

Epoch 2:   0%|          | 0/8320 [00:00<?, ?it/s, loss=199, v_num=est1]
Validation: 0it [00:00, ?it/s]
Epoch 2, global step 14313: 'val_wer' reached 0.80320 (best 0.79610),
Epoch 3:   0%|          | 0/8320 [00:00<?, ?it/s, loss=172, v_num=est1]
Epoch 4:  98%|█████████▊| 8180/8320 [1:49:24<01:52,  1.25it/s, loss=111, v_num=est1]
Epoch 3, global step 18403: 'val_wer' reached 0.82236 (best 0.79610)
Epoch 4:   0%|          | 0/8320 [00:00<?, ?it/s, loss=147, v_num=est1]
Epoch 5:  98%|█████████▊| 8180/8320 [1:49:14<01:52,  1.25it/s, loss=70.8, v_num=est1]
Epoch 4, global step 22493: 'val_wer' was not in top 3
Epoch 5:   0%|          | 0/8320 [00:00<?, ?it/s, loss=111, v_num=est1]
Epoch 6:  98%|█████████▊| 8180/8320 [1:49:25<01:52,  1.25it/s, loss=35.7, v_num=est1]
Epoch 5, global step 26583: 'val_wer' was not in top 3
Epoch 6:   0%|          | 0/8320 [00:00<?, ?it/s, loss=70.8, v_num=est1]
Epoch 7:  97%|█████████▋| 8091/8320 [1:48:10<03:03,  1.25it/s, loss=14.9, v_num=est1]
Epoch 6, global step 30673: 'val_wer' was not in top 3
Epoch 7:   0%|          | 0/8320 [00:00<?, ?it/s, loss=35.7, v_num=est1]
Epoch 8:  98%|█████████▊| 8180/8320 [1:49:17<01:52,  1.25it/s, loss=5.94, v_num=est1]
Epoch 7, global step 34763: 'val_wer' was not in top 3
Epoch 8:   0%|          | 0/8320 [00:00<?, ?it/s, loss=14.7, v_num=est1]
Epoch 9:  98%|█████████▊| 8180/8320 [1:49:19<01:52,  1.25it/s, loss=2.57, v_num=est1]
Epoch 8, global step 38853: 'val_wer' was not in top 3
Epoch 9:   0%|          | 0/8320 [00:00<?, ?it/s, loss=5.94, v_num=est1]
Epoch 10:  98%|█████████▊| 8180/8320 [1:49:18<01:52,  1.25it/s, loss=1.56, v_num=est1]
Epoch 9, global step 42943: 'val_wer' was not in top 3
Epoch 10:   0%|          | 0/8320 [00:00<?, ?it/s, loss=2.57, v_num=est1]
Epoch 11:  98%|█████████▊| 8180/8320 [1:49:23<01:52,  1.25it/s, loss=1.46, v_num=est1]
Epoch 10, global step 47033: 'val_wer' was not in top 3
Epoch 11:   0%|          | 0/8320 [00:00<?, ?it/s, loss=1.56, v_num=est1]
Epoch 13:  39%|███▊      | 3213/8320 [43:11<1:08:39,  1.24it/s, loss=0.66, v_num=est1]
Epoch 11, global step 51123: 'val_wer' was not in top 3
Epoch 12:   0%|          | 0/8320 [00:00<?, ?it/s, loss=1.46, v_num=est1]
Validation: 0it [00:00, ?it/s]
Epoch 12, global step 55213: 'val_wer' was not in top 3
Epoch 13:   0%|          | 0/8320 [00:00<?, ?it/s, loss=0.843, v_num=est1]

Here is the trainer

trainer = ptl.Trainer(
    accelerator=accelerator,
    max_epochs=EPOCHS,
    accumulate_grad_batches=2,
    log_every_n_steps=10,
    check_val_every_n_epoch=1,
    strategy="ddp_find_unused_parameters_false",
    logger=False,
    enable_checkpointing=False,
    devices=-1,
    precision=16,
)

[titu1994]

Your train loss is almost zero - are you using spec augment ? Make sure the text can be correctly Tokenized by the model tokenizer - if you see ?? During training or inference then your model will not learn properly.
Finally check what is the score of the model on the validation set before training - maybe validation data is very different from training data.

You should also use an ASRSet model to fine-tune instead of LS - simply drop the _ls part of model name

[mehadi92]

@titu1994,
I have also taken 3.5 hours of data from my training set and made an evaluation on the saved model using this (3.5 hours) data using a script speech_to_text_eval.py and get WER/CER 6.42%/2.09%

[titu1994]

Then can you check what's the result of the original model on your val set ? The model has probably completely overfit the train set if wer is low then

[mehadi92]

@titu1994,
Do you mean stt_en_conformer_ctc_large_ls.nemo as the original model? because this was used as a pre-train model? If yes how can it be done? Because it has a different set of tokens for the English language. But I'm working on a different language.

Do you suggest calculating the result after changing the tokenizer of stt_en_conformer_ctc_large_ls.nemo and making predictions? or just make predictions using stt_en_conformer_ctc_large_ls.nemo model?

[titu1994]

If it's a different language then it can't be used. I dunno, seems your model is learning and overfitting your train set. Maybe your val wer is being incorrectly handled by the tokenizer or maybe the train and dev set have some other incompatibility but it's odd that even with spec augment your train loss is going so low

[mehadi92]

I was using nvcr.io/nvidia/nemo:23.02 for the bucketing dataset. which has the issue mention in #6191

[titu1994]

Ah that was a major bug. You can try 23.03 which contains the fix. https://catalog.ngc.nvidia.com/orgs/nvidia/containers/nemo