Use ASR Adapters with Citrinet model

[manuel3265]

Hello, is there someone who can help me use the adapters with models other than the conformer ones. I want to use the adapters with the citrinet models and would like to know if it is possible and how I could do it.

[titu1994]

Citrinet is compatible with adapters and followed the same steps. You may need to modify the optimization hyper parameters slightly but it should still work.

Can you explain in detail what you tried and what was the result.

[manuel3265]

@titu1994 Too bad for you, the Adapters training functionality works but it doesn't generate good results. This is the quick implementation I did.

Citrinet_with_adapters

Sorry if it's a bit messy, but it shows the results of the model returning a wer of 1.0 after training.

Thanks

[titu1994]

optim:
  name: novograd
  lr: 0.05
  betas:
  - 0.8
  - 0.25
  weight_decay: 0.001
  sched:
    name: CosineAnnealing
    warmup_steps: 1000
    warmup_ratio: null
    min_lr: 1.0e-05
    last_epoch: -1

As I said, optimization parameters need to be changed. The base LR for Novograd is 0.05, your notebook sets it as 0.1, which I higher than when even training with Global Batch Size of 1024 of more.

I would advise using much lower LR, and preferable switch to Adam for adapter optimization. It's usually much less to tune that way.

Basically set optim.name=adamw, betas=(0.9, 0.98), and LR=0.001. For Adam based optimizer, modify LR is sufficient.

[titu1994]

I assume you're using the parameters from the tutorial as is, but remember that Citrinet and Conformer are two very different models and use different optimizers as their default. Citrinet is Novograd + Cosine decay, Conformer is AdamW + Noam. Noam scheduler takes LR and multiplies it with the LR from the schedule itself - so you can't just drop the same values for other optimizer and scheduler and expect it to work c

Adam + Cosine is usually simpler to tune so use that for your experiments.

[manuel3265]

@titu1994 Thank you for your comments and your time. I have been trying to modify the LR for quite some time and always come to the same result. The only difference is that if I increase the LR, it reaches the WER of 1.0 faster, with a lower LR, it takes longer to reach it.

[manuel3265]

Could you please give me a suggestion?

[titu1994]

@manuel3265 This was a fun experiment. There are a few issues with the script you had written and an interesting side effect of Citrinet old checkpoints itself. A modified copy of your colab is here - https://colab.research.google.com/drive/1plDHQVucsNr7QqMRCl_EY5E1X0HBrI7z?usp=sharing

1. Citrinet had a breaking change regarding the calculation of Squeeze Excitation in NeMo 1.6 or so. This caused older checkpoints to dramatically worsen in performance if there was significant padding. Older checkpoints were not updated to the new SE training code. The checkpoints you are using were all trained using the problematic SE implementation, so the moment you tried to train this model on new dataset, it had significant difficulty in optimization during training. The way to "fix" this is to use train slowly, and for longer. I changed LR to 0.001, adamw, and increased warmup to 100 steps.

Still, the above model got to around 75% WER after 4 epochs. So what was the problem?

2. Overfitting - yep, turns out adapters can overfit.. weird concept, given they have so small number of parameters. But I changed your adapter config to add dropout. 50% dropout immediately got the WER to go up to only 10% after 4 epochs rather than 75%. So i increased dropout in steps of 10 up to 90%. At 80-90 %, the model WER actually went down to 5.4% after 4 epochs of training. Then training for 20 epochs got it to about 5% WER.

3. Size of adapter - I dunno where you got that code for adapters, maybe from the PR itself. The ASR tutorial does not have such config. Its hidden dimension of 5 was too small to train reasonably. I bumped it to 32, so around 450k params, and then trained for 20 epochs and got WER down to 3.5%.

Theres obviously other optimization you can do, reduce dropout, increase LR, reduce adapter dim, increase adapter dim etc. I will leave that exploration to you.

Just know that adapters in ASR are relatively new (at least in NeMo). There arent so many empirically well defined topics, and each model is bound to have some challenges. But the technique does work well when it works, bringing the WER down from 7% to 3.5% in around 10 mins of training.

[manuel3265]

Wow, thank you very much for your time and dedication, I will study what you tell me and the notebook, I will continue with the tests, modifying the parameters. Again I thank you.

[manuel3265]

Hello @titu1994 , I have been testing the training of the adapters for the citrinet model, but the training breaks the original training. Adapter training causes the WER for the original training to be affected. I have trained with numbers and in this case, the WER decreases, but if I do the test for other types of data such as names, the WER increases. Do you know what could be due?

[titu1994]

Yes this is expected. Adapter does not stop the model from degrading on the original domain, only it allows you to

1. Train a adapter weights efficient without full model cost
2. Allow to disable adapter to get back original model.

In step 1 - unless you train with the original dataset + your new dataset, you will still face catastrophic forgetting.