when I resume the checkpoint, the learning rate keeps zero ?

[asr-pub]

Hello, when I resume the checkpoint, why the learning rate keeps zero?
I cannot find what's wrong, can someone point it out?

train.py

    checkpoint_path = '2.ckpt'
    cfg.trainer.resume_from_checkpoint = checkpoint_path
    trainer = pl.Trainer(plugins=[NLPDDPPlugin()], **cfg.trainer)
    exp_manager(trainer, cfg.get("exp_manager", None))

    model = TokenClassificationModel(cfg.model, trainer=trainer)
    model.setup_loss(class_balancing=cfg.model.dataset.class_balancing)
    trainer.fit(model)

subset of config.yaml

  optim:
    name: adam
    lr: 5e-5
    weight_decay: 0.00

    sched:
      name: WarmupAnnealing
      # Scheduler params
      warmup_steps: null
      warmup_ratio: 0.1
      last_epoch: -1

      # pytorch lightning args
      monitor: val_loss
      reduce_on_plateau: false

this is the training log:

[titu1994]

You should use exp manager to resume training

[asr-pub]

if the 2.ckpt trained when set max_epochs = 3 that I resume training the learning rate still stays 0.000，Is there any other method, let lr reset?

[titu1994]

If your model has already finished training then it will use the minlr set in your optim config - which usually is a very small value but also defaults to 0 for many domains.

[asr-pub]

I set min_lr in sched, but the lr still stays 0.000

sched:
      min_lr: 5e-5
      name: WarmupAnnealing
      # Scheduler params
      warmup_steps: null
      warmup_ratio: 0.1
      last_epoch: -1

      # pytorch lightning args
      monitor: val_loss
      reduce_on_plateau: false

Is my nemo version(1.0.2) too old ?

[titu1994]

Yes it's too old. Also, doesnr matter if you change the config after you finish training. Only if the original config had min lr will it continue training.

[asr-pub]

Thank you , brother

[titu1994]

Fyi @MaximumEntropy