Logging validation predictions to W&B | ASR Training

[Oscaarjs]

The experiment manager easily provides the opportunity to log a variety of metrics with Weights & Biases.
However, during training, if log_predictions=True, pytorch lighting/exp manager also outputs the actual predictions for a few examples. E.g. like:

[NeMo I 2022-11-15 09:38:01 rnnt_wer_bpe:391] reference :motorväg
[NeMo I 2022-11-15 09:38:01 rnnt_wer_bpe:392] predicted :motorväg

Is it possible, in a simple way to log these predictions to W&B as well?

Currently I'm configuring the experiment manager in the following way:

os.environ.pop('NEMO_EXPM_VERSION', None)

config = exp_manager.ExpManagerConfig(
    exp_dir=f'PATH',
    name=f"experiment-name",
    checkpoint_callback_params=exp_manager.CallbackParams(
        monitor="val_wer",
        mode="min",
        always_save_nemo=True,
        save_best_model=True,
        save_top_k=20,
        every_n_epochs=1,
        save_nemo_on_train_end=True        
    ),
    resume_if_exists=True,
    resume_ignore_no_checkpoint=True,
    version='version_0',
    create_tensorboard_logger=False,
    create_wandb_logger = True,
    wandb_logger_kwargs = {'project': 'project-name', 'name': 'name'}
)

config = OmegaConf.structured(config)

logdir = exp_manager.exp_manager(trainer, config)

Thanks in advance!

[VahidooX]

Currently we don't have this feature, not sure if there is an easy way to log text predictions into the w&b.
May I ask why you need it?
You can see the output logs in the w&b which contains these predictions. Also note that these samples is just one sample per training batch.

[cinjon]

How do I turn this off? I tried disabling it by setting model.log_prediction = false and that did not work. I also tried model.log_predictions = false, also didn't work.

[mehadi92]

@cinjon do you able to disable logs by setting model.log_prediction = false