How to pinpoint issue in data

[itzsimpl]

I'm having issues with my ASR data, which is braking my training. I'm getting the following error:

ValueError: normalize_batch with `per_feature` normalize_type received a tensor of length 1. This will result in torch.std() returning nan. Make sure your audio length has enough samples for a single feature (ex. at least `hop_length` for Mel Spectrograms).

Is there any easy/fast way to pinpoint the files that are causing this issue? E.g. how could one dump the files that constitute the current batch?

[titu1994]

You're probably using multi channel audio. Normalize to monochannel, 16 khz audio

[titu1994]

As to how to diagnose the issue, as in all pytorch you can create the data loader with setup_*_data and then manually loop over the data loaders batch. The above error will pop up when samples are not monochannel but passed through the WaveformFeaturizer.

If this error occurs at the preprocessing module, you'll need to also call model.preprocessor with the correct arguments from the data loader to detect the problematic batch.

[itzsimpl]

@titu1994 thanks for a quick reply. My initial guess was too, that there may be some multi channel audio, but I checked and there seems to be none, all data is mono channel, 16kHz. I guess I'll need to recheck. Any ideas what could be wrong if all data is indeed mono, 16kHz, 16bit?

The error seems to originate from the preprocessing module. The full stack trace is:

Traceback (most recent call last):                                                                                                                                                                                                   
  File "examples/asr/asr_ctc/speech_to_text_ctc_bpe.py", line 95, in <module>                                                                                                                                                        
    main()                                                                                                                                                                                                                           
  File "/opt/conda/lib/python3.8/site-packages/nemo/core/config/hydra_runner.py", line 104, in wrapper                                                                                                                               
    _run_hydra(                                                                                                                                                                                                                      
  File "/opt/conda/lib/python3.8/site-packages/hydra/_internal/utils.py", line 377, in _run_hydra                                                                                                                                    
    run_and_report(                                                                                                                                                                                                                  
  File "/opt/conda/lib/python3.8/site-packages/hydra/_internal/utils.py", line 214, in run_and_report                                                                                                                                
    raise ex                                                                                                                                                                                                                         
  File "/opt/conda/lib/python3.8/site-packages/hydra/_internal/utils.py", line 211, in run_and_report                                                                                                                                
    return func()                                                                                                                                                                                                                    
  File "/opt/conda/lib/python3.8/site-packages/hydra/_internal/utils.py", line 378, in <lambda>                                                                                                                                      
    lambda: hydra.run(                                                                                                                                                                                                               
  File "/opt/conda/lib/python3.8/site-packages/hydra/_internal/hydra.py", line 111, in run                                                                                                                                           
    _ = ret.return_value                                                                                                                                                                                                             
  File "/opt/conda/lib/python3.8/site-packages/hydra/core/utils.py", line 233, in return_value                                                                                                                                       
    raise self._return_value                                                                                                                                                                                                         
  File "/opt/conda/lib/python3.8/site-packages/hydra/core/utils.py", line 160, in run_job                                                                                                                                            
    ret.return_value = task_function(task_cfg)                                                                                                                                                                                       
  File "examples/asr/asr_ctc/speech_to_text_ctc_bpe.py", line 87, in main                                                                                                                                                            
    trainer.fit(asr_model)                                                                                                                                                                                                           
  File "/opt/conda/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 696, in fit                                                                                                                               
    self._call_and_handle_interrupt(                                                                                                                                                                                                 
  File "/opt/conda/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 650, in _call_and_handle_interrupt                                                                                                        
    return trainer_fn(*args, **kwargs)                                                                                                                                                                                               
  File "/opt/conda/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 735, in _fit_impl                                                                                                                         
    results = self._run(model, ckpt_path=self.ckpt_path)                                                                                                                                                                             
  File "/opt/conda/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1166, in _run                                                                                                                             
    results = self._run_stage()                                                                                                                                                                                                      
  File "/opt/conda/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1252, in _run_stage                                                                                                                       
    return self._run_train()                                                                                                                                                                                                         
  File "/opt/conda/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1283, in _run_train                                                                                                                       
    self.fit_loop.run()                                                                                                                                                                                                              
  File "/opt/conda/lib/python3.8/site-packages/pytorch_lightning/loops/loop.py", line 200, in run                                                                                                                                    
    self.advance(*args, **kwargs)                                                                                                                                                                                                    
  File "/opt/conda/lib/python3.8/site-packages/pytorch_lightning/loops/fit_loop.py", line 271, in advance                                                                                                                            
    self._outputs = self.epoch_loop.run(self._data_fetcher)                                                                                                                                                                          
  File "/opt/conda/lib/python3.8/site-packages/pytorch_lightning/loops/loop.py", line 200, in run                                                                                                                                    
    self.advance(*args, **kwargs)                                                                                                                                                                                                    
  File "/opt/conda/lib/python3.8/site-packages/pytorch_lightning/loops/epoch/training_epoch_loop.py", line 203, in advance                                                                                                           
    batch_output = self.batch_loop.run(kwargs)                                                                                                                                                                                       
  File "/opt/conda/lib/python3.8/site-packages/pytorch_lightning/loops/loop.py", line 200, in run                                                                                                                                    
    self.advance(*args, **kwargs)                                                                                                                                                                                                    
  File "/opt/conda/lib/python3.8/site-packages/pytorch_lightning/loops/batch/training_batch_loop.py", line 87, in advance                                                                                                            
    outputs = self.optimizer_loop.run(optimizers, kwargs)                                                                                                                                                                            
  File "/opt/conda/lib/python3.8/site-packages/pytorch_lightning/loops/loop.py", line 200, in run                                                                                                                                    
    self.advance(*args, **kwargs)                                                                                                                                                                                                    
  File "/opt/conda/lib/python3.8/site-packages/pytorch_lightning/loops/optimization/optimizer_loop.py", line 201, in advance                                                                                                         
    result = self._run_optimization(kwargs, self._optimizers[self.optim_progress.optimizer_position])                                                                                                                                
  File "/opt/conda/lib/python3.8/site-packages/pytorch_lightning/loops/optimization/optimizer_loop.py", line 248, in _run_optimization                                                                                               
    self._optimizer_step(optimizer, opt_idx, kwargs.get("batch_idx", 0), closure)                                                                                                                                                    
  File "/opt/conda/lib/python3.8/site-packages/pytorch_lightning/loops/optimization/optimizer_loop.py", line 358, in _optimizer_step                                                                                                 
    self.trainer._call_lightning_module_hook(                                                                                                                                                                                        
  File "/opt/conda/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1550, in _call_lightning_module_hook                                                                                                      
    output = fn(*args, **kwargs)                         
  File "/opt/conda/lib/python3.8/site-packages/pytorch_lightning/core/module.py", line 1705, in optimizer_step    
    optimizer.step(closure=optimizer_closure)            
  File "/opt/conda/lib/python3.8/site-packages/pytorch_lightning/core/optimizer.py", line 168, in step            
    step_output = self._strategy.optimizer_step(self._optimizer, self._optimizer_idx, closure, **kwargs)          
  File "/opt/conda/lib/python3.8/site-packages/pytorch_lightning/strategies/ddp.py", line 289, in optimizer_step  
    optimizer_output = super().optimizer_step(optimizer, opt_idx, closure, model, **kwargs)                       
  File "/opt/conda/lib/python3.8/site-packages/pytorch_lightning/strategies/strategy.py", line 216, in optimizer_step                                                                                                                
    return self.precision_plugin.optimizer_step(model, optimizer, opt_idx, closure, **kwargs)                     
  File "/opt/conda/lib/python3.8/site-packages/pytorch_lightning/plugins/precision/precision_plugin.py", line 153, in optimizer_step                                                                                                 
    return optimizer.step(closure=closure, **kwargs)     
  File "/opt/conda/lib/python3.8/site-packages/torch/optim/lr_scheduler.py", line 68, in wrapper                  
    return wrapped(*args, **kwargs)                      
  File "/opt/conda/lib/python3.8/site-packages/torch/optim/optimizer.py", line 114, in wrapper                    
    return func(*args, **kwargs)                         
  File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context         
    return func(*args, **kwargs)                         
  File "/opt/conda/lib/python3.8/site-packages/torch/optim/adamw.py", line 120, in step                           
    loss = closure()                                     
  File "/opt/conda/lib/python3.8/site-packages/pytorch_lightning/plugins/precision/precision_plugin.py", line 138, in _wrap_closure                                                                                                  
    closure_result = closure()                           
  File "/opt/conda/lib/python3.8/site-packages/pytorch_lightning/loops/optimization/optimizer_loop.py", line 146, in __call__                                                                                                        
    self._result = self.closure(*args, **kwargs)         
  File "/opt/conda/lib/python3.8/site-packages/pytorch_lightning/loops/optimization/optimizer_loop.py", line 132, in closure                                                                                                         
    step_output = self._step_fn()                        
  File "/opt/conda/lib/python3.8/site-packages/pytorch_lightning/loops/optimization/optimizer_loop.py", line 407, in _training_step                                                                                                  
    training_step_output = self.trainer._call_strategy_hook("training_step", *kwargs.values())                    
  File "/opt/conda/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1704, in _call_strategy_hook                                                                                                              
    output = fn(*args, **kwargs)                         
  File "/opt/conda/lib/python3.8/site-packages/pytorch_lightning/strategies/ddp.py", line 352, in training_step   
    return self.model(*args, **kwargs)                   
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1186, in _call_impl              
    return forward_call(*input, **kwargs)                
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/parallel/distributed.py", line 1009, in forward           
    output = self._run_ddp_forward(*inputs, **kwargs)    
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/parallel/distributed.py", line 970, in _run_ddp_forward   
    return module_to_run(*inputs[0], **kwargs[0])        
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1186, in _call_impl              
    return forward_call(*input, **kwargs)                
  File "/opt/conda/lib/python3.8/site-packages/pytorch_lightning/overrides/base.py", line 79, in forward          
    output = self.module.training_step(*inputs, **kwargs)                                                         
  File "/opt/conda/lib/python3.8/site-packages/nemo/utils/model_utils.py", line 364, in wrap_training_step        
    output_dict = wrapped(*args, **kwargs)               
  File "/opt/conda/lib/python3.8/site-packages/nemo/collections/asr/models/ctc_models.py", line 570, in training_step                                                                                                                
    log_probs, encoded_len, predictions = self.forward(input_signal=signal, input_signal_length=signal_len)       
  File "/opt/conda/lib/python3.8/site-packages/nemo/core/classes/common.py", line 1084, in __call__               
    outputs = wrapped(*args, **kwargs)                   
  File "/opt/conda/lib/python3.8/site-packages/nemo/collections/asr/models/ctc_models.py", line 540, in forward   
    processed_signal, processed_signal_length = self.preprocessor(                                                
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1186, in _call_impl              
    return forward_call(*input, **kwargs)                
  File "/opt/conda/lib/python3.8/site-packages/nemo/core/classes/common.py", line 1084, in __call__               
    outputs = wrapped(*args, **kwargs)                   
  File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context         
    return func(*args, **kwargs)                         
  File "/opt/conda/lib/python3.8/site-packages/nemo/collections/asr/modules/audio_preprocessing.py", line 85, in forward                                                                                                             
    processed_signal, processed_length = self.get_features(input_signal, length)                                  
  File "/opt/conda/lib/python3.8/site-packages/nemo/collections/asr/modules/audio_preprocessing.py", line 268, in get_features                                                                                                       
    return self.featurizer(input_signal, length)         
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1186, in _call_impl              
    return forward_call(*input, **kwargs)                
  File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context         
    return func(*args, **kwargs)                         
  File "/opt/conda/lib/python3.8/site-packages/nemo/collections/asr/parts/preprocessing/features.py", line 388, in forward                                                                                                           
    x, _, _ = normalize_batch(x, seq_len, normalize_type=self.normalize)                                          
  File "/opt/conda/lib/python3.8/site-packages/nemo/collections/asr/parts/preprocessing/features.py", line 58, in normalize_batch                                                                                                    
    raise ValueError(                                    
ValueError: normalize_batch with `per_feature` normalize_type received a tensor of length 1. This will result in torch.std() returning nan. Make sure your audio length has enough samples for a single feature (ex. at least `hop_le
ngth` for Mel Spectrograms).                             

[itzsimpl]

This is the variable dump when the Value error is raised:

shape=torch.Size([80, 1]) 
x=tensor([[[-13.1080, -15.0499, -15.3910,  ..., -16.6355, -16.6355, -16.6355],
         [-12.3124, -13.6862, -14.4951,  ..., -16.6355, -16.6355, -16.6355],
         [-12.3950, -13.5944, -13.7008,  ..., -16.6355, -16.6355, -16.6355],
         ...,
         [-12.0710, -12.0796, -12.4106,  ..., -16.6072, -16.6141, -16.6139],
         [-12.3980, -12.0559, -12.5859,  ..., -16.6233, -16.6096, -16.6272],
         [-14.5061, -14.3491, -14.3530,  ..., -16.6124, -16.6259, -16.6303]],

        [[-16.2118, -16.1531, -16.3190,  ..., -16.6355, -16.6355, -16.6355],
         [-16.2727, -15.9437, -16.4040,  ..., -16.6355, -16.6355, -16.6355],
         [-15.3276, -15.8527, -15.5322,  ..., -16.6355, -16.6355, -16.6355],
         ...,
         [-14.9770, -14.4016, -12.9210,  ..., -16.5926, -16.5834, -16.6250],
         [-13.8627, -14.4163, -11.8835,  ..., -16.6124, -16.5880, -16.6143],
         [-15.3740, -16.0521, -13.7070,  ..., -16.6148, -16.6053, -16.6244]],

        [[-16.6355, -16.6355, -16.6355,  ..., -16.6355, -16.6355, -16.6354],
         [-16.6355, -16.6355, -16.6355,  ..., -16.6355, -16.6355, -16.6354],
         [-16.6354, -16.6355, -16.6355,  ..., -16.6355, -16.6355, -16.6354],
         ...,
         [-16.6197, -16.6075, -16.6112,  ..., -16.5907, -16.6057, -16.5808],
         [-16.5938, -16.6020, -16.6157,  ..., -16.5928, -16.6115, -16.5883],
         [-16.5640, -16.6054, -16.6023,  ..., -16.6241, -16.6153, -16.6104]],

        ...,

        [[-12.1947, -13.5438, -15.2448,  ..., -16.6355, -16.6355, -16.6354],
         [-11.1802, -11.8474, -11.5849,  ..., -16.6355, -16.6355, -16.6355],
         [ -7.9363,  -9.0998,  -8.7032,  ..., -16.6355, -16.6355, -16.6355],
         ...,
         [ -6.6750,  -7.3614,  -6.8789,  ..., -16.6213, -16.6184, -16.5610],
         [ -6.5993,  -6.4922,  -6.1163,  ..., -16.5803, -16.5990, -16.5838],
         [ -7.1107,  -7.7420,  -7.3900,  ..., -16.5573, -16.5952, -16.5877]],

        [[-12.6002, -14.4936, -12.5170,  ..., -16.6355, -16.6355, -16.6355],
         [-11.4092, -11.6644, -10.1256,  ..., -16.6355, -16.6355, -16.6355],
         [-11.6143,  -9.8235,  -9.4089,  ..., -16.6355, -16.6355, -16.6355],
         ...,
         [-10.9189,  -9.6475, -10.6510,  ..., -16.5976, -16.5856, -16.6092],
         [-10.5369, -10.3292, -11.0258,  ..., -16.6191, -16.6133, -16.6218],
         [-10.6685, -12.4573, -12.7060,  ..., -16.6211, -16.6159, -16.6208]],

        [[-15.5550, -16.6268, -16.6353,  ..., -16.6355, -16.6355, -16.6355],
         [-15.3038, -16.5955, -16.6253,  ..., -16.6355, -16.6355, -16.6355],
         [-14.9854, -16.5553, -16.5145,  ..., -16.6355, -16.6355, -16.6355],
         ...,
         [-13.7720, -13.5534, -13.4671,  ..., -16.6106, -16.6016, -16.6189],
         [-14.9589, -14.9202, -14.4585,  ..., -16.6005, -16.5900, -16.6244],
         [-16.3464, -16.0353, -15.8030,  ..., -16.6042, -16.5939, -16.6240]]],
       device='cuda:0') 
i=2 
seq_len=tensor([119,  68,   1, 194, 290,  92, 103, 295, 133, 331, 281, 253, 326,  98,
        110, 164], device='cuda:0')  

I'm using the official nemo:22.08 container, hence nemo:1.12.0.

[itzsimpl]

I've noticed another peculiarity; to make things repeatable I've switched to single GPU training, disabled shuffling, and set validation_ds to null. The batch size is set to 16, and number workers is default (8). The previous dump was at Epoch 0, iteration 77/111 (1776 total files). Keeping only the head portion of the dataset (1200 files) I assumed that there should be no error, as in the previous run 77*16=1232 had no issues (shuffling is set to false). However, in this case the error occurs sooner; what am I missing?

ValueError: 
shape=torch.Size([80, 1]) 
x=tensor([[[-12.1400, -12.0013, -11.7135,  ..., -16.6355, -16.6355, -16.6355],
         [ -9.4995,  -9.8277, -10.2974,  ..., -16.6355, -16.6355, -16.6355],
         [ -7.9657,  -9.2955, -10.4785,  ..., -16.6355, -16.6355, -16.6354],
         ...,
         [-10.6667, -10.9675, -10.9428,  ..., -16.6055, -16.5725, -16.5755],
         [-10.9671, -11.6349, -10.4930,  ..., -16.6086, -16.5987, -16.5843],
         [-12.6284, -13.6401, -12.6856,  ..., -16.6072, -16.6059, -16.6044]],

        [[-15.4709, -15.3824, -15.8234,  ..., -16.6355, -16.6355, -16.6355],
         [-14.4115, -14.6622, -14.7424,  ..., -16.6355, -16.6355, -16.6355],
         [-12.6471, -12.6694, -12.1719,  ..., -16.6355, -16.6355, -16.6355],
         ...,
         [-14.1391, -13.3510, -12.7177,  ..., -16.6020, -16.5841, -16.6185],
         [-14.9618, -14.0504, -12.9129,  ..., -16.6024, -16.5978, -16.6126],
         [-14.8218, -15.2100, -14.6666,  ..., -16.5990, -16.6142, -16.6152]],

        [[-14.9606, -13.7396, -14.0115,  ..., -16.6355, -16.6355, -16.6355],
         [-13.1843, -13.3392, -13.1362,  ..., -16.6355, -16.6355, -16.6355],
         [-13.4342, -12.8727, -11.3514,  ..., -16.6355, -16.6355, -16.6355],
         ...,
         [ -9.5034,  -9.9679,  -9.9411,  ..., -16.6120, -16.6156, -16.6112],
         [ -9.5442,  -9.5324,  -9.6637,  ..., -16.6101, -16.6145, -16.6192],
         [-11.7415, -11.5879, -12.4916,  ..., -16.6108, -16.6185, -16.6131]],

        ...,

        [[-15.4600, -16.2199, -13.0660,  ..., -14.8596, -14.9728, -13.7306],
         [-12.8568, -13.3788, -11.4212,  ..., -13.7381, -12.6403, -11.9894],
         [-10.7183, -11.0641, -11.3699,  ..., -13.0889, -11.8980, -11.5180],
         ...,
         [-11.0726, -11.3289, -11.8909,  ..., -12.0608, -12.3335, -11.8812],
         [-11.4825, -12.3450, -12.3970,  ..., -13.1236, -13.0220, -12.2060],
         [-13.3480, -14.1608, -14.5494,  ..., -14.8705, -14.7676, -14.7237]],

        [[-11.8446, -14.9282, -14.8104,  ..., -16.6355, -16.6355, -16.6355],
         [-11.2285, -13.6097, -14.0806,  ..., -16.6355, -16.6355, -16.6355],
         [-10.4712, -12.5206, -12.6378,  ..., -16.6355, -16.6355, -16.6355],
         ...,
         [-11.3380, -12.2182, -11.8772,  ..., -16.6168, -16.6107, -16.6193],
         [-12.0109, -13.3219, -12.9040,  ..., -16.6038, -16.6008, -16.6123],
         [-14.3098, -15.3115, -15.2068,  ..., -16.5903, -16.5912, -16.5885]],

        [[-14.7913, -16.5903, -16.6228,  ..., -16.6355, -16.6355, -16.6355],
         [-14.7861, -16.5675, -16.5435,  ..., -16.6355, -16.6355, -16.6355],
         [-14.4304, -16.4519, -16.2934,  ..., -16.6355, -16.6355, -16.6355],
         ...,
         [-13.1459, -14.2566, -13.7941,  ..., -16.6103, -16.6051, -16.6076],
         [-12.9366, -14.6524, -13.3519,  ..., -16.6126, -16.6125, -16.6015],
         [-13.6179, -15.5170, -15.3774,  ..., -16.6195, -16.6131, -16.5883]]],
       device='cuda:0') 
i=10 
seq_len=tensor([246, 148, 135, 176, 286, 244, 317, 331, 225, 239,   1, 190, 457, 609, 
        156, 267], device='cuda:0') 
normalize_type=per_feature 
normalize_batch with `per_feature` normalize_type received a tensor of length 1. This will result in torch.std() returning nan. Make sure your audio length has enough
 samples for a single feature (ex. at least `hop_length` for Mel Spectrograms).
Epoch 0:  36%|███▌      | 27/75 [00:14<00:25,  1.89it/s, loss=147, v_num=2-46]

[titu1994]

The actual value isn't needed, just shape is enough. Seems your audio file is actually just 1 timestep. I dunno how that's possible, but normalization won't work with such audio.

[itzsimpl]

Yeah, you are right. There's a bunch of zero length files. Thank's @titu1994 for helping me find the problem.