Diarization inference for an external file, without accompanied rttm_file

[DavidHarar]

tl;dr

Hello, I am trying to follow this tutorial, with the only change being that I want to diarize new audio, so I don't have an rttm_file. The process seems to break in the 3rd stage (vad), with an error rise due to mismatch dimensions:

[NOTE]: The same configurations work using the original .wav file.

[NeMo W 2023-04-11 10:58:57 clustering_diarizer:411] Deleting previous clustering diarizer outputs.
[NeMo I 2023-04-11 10:58:57 speaker_utils:93] Number of files to diarize: 1
[NeMo I 2023-04-11 10:58:57 clustering_diarizer:309] Split long audio file to avoid CUDA memory issue
splitting manifest: 100%|██████████| 1/1 [00:00<00:00,  2.54it/s][NeMo I 2023-04-11 10:58:58 vad_utils:101] The prepared manifest file exists. Overwriting!
[NeMo I 2023-04-11 10:58:58 classification_models:267] Perform streaming frame-level VAD
[NeMo I 2023-04-11 10:58:58 collections:298] Filtered duration for loading collection is 0.000000.
[NeMo I 2023-04-11 10:58:58 collections:301] Dataset loaded with 3 items, total duration of  0.04 hours.
[NeMo I 2023-04-11 10:58:58 collections:303] # 3 files loaded accounting to # 1 labels

vad:   0%|          | 0/3 [00:00<?, ?it/s]
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
[<ipython-input-92-c796bc49439e>](https://localhost:8080/#) in <cell line: 1>()
----> 1 oracle_vad_clusdiar_model.diarize()

7 frames
[/usr/local/lib/python3.9/dist-packages/torch/_utils.py](https://localhost:8080/#) in reraise(self)
    642             # instantiate since we don't know how to
    643             raise RuntimeError(msg) from None
--> 644         raise exception
    645 
    646 

RuntimeError: Caught RuntimeError in DataLoader worker process 0.
Original Traceback (most recent call last):
  File "/usr/local/lib/python3.9/dist-packages/torch/utils/data/_utils/worker.py", line 308, in _worker_loop
    data = fetcher.fetch(index)
  File "/usr/local/lib/python3.9/dist-packages/torch/utils/data/_utils/fetch.py", line 54, in fetch
    return self.collate_fn(data)
  File "/usr/local/lib/python3.9/dist-packages/nemo/collections/asr/data/audio_to_label.py", line 443, in vad_frame_seq_collate_fn
    return _vad_frame_seq_collate_fn(self, batch)
  File "/usr/local/lib/python3.9/dist-packages/nemo/collections/asr/data/audio_to_label.py", line 184, in _vad_frame_seq_collate_fn
    sig = torch.cat((start, sig, end))
RuntimeError: Tensors must have same number of dimensions: got 1 and 2

Description

I tried to modify the original code as little as possible, I used the following configurations

import json
meta = {
    # 'audio_filepath': an4_audio,
    'audio_filepath': path_to_my_file,
    'offset': 0, 
    'duration':None, 
    'label': 'infer', 
    'text': '-', 
    'num_speakers': None, 
    'rttm_filepath': None, 
    'uem_filepath' : None
}
with open('data/input_manifest.json','w') as fp:
    json.dump(meta,fp)
    fp.write('\n')

!cat data/input_manifest.json

output_dir = os.path.join(ROOT, 'oracle_vad')
os.makedirs(output_dir,exist_ok=True)

from omegaconf import OmegaConf
MODEL_CONFIG = os.path.join(data_dir,'diar_infer_telephonic.yaml')
if not os.path.exists(MODEL_CONFIG):
    config_url = "https://raw.githubusercontent.com/NVIDIA/NeMo/main/examples/speaker_tasks/diarization/conf/inference/diar_infer_telephonic.yaml"
    MODEL_CONFIG = wget.download(config_url,data_dir)

config = OmegaConf.load(MODEL_CONFIG)
print(OmegaConf.to_yaml(config))

config.diarizer.manifest_filepath = 'data/input_manifest.json'
config.diarizer.out_dir = output_dir # Directory to store intermediate files and prediction outputs
pretrained_speaker_model = 'titanet_large'
config.diarizer.speaker_embeddings.model_path = pretrained_speaker_model
config.diarizer.speaker_embeddings.parameters.window_length_in_sec = [1.5,1.25,1.0,0.75,0.5] 
config.diarizer.speaker_embeddings.parameters.shift_length_in_sec = [0.75,0.625,0.5,0.375,0.1] 
config.diarizer.speaker_embeddings.parameters.multiscale_weights= [1,1,1,1,1] 
config.diarizer.oracle_vad = False # ----> ORACLE VAD [NOTE: I changed it into False, since I don't have ground truth. Keeping it as True leads to an error in the first stage]
config.diarizer.clustering.parameters.oracle_num_speakers = False

# Here things get broken
# -------------
from nemo.collections.asr.models import ClusteringDiarizer
oracle_vad_clusdiar_model = ClusteringDiarizer(cfg=config)
oracle_vad_clusdiar_model.diarize()


---- OUTPUT -----> 
[NeMo W 2023-04-11 10:58:57 clustering_diarizer:411] Deleting previous clustering diarizer outputs.
[NeMo I 2023-04-11 10:58:57 speaker_utils:93] Number of files to diarize: 1
[NeMo I 2023-04-11 10:58:57 clustering_diarizer:309] Split long audio file to avoid CUDA memory issue
splitting manifest: 100%|██████████| 1/1 [00:00<00:00,  2.54it/s][NeMo I 2023-04-11 10:58:58 vad_utils:101] The prepared manifest file exists. Overwriting!
[NeMo I 2023-04-11 10:58:58 classification_models:267] Perform streaming frame-level VAD
[NeMo I 2023-04-11 10:58:58 collections:298] Filtered duration for loading collection is 0.000000.
[NeMo I 2023-04-11 10:58:58 collections:301] Dataset loaded with 3 items, total duration of  0.04 hours.
[NeMo I 2023-04-11 10:58:58 collections:303] # 3 files loaded accounting to # 1 labels

vad:   0%|          | 0/3 [00:00<?, ?it/s]
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
[<ipython-input-92-c796bc49439e>](https://localhost:8080/#) in <cell line: 1>()
----> 1 oracle_vad_clusdiar_model.diarize()

7 frames
[/usr/local/lib/python3.9/dist-packages/torch/_utils.py](https://localhost:8080/#) in reraise(self)
    642             # instantiate since we don't know how to
    643             raise RuntimeError(msg) from None
--> 644         raise exception
    645 
    646 

RuntimeError: Caught RuntimeError in DataLoader worker process 0.
Original Traceback (most recent call last):
  File "/usr/local/lib/python3.9/dist-packages/torch/utils/data/_utils/worker.py", line 308, in _worker_loop
    data = fetcher.fetch(index)
  File "/usr/local/lib/python3.9/dist-packages/torch/utils/data/_utils/fetch.py", line 54, in fetch
    return self.collate_fn(data)
  File "/usr/local/lib/python3.9/dist-packages/nemo/collections/asr/data/audio_to_label.py", line 443, in vad_frame_seq_collate_fn
    return _vad_frame_seq_collate_fn(self, batch)
  File "/usr/local/lib/python3.9/dist-packages/nemo/collections/asr/data/audio_to_label.py", line 184, in _vad_frame_seq_collate_fn
    sig = torch.cat((start, sig, end))
RuntimeError: Tensors must have same number of dimensions: got 1 and 2

The same configurations, using the original .wav file, work.

[nehat005]

Did you check if your audio is single-channel or multi-channel ?
The possible explanation for such error would be that the padding (start and end arrays) is a one-dim tensor while audio would be two-dim array in which case the torch concatenation function fails because it needs the second tensor to match the number of dimensions of the first tensor.

Similar Issue