Is it possible to use offline_diar_with_asr_infer with stt_ru_conformer_ctc_large?

[zzomg]

Hi 👋 I've been struggling with making this script work with stt_ru_conformer_ctc_large model. I see in docs that it says:

Currently, NeMo offline diarization inference supports QuartzNet English model and ConformerCTC ASR models (e.g.,QuartzNet15x5Base-En, stt_en_conformer_ctc_large).

So, I assume it should work with models trained for Russian language as well. However, as I try to use the script:

python offline_diar_with_asr_infer.py \
    diarizer.manifest_filepath=an4_test_diar/golos_1file_test.json \
    diarizer.out_dir='demo_asr_output' \
    diarizer.speaker_embeddings.model_path='ecapa_tdnn' \
    diarizer.asr.model_path='stt_ru_conformer_ctc_large' \
    diarizer.asr.parameters.asr_based_vad=True \
    diarizer.speaker_embeddings.parameters.save_embeddings=False

I get this exception:

Error executing job with overrides: ['diarizer.manifest_filepath=an4_test_diar/golos_1file_test.json', 'diarizer.out_dir=demo_asr_output', 'diarizer.speaker_embeddings.model_path=ecapa_tdnn', 'diarizer.asr.model_path=stt_ru_conformer_ctc_large', 'diarizer.asr.parameters.asr_based_vad=True', 'diarizer.speaker_embeddings.parameters.save_embeddings=False']
Traceback (most recent call last):
  File "offline_diar_with_asr_infer.py", line 69, in main
    diar_hyp, diar_score = asr_diar_offline.run_diarization(cfg, word_ts_hyp)
  File "/nas/Anaconda/anaconda3/envs/oMeN/lib/python3.8/site-packages/nemo/collections/asr/parts/utils/diarization_utils.py", line 495, in run_diarization
    score = diar_model.diarize()
  File "/nas/Anaconda/anaconda3/envs/oMeN/lib/python3.8/site-packages/nemo/collections/asr/models/clustering_diarizer.py", line 421, in diarize
    self._perform_speech_activity_detection()
  File "/nas/Anaconda/anaconda3/envs/oMeN/lib/python3.8/site-packages/nemo/collections/asr/models/clustering_diarizer.py", line 330, in _perform_speech_activity_detection
    validate_vad_manifest(self.AUDIO_RTTM_MAP, vad_manifest=self._speaker_manifest_path)
  File "/nas/Anaconda/anaconda3/envs/oMeN/lib/python3.8/site-packages/nemo/collections/asr/parts/utils/speaker_utils.py", line 622, in validate_vad_manifest
    raise ValueError("All files present in manifest contains silence, aborting next steps")
ValueError: All files present in manifest contains silence, aborting next steps

golos_1file_test.json:

{"audio_filepath": "/nas/Data/Audio/Processed/RU/Golos/test/crowd/files/c45bc24490f3e2395bdd4649bc16f165.wav", "offset": 0, "duration": 4.84, "label": "infer", "num_speakers": 1, "text": "закажите мне комплексный обед в ресторане в номер"}

I use the audio file from the dataset Golos https://github.com/sberdevices/golos , tried it with multiple files from the dataset and got the same result.
Also, as I dug into the problem, it turned out that during decoding stage in WERBPE_TS.ctc_decoder_predictions_tensor_with_ts not single character is recognized and hypotheses consists solely of unknown symbols '⁇'. So, timestamps are not produced whatsoever. Everything works fine with stt_en_conformer_ctc_large, though.

Config file that I use (default diar_infer_meeting.yaml):

# This YAML file is created for all types of offline speaker diarization inference tasks in `<NeMo git root>/example/speaker_tasks/diarization` folder.
# The inference parameters for VAD, speaker embedding extractor, clustering module, MSDD module, ASR decoder are all included in this YAML file. 
# All the keys under `diarizer` key (`vad`, `speaker_embeddings`, `clustering`, `msdd_model`, `asr`) can be selectively used for its own purpose and also can be ignored if the module is not used.
# The configurations in this YAML file is suitable for 3~5 speakers participating in a meeting and may not show the best performance on other types of dialogues.
# An example line in an input manifest file (`.json` format):
# {"audio_filepath": "/path/to/audio_file", "offset": 0, "duration": null, "label": "infer", "text": "-", "num_speakers": null, "rttm_filepath": "/path/to/rttm/file", "uem_filepath": "/path/to/uem/file"}
name: &name "ClusterDiarizer"

num_workers: 1
sample_rate: 16000
batch_size: 64
device: null # can specify a specific device, i.e: cuda:1 (default cuda if cuda available, else cpu)

diarizer:
  manifest_filepath: ???
  out_dir: ???
  oracle_vad: False # If True, uses RTTM files provided in the manifest file to get speech activity (VAD) timestamps
  collar: 0.25 # Collar value for scoring
  ignore_overlap: True # Consider or ignore overlap segments while scoring

  vad:
    model_path:  vad_multilingual_marblenet # .nemo local model path or pretrained VAD model name 
    external_vad_manifest: null # This option is provided to use external vad and provide its speech activity labels for speaker embeddings extraction. Only one of model_path or external_vad_manifest should be set

    parameters: # Tuned parameters for CH109 (using the 11 multi-speaker sessions as dev set) 
      window_length_in_sec: 0.63  # Window length in sec for VAD context input 
      shift_length_in_sec: 0.01 # Shift length in sec for generate frame level VAD prediction
      smoothing: False # False or type of smoothing method (eg: median)
      overlap: 0.5 # Overlap ratio for overlapped mean/median smoothing filter
      onset: 0.9 # Onset threshold for detecting the beginning and end of a speech 
      offset: 0.5 # Offset threshold for detecting the end of a speech
      pad_onset: 0 # Adding durations before each speech segment 
      pad_offset: 0 # Adding durations after each speech segment 
      min_duration_on: 0 # Threshold for small non_speech deletion
      min_duration_off: 0.6 # Threshold for short speech segment deletion
      filter_speech_first: True 

  speaker_embeddings:
    model_path: titanet_large # .nemo local model path or pretrained model name (titanet_large, ecapa_tdnn or speakerverification_speakernet)
    parameters:
      window_length_in_sec: [3.0,2.5,2.0,1.5,1.0,0.5] # Window length(s) in sec (floating-point number). either a number or a list. ex) 1.5 or [1.5,1.0,0.5]
      shift_length_in_sec: [1.5,1.25,1.0,0.75,0.5,0.25] # Shift length(s) in sec (floating-point number). either a number or a list. ex) 0.75 or [0.75,0.5,0.25]
      multiscale_weights: [1,1,1,1,1,1] # Weight for each scale. should be null (for single scale) or a list matched with window/shift scale count. ex) [0.33,0.33,0.33]
      save_embeddings: True # If True, save speaker embeddings in pickle format. This should be True if clustering result is used for other models, such as `msdd_model`.
  
  clustering:
    parameters:
      oracle_num_speakers: False # If True, use num of speakers value provided in manifest file.
      max_num_speakers: 8 # Max number of speakers for each recording. If an oracle number of speakers is passed, this value is ignored.
      enhanced_count_thres: 80 # If the number of segments is lower than this number, enhanced speaker counting is activated.
      max_rp_threshold: 0.25 # Determines the range of p-value search: 0 < p <= max_rp_threshold. 
      sparse_search_volume: 30 # The higher the number, the more values will be examined with more time. 
      maj_vote_spk_count: False  # If True, take a majority vote on multiple p-values to estimate the number of speakers.
  
  msdd_model:
    model_path: null # .nemo local model path or pretrained model name for multiscale diarization decoder (MSDD)
    parameters:
      use_speaker_model_from_ckpt: True # If True, use speaker embedding model in checkpoint. If False, the provided speaker embedding model in config will be used.
      infer_batch_size: 25 # Batch size for MSDD inference. 
      sigmoid_threshold: [0.7] # Sigmoid threshold for generating binarized speaker labels. The smaller the more generous on detecting overlaps.
      seq_eval_mode: False # If True, use oracle number of speaker and evaluate F1 score for the given speaker sequences. Default is False.
      split_infer: True # If True, break the input audio clip to short sequences and calculate cluster average embeddings for inference.
      diar_window_length: 50 # The length of split short sequence when split_infer is True.
      overlap_infer_spk_limit: 5 # If the estimated number of speakers are larger than this number, overlap speech is not estimated.
  
  asr:
    model_path: stt_en_conformer_ctc_large # Provide NGC cloud ASR model name. stt_en_conformer_ctc_* models are recommended for diarization purposes.
    parameters:
      asr_based_vad: False # if True, speech segmentation for diarization is based on word-timestamps from ASR inference.
      asr_based_vad_threshold: 1.0 # Threshold (in sec) that caps the gap between two words when generating VAD timestamps using ASR based VAD.
      asr_batch_size: null # Batch size can be dependent on each ASR model. Default batch sizes are applied if set to null.
      decoder_delay_in_sec: null # Native decoder delay. null is recommended to use the default values for each ASR model.
      word_ts_anchor_offset: null # Offset to set a reference point from the start of the word. Recommended range of values is [-0.05  0.2]. 
      word_ts_anchor_pos: "start" # Select which part of the word timestamp we want to use. The options are: 'start', 'end', 'mid'.
      fix_word_ts_with_VAD: False # Fix the word timestamp using VAD output. You must provide a VAD model to use this feature.
      colored_text: False # If True, use colored text to distinguish speakers in the output transcript.
      print_time: True # If True, the start and end time of each speaker turn is printed in the output transcript.
      break_lines: False # If True, the output transcript breaks the line to fix the line width (default is 90 chars)
    
    ctc_decoder_parameters: # Optional beam search decoder (pyctcdecode)
      pretrained_language_model: null # KenLM model file: .arpa model file or .bin binary file.
      beam_width: 32
      alpha: 0.5
      beta: 2.5

    realigning_lm_parameters: # Experimental feature
      arpa_language_model: null # Provide a KenLM language model in .arpa format.
      min_number_of_words: 3 # Min number of words for the left context.
      max_number_of_words: 10 # Max number of words for the right context.
      logprob_diff_threshold: 1.2  # The threshold for the difference between two log probability values from two hypotheses.

So, is it possible to use not only English models with offline diarization with ASR, and if yes, what could be the issue?

I am using nemo ver. 1.15.0.

Thanks!

[nithinraok]

Have you just run only ASR inference with stt_ru_conformer_ctc_large using this script? Did it transcribe flawlessly?

[zzomg]

Hi! Sorry for the late reply, no, it doesn't work as well, that is the output:

{"audio_filepath": "/nas/Data/Audio/Processed/RU/Golos/test/crowd/files/c45bc24490f3e2395bdd4649bc16f165.wav", "offset": 0, "duration": 4.84, "label": "infer", "num_speakers": 1, "text": "закажите мне комплексный обед в ресторане в номер", "pred_text": " ⁇ "}

for

python transcribe_speech.py \
    pretrained_name=stt_ru_conformer_ctc_large \
    dataset_manifest=an4_test_diar/golos_1file_test.json \
    output_filename=test.out \
    batch_size=32 \
    compute_timestamps=False \
    compute_langs=False \
    cuda=0 \
    amp=True \
    append_pred=False 

I used the same dataset_manifest as in the original post.
It still works fine with stt_en_conformer_ctc_large, though.

[nithinraok]

Then the issue is with the Ru ASR model. @titu1994 any pointers to getting the best ASR predictions?
btw is it possible to share audio here?

[zzomg]

c45bc24490f3e2395bdd4649bc16f165.zip
Yes, sure

[VahidooX]

Would you please try without amp?

[zzomg]

Yes, I removed with torch.cuda.amp.autocast() context manager from ASRDecoderTimeStamps.run_ASR_BPE_CTC and that actually helped, thank you! What could be the reason why AMP affects inference in this case?