[HELP NEEDED]: Evaluating different methods of getting Word and Utterance (sentence) level Timestamps from NeMo models

[ishansharma1320]

Hi,

Thanks for reading this message.

I have been recently experimenting with NeMo to extract word-level and utterance-level timestamps for text transcriptions.

Based on information available as tutorials and discussion/PR threads, I was able to gather the following methods:

Word Level Timestamps

1. Getting Word Timestamps by following #4342 and this comment at issue 4629

In this, we get word timestamps by multiplying stride and preprocessor.window_stride with both start and end offset available in the word attribute from the hypothesis object generated after inferencing on audio.
This is available for both CTC as well as RNN-T based architectures.

2. Getting Word Timestamps by following ASR_withSpeakerDiarization.ipynb

This is only available for the following models

stt_en_quartznet15x5,
stt_en_citrinet*,
stt_en_conformer_ctc-large

3. Using Montreal Forced Aligner as mentioned in Discussion [2657]
   (Forced Alignment #2657 (reply in thread))

Utterance (sentence) Level Timestamps

1. There is also a way of getting utterance-level timestamps by converting the RTTM file obtained after Speaker Diarization Inference into an input manifest file for speech transcription to be done down the line.

Based on the above information,

I have the following doubts,

1. Out of the 3 methods mentioned above, which is more accurate for word-level timestamps that can be implemented for both CTC and RNN-T architectures and are there any other approaches to achieve the same that are more accurate?

2. For utterance level timestamps, is the approach correct and are there any other approaches to achieve the same that are more accurate?

Any advice/insights regarding this will be highly appreciated.

Thanks Again
Ishan

[tango4j]

1. For the dataset we have tested, stt_en_conformer_ctc_large showed the lowest error for ASR with diarization
2. With the default settings, it is stt_en_conformer_ctc_large .