v2 (ctc_align) versus v3 (whisper timestamps)

[joazoa]

Thank you for the great project!

In v2 you were using CTC align, in v3 you are using whisper timestamps.
As far as i know, the timestamps from the CTC_align are more accurate than the whisper timestamps.

What was the reason to use the Whisper timestamps instead of CTC?

We have more accurate transcripts than Whisper generates, how could i use those directly ?

I tried manually making the v3 speaker representations but based on CTC timestamps and then use those, but the results are not good, would it require retraining / finetuning to use the CTC timestamps ?

[edwko]

Hi,

The main reason is that CTC alignment only supports Latin characters. To make it work across different languages, a lot of preprocessing is required, such as romanizing non-Latin scripts, converting numbers, and other manual adjustments. I’ve used CTC to build datasets, but it often required case-by-case fixes. Whisper, on the other hand, works universally across many languages without needing all that extra handling.

At runtime, the model doesn’t rely heavily on precise timestamps for speaker reference.

[joazoa]

Thank you for the quick reply!
For CTC alignment, you could use uroman first but the numbers would indeed remain an issue.
I noticed that with the CTC we have lowercase results and punctuation removed, i tried restoring them after aligning, but words are sometimes split in 2 with whisper and ctc align combines them (words with ' inside) The ctc speaker timestamps seem to work fine on some samples but messes up most of the samples :/
My use case is for languages where Whisper does not do well, so I'd need to retrain anyway.

update: just saw you used uroman in V1 and anyascii in v2, so does the same thing.
update2: i got it to work with the CTC timestamps :)

thank you!!