[Question] How to introduce VAD to solve the problem of hallucinations

[GuuuWei]

Background:
I've noticed that when processing audio files containing silent or non-speech segments, Whisper tends to generate hallucinatory content. This not only affects the segments with silence or non-human voices but also seems to impact the subsequent normal speech parts in the audio.

Inquiry:
Given that this is an inherent issue with Whisper, I am curious to know if it's feasible to incorporate strategies similar to VAD in Whisper-turbo. I am aware of approaches like those used in projects such as WhisperX, which seem to effectively mitigate such issues.

Thank you for your time and the incredible work on this project.

[FL33TW00D]

Background: I've noticed that when processing audio files containing silent or non-speech segments, Whisper tends to generate hallucinatory content. This not only affects the segments with silence or non-human voices but also seems to impact the subsequent normal speech parts in the audio.

Inquiry: Given that this is an inherent issue with Whisper, I am curious to know if it's feasible to incorporate strategies similar to VAD in Whisper-turbo. I am aware of approaches like those used in projects such as WhisperX, which seem to effectively mitigate such issues.

Thank you for your time and the incredible work on this project.

I think the approach from WhisperX is very good: https://www.robots.ox.ac.uk/~vgg/publications/2023/Bain23/bain23.pdf

Unfortunately - it is lots of work to implement.

If whisper-turbo gets more traction and enterprise customers, it may be feasible for me to implement it.

[obenjiro]

@from-gu-wei
FIY: You can use https://github.com/ricky0123/vad look like this library is compatible with whisper-turbo