whisper : add Silero VAD built-in support

[ggerganov]

It seems the Silero VAD project is a very good candidate for being integrated directly in whisper.cpp:

https://github.com/snakers4/silero-vad

The model appears to be very lightweight and can be used to extract voice activity timestamps that the user can then use to process with the core whisper.cpp functionality, avoiding non-speech segments.

The model data of Silero VAD is stored in ONNX data format. From a quick search, I cannot find where the actual model tensors and architecture are defined, so we'll have to do some reverse engineering to convert this to a GGUF file. The only information I was able to find is that the model uses a multi-head attention (MHA) + short-time Fourier transform, which shouldn't be a problem to implement within whisper.cpp.

Having an integrated VAD solution in the project would simplify the work of 3rd-party projects using whisper.cpp. It is also likely to improve the notoriously bad performance of the Whisper Large v3 models.

[JRWSP]

I have been using the Silero VAD with whisper.cpp by making a script to manage files manually.
Don't know if this is what you need.

https://github.com/JRWSP/SileroVAD_for_Whisper-cpp

[tbarbugli]

I used the VAD in Go using onnxruntime, you can see the interface here https://github.com/streamer45/silero-vad-go/blob/master/speech/ort_bridge.h

[ggerganov]

Thanks. The goal is to not depend on ONNX or python and to natively run the VAD using ggml.

[danbev]

I want to try it.

Just for your information and so we don't duplicate efforts, I've started working on this issue (if an issue is assigned it is most often being worked on). I hope to be able to share some progress soon.

Sorry about the late response, I have gathered some notes about the progress and I hope to have a pull request for this next week (short week here due to Easter).

[basemkhirat]

Thanks. waiting for it.