Constrain Whisper's recognition search tree in real time?

[LexiconCode]

Is it possible to constrain Whisper's recognition search tree in real time? In other words to have the software strictly consider only those hypotheses defined in a grammar thus allowing for command recognition.

[ggerganov]

See initial prototype via the "Guided mode" of the command example

I want to extend this idea further, but haven't came around to do so yet.

[LexiconCode]

Thank you for posting the demo!

There's some interest in evaluating if it could be integrated into Dragonfly as a speech recognition backend. Dragonfly is a framework for building grammars and controlling applications on Mac/Linux/Windows. In order for whisper to be viable it needs to be fast and accurate.

People kind of hack their way into using whisper as a dictation element alongside another engine. However, it would be much more efficient if Whisper could process commands. Whisper has proven to be pretty accurate but latency until utterance recognition can be variable.

I noticed in demo it takes about 1,000 to 1200ish milliseconds per utterance. Are there ideas for optimizations to be made to reduce the latency? If I recall correctly, the model requires at least 30 seconds of audio. Hence, an utterance needs to be padded with silence to effectively be processed by the model.

In order for Dragonfly to work it's going to need to be able to dynamically change while the model is loaded which words are available from the grammars for recognition. Multiple grammars might be defined by the end-user. An end-user change at any moment what application is in the foreground with dragonfly's associated grammar.

[drmfinlay]

I'd like to weigh in here as the maintainer of Dragonfly, the library @LexiconCode mentions above. Dragonfly uses a sophisticated grammar format to define voice commands. In my estimation, Whisper is not suitable for use in that library because of this and for a number of other reasons.