Testing OpenVINO Whisper Transcription with Audacity 3.4.2

[petersampsonaudacity]

Testing the Whisper Transcription analyzer -

I don't have any covert recordings with whispering so I tried Chris de Burgh's Lady in Red where he whispers "I love you" at the end.

Not only did it transcribe the whispered bit - it transcribed the whole lyrics of the song - mighty impressive.
Perhaps it needs a threshold setting to exclude non-whispered words?

[petersampsonaudacity]

It didn't fare so well with a track by a folk singer friend of mine who sings in Lancashire dialect - but I suspected it would struggle with that. 😉

But having said that it takes a fair stab at it, not easy to do with Michelle's dialect singing.

The part outlined in Orange says (translated into normal English):

If Pendle Hill were made of gold

Michelle's dialect sing made of gold as "med o gowd" or similar - no worries as her accompanist who is a UK southerner can't make that dialect pronunciation (nor me either) she winds us up about it.

[petersampsonaudacity]

@RyanMetcalfeInt8 replied (on Discord)

And just to clarify, the AI model that we use is called 'whisper' (https://openai.com/research/whisper) -- but it's not really meant for only whispered speech.. so yeah, kind of confusing. It's intended for any kind of speech transcription, even if it's yelled 😉

[RyanMetcalfeInt8]

@petersampsonaudacity -- Thank you for the feedback!

Tagged this also with 'enhancement', as there are bigger whisper models (small, large, etc.) that could technically add support for, which might give more accurate transcriptions / translations (at the cost of longer processing times).

[joedale100]

As OpenVINO Whisper Transcription is installed locally, does that mean it doesn't require sharing the audio files with third-party services, raising potential privacy and security concerns, particularly when handling sensitive or confidential material? I'm wondering if schools could use the plug-in safely and be compliant with the data handling policy. Could you clarify?

[RyanMetcalfeInt8]

Hi @joedale100 -- Yes, all of the AI models used by this plugin are installed and run locally. There are no third-party (cloud) services needed or used.. you could disconnect from the network and the plugins would work just fine.

[petersampsonaudacity]

@RyanMetcalfeInt8

Hi Ryan,

I tested the transcription of track1 of Sgt Pepper, wich is a vocal track, but it found no words at all to transcribe:

This of course tallies with the result I found earlier when testing noise suppression on #5

[RyanMetcalfeInt8]

Hi @petersampsonaudacity,

I tested the transcription of track1 of Sgt Pepper, wich is a vocal track, but it found no words at all to transcribe:

I can try this track on my end but these results may be expected -- I've had mixed results trying to transcribe vocals from a track that includes background instrumentals. You might have better luck by first applying music separation, and then applying transcription to the resulting vocal track.

Thanks!
Ryan