What's the best way to build an API on top of whisper.cpp?

[nikhilro]

Hey @ggerganov,

Thanks for Whisper.cpp. It's indispensable for us and I'm excited by everything in your roadmap. Particularly, diarization.

We are making an AI Meeting Notes product and using Whisper.cpp to transcribe meeting audio. We first tried to run locally on user's machines. It worked but was unreliable—most people don't have Apple ARM chips. So, while the toolkit to transcribe locally gets built out, we're thinking of deploying Whisper.cpp in our servers to transcribe the audio.

This has been working well but is expensive. I feel there are ways to implement our setup a lot more efficiently. Hoping you can help.

Here is our current setup: audio comes in -> it's put on the queue -> some worker picks it up / create new worker to accommodate load -> clone Whisper.cpp repo, make, invoke main with audio, return output. We use Node so everything is done using exec statements. Usually, audio coming is 1 minute chunks but we have flexibility on this from FE.

Two improvements we're thinking:

1. On worker initialization, setup the repo and do make. (Easy)
2. Init Whisper.cpp "main" process to run beside the Node Worker process. (I don't know how to do this or if it's even possible)
   • Then, whenever the Node Worker receives job, it "sends" it to already running Whisper.cpp and "receives" output back.
   • Ideally, one Node Worker can receive many jobs at once.

So, for you,

• How could we do 2 if possible and recommended?
• Do you have any better ideas to make our setup more efficient?

[ItsMos]

@nikhilro why dont you just use a prebuilt whisper.cpp instead of building it everytime you recieve an audio?

[nikhilro]

Yup, done. Didn't help that much.