Add a Subtitles Feature

[mattmcalister]

Given the integrated transcription feature and LLMs in the browser we have the data and capabilities required to offer Subtitles, including translated versions of them. (h/t @thypon).

[thypon]

it looks like the subtitle API is already there.
Is there any reason we did not enabled this @mrose17 ?

I see I can enable them programmatically hooking the API

[mrose17]

the last time I researched this, the toggleSubtitles and setSubtitles calls dealt with sending the audio to Google servers for processing, not for displaying text on the screen. I will confirm whether this is still the case!

[saghul]

We currently support 3 transcribers which can be used for subtitles: Google, Oracle and 8x8 (ourselves).

You can select the one you like from the JaaS console.

At this time, our own Whisper based transcriber only does English, so translations are handled with Google. You can disable that behavior by configureing the translationLanguages option in the transcription options block.

[mrose17]

Thanks - just so i'm on the same page: all we really want is an API that let's us place a caption on the screen. in other words, we continue to use our existing Brave Talk Marvin arrangement, but in our hook, we make a call to also display the text. (Obviously, this would be a per-user option.)

I don't see such an API call, am I missing it?

Thank you!

[saghul]

That's not currently possible and given how complex the rendering logic is there I'm not sure we can provide one.

Let me think about this a bit.

[saghul]

@mrose17 Got it.

We could add such an API.

Note however that our subtitles display the transcript, the same you get via the iframe, in real time. Are you interested in rendering something different from that?

[mrose17]

Thanks. As long as the text comes from your servers (not Google's), I don't really care what the API does!

Option 1: give me an API that takes text and puts it on the screen; or,
Option 2: give me an option to turn on/off the transcript going on the screen.

I kind of like option 1 as it is more flexible, but whatever is easier...

[saghul]

We can do either, though option 1 is already there and implemented.

[mrose17]

Excellent, in that case, please point me to the API (I couldn't find it)... sorry, I know that's my constant refrain, but I never seem to b able to find the APIs you've already documented...

[saghul]

Haha, no worries at all. Here it is: https://jitsi.github.io/handbook/docs/dev-guide/dev-guide-iframe-commands#setsubtitles make sure to set the language to 'en' .

[mrose17]

Thank you!

[mrose17]

Oops, closed by mistake!