Livekit does not react to user speech when turn_detection set to "stt" in the Google plugin for certain models

[mrkowalski]

livekit-agents==1.2.15 never reacts to user speech for the following Google plugin STT configuration:

google.STT(
  languages="pl-PL",
  detect_language=False,
  model="latest_long",
  location="eu",
  use_streaming=True,
  punctuate=False,
  interim_results=False,
)

and AgentSession(turn_detection="stt" ...

It works ok on 1.2.14 or when turn_detection is unset.

I see the difference in debugger caused by this PR: #3559 :

Lines 351-353 are not executed in 1.2.15:

if not self._speaking:
  chat_ctx = self._hooks.retrieve_chat_ctx().copy()
  self._run_eou_detection(chat_ctx)

In current code, there exists a code path for the FINAL_TRANSCRIPT event handler for which nothing happens: When _vad_base_turn_detection is False and _user_turn_committed is False.

[longcw]

when using turn_detection = "stt", we use stt.SpeechEventType.END_OF_SPEECH event to commit the user turn. can you check if this event is fired correctly when using google STT?

It works ok on 1.2.14

this should be a bug introduced specifically in 1.2.14 that commits user turn for any final user transcripts.

[mrkowalski]

@longcw, for the latest_long model, when turn_detection="stt", the fired event is SpeechEventType.FINAL_TRANSCRIPT. And, in the code above, nothing happens. It seems to work ok for chirp_2 and chirp_3.

[ChenghaoMou]

I just tested it with the main branch. After setting enable_voice_activity_events to True, it works fine:

stt=google.STT(
    model="latest_long",
    enable_voice_activity_events=True,
),
turn_detection="stt",

Without it, it only reported SPEECH_EVENT_TYPE_UNSPECIFIED events so no stt.SpeechEventType.END_OF_SPEECH was fired.

[mrkowalski]

I just tested it with the main branch. After setting enable_voice_activity_events to True, it works fine:

stt=google.STT(
model="latest_long",
enable_voice_activity_events=True,
),
turn_detection="stt",

Without it, it only reported SPEECH_EVENT_TYPE_UNSPECIFIED events so no stt.SpeechEventType.END_OF_SPEECH was fired.

I double checked. It is correct and works for Polish, too.

If there is a relevant bit of documentation to update - I am happy to do it. If not - the issue can be closed.

[ChenghaoMou]

@mrkowalski Glad it worked for you!

If you want, adding a check somewhere in AgentActivity.__init__ to enable that flag when using Google STT and self._turn_detection_mode == "stt" might work better. Not sure how docs are generated/updated though. cc @longcw who has a better context.

I can see Deepgram has the vad_events option hardcoded to True, so there might be considerations behind the False value that I am not aware of. Otherwise, setting it to True by default might be another alternative.