Streaming Support for Kokoro

[putt6359]

Hi,

I'm not sure if I understand what I'm posting. I wonder if streaming can be implemented for kokoro-fastapi to reduce latency?

[roryeckel]

Hi, after the full transcription has been received, the audio is streamed. Implemented here:

wyoming_openai/src/wyoming_openai/handler.py

Line 866 in a42abac

async def _stream_tts_audio(self, voice: TtsVoiceModel, text: str, send_audio_start: bool = True, start_timestamp: float = 0) -> float | None:

and this support is native from the OpenAI library (with_streaming_response).
You may have seen the new release: https://github.com/roryeckel/wyoming_openai/releases/tag/v0.3.7 which introduced pysbd text chunking to split at sentence boundaries for parallelized processing speedups: https://github.com/roryeckel/wyoming_openai?tab=readme-ov-file#table-of-environment--command-line-options (see TTS_STREAMING_MODELS)
For even more responsiveness, you would also need to combine it with STT streaming transcription using STT_STREAMING_MODELS=gpt-4o-transcribe. To my knowledge, there are not any open source projects implementing streaming STT via OpenAI-compatible endpoints. I would love if someone is able to find any.

[putt6359]

Hi, yes. I just played with the TTS_STREAMING_MODELS yesterday, and it is working as intended. Thank you very much for this wonderful project.

You're right. I hope there will be STT projects there that would support streaming besides OpenAI.