"response.audio_transcript.done" server event arrives too late

[fedorenchik]

Our use case is robot with voice replies and relevant actions. Voice replies are sent via audio channel and relevant robot actions are sent via data channel with "response.create" and "response.done". And "response.create" sends data from "response.audio_transcript.done" server event.

Actual results:

1. "response.audio_transcript.done" server event arrives 5-35 seconds after relevant audio data finished transmitting (that is, after AI agent stops speaking reply).
2. "response.done" arrives 5-10 seconds after "response.create" is sent.

So, at the end, robots actions are executed after some delay after robot stops speaking, which looks unnatural and gives an impression that robot is stuck.

Expected results:

1. "response.audio_transcript.done" server event arrives at the same time or before audio data transmission is finished (robot stops talking)
2. "response.done" server event arrives somewhat faster? 1-2 seconds if possible.

[thebhulawat]

I am working on esp32 and noticed that sporadically these events are not even sent. Have you noticed something like that?

[thebhulawat]

I also observed that sometimes the delay is not in secs but minutes for server events.

[thebhulawat]

Weird part is when I run real time api on a browser client, the latency for the events go away

[fedorenchik]

I am working on esp32 and noticed that sporadically these events are not even sent. Have you noticed something like that?

I've noticed some events are returned, but have type set to "null" or sometimes also error is returned.

[thebhulawat]

this has fixed it to some extent: Sean-Der/libpeer#1
but yet to fully test it.