Agent state management issues in `useAgent` hook

[frankfarzan]

Select which package(s) are affected

@livekit/components-react

Describe the bug

Hi @1egoman, I'm playing around with Agent SDK changes (from #1207) and observed some issues with state tracking/reporting in useAgent (Overall I like the changes though).

Summary

The useAgent hook has two bugs that cause incorrect state reporting when
agents disconnect or are missing:

1. Agent disconnections mid-conversation don't trigger failed state -
   Agents that disconnect after successful connection don't set failure reasons
2. listening state can occur with no agent participant - State shows
   'listening' when agent.internal.agentParticipant is null

---

Bug 1: Missing failure state on agent disconnection

Problem

When an agent successfully connects initially but then disconnects
mid-conversation, the state doesn't transition to failed and no failure
reasons are provided.

Expected vs Actual

Expected:

• agent.state should be 'failed'
• agent.failureReasons should be set

Actual:

• agent.state remains as 'listening'
• No failure reasons are set

Root Cause

It seems the failure detection in useAgentTimeoutIdStore (lines 192-204) only
works during initial connection timeout. Once the agent successfully connects,
the timeout completes and there's no mechanism to detect subsequent
disconnections.

File: components-js/packages/react/src/hooks/useAgent.ts

• Lines 192-204: Timeout logic
• Lines 436-446: Timeout lifecycle

Reproduction

// Agent process is started.
await session.start();
// Agent connects and works fine
// ...
// Kill the agent process
// Expected: state = 'failed', failureReasons = ['Agent disconnected']
// Actual: state = 'connecting' (or previous), failureReasons = null

---

Bug 2: Inconsistent listening state

Problem

agent.state can be 'listening' while agent.internal.agentParticipant is
null, indicating the agent is listening when no agent actually exists.

Expected vs Actual

Expected:

• State reflects actual agent presence
• 'listening' state should only occur when an agent participant exists

Actual:

agent.state === "listening";
agent.internal.agentParticipant === null; // Inconsistent!

Root Cause

It seems the state calculation (lines 394-425) sets state to 'listening' based
solely on localMicTrack:

if (localMicTrack) {
  state = "listening";
  bufferingSpeachLocally = true;
}

This happens when the local microphone is published (buffering audio) but no
agent has joined the room yet.

File: components-js/packages/react/src/hooks/useAgent.ts

• Lines 408-411: State calculation
• Line 453: Agent participant in internal state

Reproduction

See above.

Logs

System Info

System:
    OS: macOS 15.5
    CPU: (10) arm64 Apple M4
    Memory: 65.66 MB / 16.00 GB
    Shell: 5.9 - /bin/zsh
  Binaries:
    Node: 24.2.0 - /opt/homebrew/bin/node
    npm: 11.4.2 - /opt/homebrew/bin/npm
    pnpm: 10.18.3 - /opt/homebrew/bin/pnpm
    Watchman: 2025.05.26.00 - /opt/homebrew/bin/watchman
  Browsers:
    Chrome: 141.0.7390.123
    Firefox: 143.0.1
    Safari: 18.5
  npmPackages:
    @livekit/components-core: file:vendor/livekit-components-core-0.12.10.tgz => 0.12.10 
    @livekit/components-react: file:vendor/livekit-components-react-2.9.15.tgz => 2.9.15 
    @livekit/components-styles: ^1.1.6 => 1.1.6 


Using packed packages from f118da6e678c4a91be91c4dfc9b3b61eb7f64e2a (#1207)

Severity

blocking an upgrade

Additional Information

No response

[1egoman]

Hi @frankfarzan, thanks for opening an issue and raising these points!

In regards to 1 (Missing failure state on agent disconnection) - this wasn't a case that had been considered. At least initially, the only use case for failed was only for for handle agent connection timeouts on session start, though it was named in a generic fashion so that it could be extended to more cases in the future. The thing that makes me hesitate a little though it it could be possible for an agent to disconnect manually rather than disconnecting due to a crash or being terminated manually, so something like disconnected could make more sense as a status sometimes. Let me sync with some folks internally, but I am curious your thoughts on what you'd expect to see in both scenarios!

In regards to 2 (Inconsistent listening state) - this behavior you are seeing is actually by design! This is a feature called the pre connect audio buffer, which allows a user to speak while an agent is still connecting, and once the agent joins, the stored audio is sent to the agent to "catch it up" so it can resume as if it was listening the whole time. This reduces perceived latency for agent use cases when a user speaks first!

This will be documented more officially soon, but the state field being returned from the useAgent hook should be thought of as "the state your application should consider your agent to be in" rather than "the actual state your agent is in".

I do want to mention that some official documentation will be coming to the livekit docs site in the near future covering these new agents related hooks, but as of today that isn't ready quite yet. I'll make sure to surface both of these points to the relevant folks so that the docs explicitly call out both of these cases!

[frankfarzan]

Thanks @1egoman for the reply. It's good to hear docs are coming, but I think the best form of documentation is through API design itself. My understanding is that the intent here is to provide a higher-level abstraction that is more developer friendly compared to low-level constructs that exist today. This is what I would expect this abstraction to provide:

1. Ability to start a session, and determine that the agent is actually "ready" (aka "available"). Ready means it's actually connected and interactive. Currently, I'm hacking this together as such:

  if (session.connectionState === LiveKitConnectionState.Connected) {
    // "ready" means agent is ACTUALLY available
    // Must check: agent.state in ["listening", "thinking", "speaking"]
    if (
      (agent.state === "listening" ||
        agent.state === "thinking" ||
        agent.state === "speaking") &&
      agent.internal.agentParticipant !== null
    ) {
      return "ready";
    }
  ...
  }

It sounds like "listening" is being overloaded for two different semantics:
a) audio is buffered but agent is not actually listening. I think this would surprise users and lead to confusion.
b) agent is actually listening

Related question: Semantics of isAvailable, waitUntilAvailable.

2. Ability to determine when agent is no longer "ready". e.g. agent process crashes mid-conversation. It's not clear how to do this today which is critical for production-readiness. Related: it'd be good to test and design for such failure scenarios, e.g. I'm getting unhandled exceptions in low-level transcription-related code when the agent crashes (happy to file a separate issue if needed).

I'm not sure I understand this:

"the state your application should consider your agent to be in" rather than "the actual state your agent is in".

At the end of the day, my application needs to respond to the actual state and handle failures accordingly (e.g. stop the session, re-create the session, notify the end-user, etc.). It'd be good to get guidance on how to do this robustly.

Thanks!