Facing JSONDecodeError randomly in between groupchat

[gvijqb]

I have a groupchat consisting of 3 agents with OpenRouter LLMs behind 2 of them and OpenAI behind one of them.

I have often observed that randomly one of the Agents (99% powered by OpenRouter LLM) ends up giving this error in between an ongoing groupchat:

Error in MyAgent initialization: JSONDecodeError: Expecting value: line 1065 column 1 (char 5852)
Traceback:
Traceback (most recent call last):

  File "/home/ubuntu/miniconda/envs/myenv/lib/python3.11/site-packages/autogen_agentchat/teams/_group_chat/_chat_agent_container.py", line 84, in handle_request
    async for msg in self._agent.on_messages_stream(self._message_buffer, ctx.cancellation_token):

  File "/home/ubuntu/miniconda/envs/myenv/lib/python3.11/site-packages/autogen_agentchat/agents/_assistant_agent.py", line 953, in on_messages_stream
    async for inference_output in self._call_llm(

  File "/home/ubuntu/miniconda/envs/myenv/lib/python3.11/site-packages/autogen_agentchat/agents/_assistant_agent.py", line 1107, in _call_llm
    model_result = await model_client.create(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/home/ubuntu/miniconda/envs/myenv/lib/python3.11/site-packages/autogen_ext/models/openai/_openai_client.py", line 676, in create
    result: Union[ParsedChatCompletion[BaseModel], ChatCompletion] = await future
                                                                     ^^^^^^^^^^^^

  File "/home/ubuntu/miniconda/envs/myenv/lib/python3.11/asyncio/futures.py", line 287, in __await__
    yield self  # This tells Task to wait for completion.
    ^^^^^^^^^^

  File "/home/ubuntu/miniconda/envs/myenv/lib/python3.11/asyncio/tasks.py", line 349, in __wakeup
    future.result()

  File "/home/ubuntu/miniconda/envs/myenv/lib/python3.11/asyncio/futures.py", line 203, in result
    raise self._exception.with_traceback(self._exception_tb)

  File "/home/ubuntu/miniconda/envs/myenv/lib/python3.11/asyncio/tasks.py", line 277, in __step
    result = coro.send(None)
             ^^^^^^^^^^^^^^^

  File "/home/ubuntu/miniconda/envs/myenv/lib/python3.11/site-packages/openai/resources/chat/completions/completions.py", line 2454, in create
    return await self._post(
           ^^^^^^^^^^^^^^^^^

  File "/home/ubuntu/miniconda/envs/myenv/lib/python3.11/site-packages/openai/_base_client.py", line 1791, in post
    return await self.request(cast_to, opts, stream=stream, stream_cls=stream_cls)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/home/ubuntu/miniconda/envs/myenv/lib/python3.11/site-packages/openai/_base_client.py", line 1596, in request
    return await self._process_response(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/home/ubuntu/miniconda/envs/myenv/lib/python3.11/site-packages/openai/_base_client.py", line 1685, in _process_response
    return await api_response.parse()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/home/ubuntu/miniconda/envs/myenv/lib/python3.11/site-packages/openai/_response.py", line 430, in parse
    parsed = self._parse(to=to)
             ^^^^^^^^^^^^^^^^^^

  File "/home/ubuntu/miniconda/envs/myenv/lib/python3.11/site-packages/openai/_response.py", line 265, in _parse
    data = response.json()
           ^^^^^^^^^^^^^^^

  File "/home/ubuntu/miniconda/envs/myenv/lib/python3.11/site-packages/httpx/_models.py", line 832, in json
    return jsonlib.loads(self.content, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/home/ubuntu/miniconda/envs/myenv/lib/python3.11/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/home/ubuntu/miniconda/envs/myenv/lib/python3.11/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/home/ubuntu/miniconda/envs/myenv/lib/python3.11/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None

json.decoder.JSONDecodeError: Expecting value: line 1065 column 1 (char 5852)

Any suggestions on how to handle this? I have tried adding fallback on on_messages but I guess this is happening even before it reaches on_messages. I am not sure how this can be handled permanently, especially considering my dependency on OpenRouter LLMs.

Just FYI, I am also setting structured_output and json_output as False in LLM config. But still getting this.

Autogen version:

autogen-agentchat==0.6.4
autogen-core==0.6.4
autogen-ext==0.6.4

[gvijqb]

Any suggestions on this? @ekzhu