agent.astream_events() does not return "on_chat_model_stream"

[alexgg278]

Checked other resources

• I added a very descriptive title to this question.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.

Commit to Help

• I commit to help with one of those options 👆

Example Code

from langchain.agents.agent import AgentExecutor
from langchain_openai import AzureChatOpenAI
from langchain.agents import Tool, initialize_agent
from langchain_community.utilities import BingSearchAPIWrapper

agent_model_azure = AzureChatOpenAI(
            azure_endpoint=***,
            azure_deployment=***,
            api_version=***,
            api_key=***,
            max_tokens=4_000,
            temperature=0
        )

search = BingSearchAPIWrapper(
        bing_search_url=***,
        bing_subscription_key=***
    )


search_tool = Tool(name="search_tool", description = "search the net", func = search.run)
tools = [search_tool]

agent = initialize_agent(tools,
                         agent_model_azure,
                         agent='zero-shot-react-description',
                         verbose=True,
                         handle_parsing_errors=True)

async for event in agent.astream_events(input=prompt, version="v1"):
    kind = event["event"]
    print(event)

Description

I am trying to use astream_events on a langchain AgentExecutor with the goal of only streaming the final output generation of the Agent. I have read in the documentation that to get the streaming, the events should be filtered looking only for "on_chat_model_stream" events. However, when I print all the events I don't see any "on_chat_model_stream" events, not even in between the "on_chat_model_start" and the "on_chatModel_end" events.

Take a look to the screnshoot:

In the docs there is a clear example on how to stream the final answer, using "on_chat_model_stream", but you can only do that if you get this events out of the iterator: https://python.langchain.com/docs/expression_language/streaming#filtering-events

System Info

langchain-core = "0.1.16"
openai = ">1"
langchain-openai = "^0.0.5"
langchain = "0.1.0"

[dosubot[bot]]

¡Hola de nuevo @alexgg278! Es un placer verte por aquí otra vez. Espero que te encuentres bien.

Based on the information you've provided and the context I've found, it seems like you're experiencing an issue where the astream_events method is not yielding the "on_chat_model_stream" events as expected. This is indeed strange, as the astream_events method is designed to asynchronously stream events from a bound runnable, and "on_chat_model_stream" should be one of the events that it yields.

However, I found a similar issue in the LangChain repository: Issue: How can I get back a streaming response with AgentExecutors?. This issue was closed, but it might contain some useful information or a solution to your problem.

Also, there's an unsolved discussion: AgentExecutor is not streaming the output. This discussion might be related to your issue, and keeping an eye on it could provide some insights or solutions.

Regarding the astream_events method, it's part of the RunnableBindingBase class in the LangChain codebase. This class acts as a wrapper around another runnable, allowing for the invocation of that runnable's methods with additional configurations and arguments. The astream_events method specifically deals with streaming events asynchronously, making it suitable for scenarios where events are produced over time and need to be processed as they arrive. Here's the relevant part of the code:

async def astream_events(
    self,
    input: Input,
    config: Optional[RunnableConfig] = None,
    **kwargs: Optional[Any],
) -> AsyncIterator[StreamEvent]:
    async for item in self.bound.astream_events(
        input, self._merge_configs(config), **{**self.kwargs, **kwargs}
    ):
        yield item

You can find this in libs/core/langchain_core/runnables/base.py.

The events returned by the astream_events method are instances of StreamEvent. However, the specific structure or types of StreamEvent are not defined within the provided context, meaning the detailed structure or possible values contained within a StreamEvent instance cannot be determined from the given code snippets.

I hope this information helps. If you have more details or if there's anything else you'd like to know, please let me know.

Sources

• AgentExecutor is not streaming the output
• Issue: How can I get back a streaming response with AgentExecutors?
• libs/core/langchain_core/runnables/base.py

About Dosu

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[alexgg278]

CLosed

[mdenison]

I am having the same issue, were you able to solve this?

[straygar]

@alexgg278 Heya! Could you provide more info on how you worked around this?

[TomTom101]

+1 What did solve this for you?

[TomTom101]

Same here! I have a setup running locally with on_chat_model_stream being yielded, the very same code, same dependencies in Docker do not yield on_chat_model_stream. I am getting crazy!

[ImadSaddik]

I am also having the same problem, you closed this issue because you solved it ? @alexgg278

[ImadSaddik]

Hello everyone,

I don't know if this will help you solve the issue or not but give it a try. I was using Python 3.11 locally and the astream_events was returning the on_chat_model_stream event, but in the Docker container we were using Python 3.10.

Here, you can see that when using Python 3.10, we don't get the on_chat_model_stream event:

But when switching to Python 3.11, we get it:

So the solution for me was to just change the Python version to 3.11. @TomTom101 @straygar @mdenison

[straygar]

Yup, thanks @ImadSaddik! We had found this out a while ago, but forgot to post it.

Should this not be raising an exception on 3.10 tbh?

[ImadSaddik]

Yes, I think so.