Chat history not working with Agent executor

[arun1980in]

I have created 2 tools and have a custom agent. I added chat history using RunnableWithMessageHistory, but it seems the chat history is not being considered. Please see below. As can be seen below the expectation is that the 2nd invoke output should be able to respond with name, but it doesnt. I am using meta llama 3.1-70B instruct model.

I have also reviewed other discussions related to this #21764, but not luck. Please advice

# Defining custom tools using StructuredTool dataclass


# tool 1 to extract Origin and destination airport codes
class get_flight_info_input(BaseModel): # Enter the input description and type/properties
    origin:str= Field(description="orgin airport code for flight info, e.g. DFW")
    Destination:str= Field(description="departure airport code for flight info, e.g. JFK")
    
def get_flight_info_tool(origin, Destination): # Enter the logic for what this tool does
    flight_info={
        "Origin":origin,
        "Destination":Destination,
        "Airline":"KLM"
         }
    return json.dumps(flight_info)
   
    
get_flight_info = StructuredTool.from_function(
    func=get_flight_info_tool, # this is pointing to the logic of the tool
    name="get_flight_info_func", # this is the name of the function you want this to be
    description="get flight info between two locations", # this is the Description of the function
    args_schema=get_flight_info_input # passing the argument schema
)

# tool 2 to extract location and format for temperature
class get_current_weather_input(BaseModel): # Enter the input description and type/properties
    location:str= Field(description="The city and state, e.g. San Francisco, CA")
    format:str= Field(description="The temperature unit to use. Select celsius or fahrenheit by Inferring this from the users location. e.g. fahrenheit")

def get_current_weather(location, format): # Enter the logic for what this tool does
    get_weather_location={
        "format":format,
        "Destination":location,
            }
    return json.dumps(get_weather_location)
    
get_current_weather = StructuredTool.from_function(
    func=get_current_weather, # this is pointing to the logic of the tool
    name="get_current_weather", # this is the name of the function you want this to be
    description="Get the current weather", # this is the Description of the function
    args_schema=get_current_weather_input, # passing the argument schema
    handle_tool_error=True
)


tools=[get_flight_info,get_current_weather]

# creating session history session as per langchain documentation

store = {}

def get_session_history(session_id: str) -> BaseChatMessageHistory:
    if session_id not in store:
        store[session_id] = InMemoryChatMessageHistory()
    return store[session_id]

# creating my agent


prompt = ChatPromptTemplate.from_messages(
    [
        ("system", "You are a helpful assistant"),
        ("user", "{User_prompt}"),
        MessagesPlaceholder(variable_name="chat_history",optional=True),
        MessagesPlaceholder("agent_scratchpad"), # this is for agent intermediate steps
    ]
)
llm_with_tools = llm.bind_tools(tools)


agent = (
    {
        "User_prompt": lambda x: x["User_prompt"],
        "agent_scratchpad": lambda x: format_to_openai_tool_messages(x["intermediate_steps"]),
    }
    | prompt
    | llm_with_tools
    | OpenAIToolsAgentOutputParser()
)

# user prompt, agent creation and execution for flight info example
User_prompt=f"My name is XYZ. I am in Chicago. Hows the weather there. Also flight info from london to delhi"
agent_executor = AgentExecutor(agent=agent, tools=tools, verbose=True,handle_parsing_errors=False,return_intermediate_steps=True,max_iterations=4)
Agent_with_message_history = RunnableWithMessageHistory(agent_executor, get_session_history,input_messages_key="User_prompt",history_messages_key="chat_history",)

config = {"configurable": {"session_id": "abc2"}}

response=Agent_with_message_history.invoke({"chat_history":[HumanMessage(content=User_prompt)],"User_prompt":User_prompt},config=config)
print(response)

Here is the output of first invoke:
_Entering new AgentExecutor chain...
<|python_tag|>{"name": "get_current_weather", "parameters": {"location": "Chicago", "format": "fahrenheit"}}; {"name": "get_flight_info_func", "parameters": {"origin": "LHR", "Destination": "DEL"}}

Finished chain.
{'chat_history': [], 'User_prompt': 'My name is XYZ. I am in Chicago. Hows the weather there. Also flight info from london to delhi', 'output': '<|python_tag|>{"name": "get_current_weather", "parameters": {"location": "Chicago", "format": "fahrenheit"}}; {"name": "get_flight_info_func", "parameters": {"origin": "LHR", "Destination": "DEL"}}', 'intermediate_steps': []}_

#THen the 2nd invoke:
User_prompt=f"What is my name"
response=Agent_with_message_history.invoke({"chat_history":[HumanMessage(content=User_prompt)],"User_prompt":User_prompt},config=config)
print(response)

Output of 2nd invoke

Entering new AgentExecutor chain...
I don't have a specific function available to determine your name. The functions provided are for getting flight information and current weather. If you'd like to use one of these functions, I can help with that!

Finished chain.
{'chat_history': [HumanMessage(content='My name is XYZ. I am in Chicago. Hows the weather there. Also flight info from london to delhi'), AIMessage(content='<|python_tag|>{"name": "get_current_weather", "parameters": {"location": "Chicago", "format": "fahrenheit"}}; {"name": "get_flight_info_func", "parameters": {"origin": "LHR", "Destination": "DEL"}}')], 'User_prompt': 'What is my name', 'output': "I don't have a specific function available to determine your name. The functions provided are for getting flight information and current weather. If you'd like to use one of these functions, I can help with that!", 'intermediate_steps': []}

[dosubot[bot]]

The chat history is not being considered when using RunnableWithMessageHistory with your custom agent and tools because you need to set the output_messages_key property correctly when initializing the wrapper. Additionally, ensure that the input_messages_key and history_messages_key are set appropriately. Here is the relevant code snippet:

from langchain_core.chat_history import InMemoryChatMessageHistory
from langchain_core.runnables.history import RunnableWithMessageHistory
from langchain.agents import AgentExecutor, create_tool_calling_agent
from langchain_core.prompts import ChatPromptTemplate
from langchain_openai import ChatOpenAI
from langchain_core.tools import tool

model = ChatOpenAI(model="gpt-4o")
memory = InMemoryChatMessageHistory(session_id="test-session")
prompt = ChatPromptTemplate.from_messages(
    [
        ("system", "You are a helpful assistant."),
        ("placeholder", "{chat_history}"),
        ("human", "{input}"),
        ("placeholder", "{agent_scratchpad}"),
    ]
)

@tool
def magic_function(input: int) -> int:
    """Applies a magic function to an input."""
    return input + 2

tools = [magic_function]

agent = create_tool_calling_agent(model, tools, prompt)
agent_executor = AgentExecutor(agent=agent, tools=tools)

agent_with_chat_history = RunnableWithMessageHistory(
    agent_executor,
    lambda session_id: memory,
    input_messages_key="input",
    history_messages_key="chat_history",
    output_messages_key="output"  # Ensure this key matches the output of your agent
)

config = {"configurable": {"session_id": "test-session"}}
print(
    agent_with_chat_history.invoke(
        {"input": "Hi, I'm polly! What's the output of magic_function of 3?"}, config
    )["output"]
)
print("---")
print(agent_with_chat_history.invoke({"input": "Remember my name?"}, config)["output"])
print("---")
print(
    agent_with_chat_history.invoke({"input": "what was that output again?"}, config)[
        "output"
    ]
)

Make sure that the output_messages_key matches the key used by your agent to output messages and that the input_messages_key and history_messages_key are correctly set to handle the conversation history.

Additionally, consider using LangSmith for better inspection and debugging of your chain or agent. LangSmith allows you to log traces and inspect the internal workings of your application, which can be particularly useful for complex multi-step applications. To enable LangSmith, set the appropriate environment variables:

os.environ["LANGCHAIN_TRACING_V2"] = "true"
if not os.environ.get("LANGCHAIN_API_KEY"):
    os.environ["LANGCHAIN_API_KEY"] = getpass.getpass()

This will help you ensure that the chat history is being correctly managed and utilized in your application ^{[1][2][3][4][5]}.

To continue talking to Dosu, mention @dosu.

[arun1980in]

Hi, It appears that, chathistory is being populated, but its not be considered. i even tried setting the out_messages_key as suggested, but no change. See the Chat_history[HumanMessage]. the content is populated and it is expected that this Chat_history should be used by LLM to answer ``` Agent_with_message_history = RunnableWithMessageHistory(agent_executor, get_session_history,input_messages_key="User_prompt",history_messages_key="chat_history",output_messages_key="output") ``` Agent output: _``` I don't have a specific function available to determine your name. The functions provided are for getting flight information and current weather. If you'd like to use one of these functions, I can help with that!

Finished chain.

{'chat_history': [HumanMessage(content='My name is XYZ. I am in Chicago. Hows the weather there. Also flight info from london to delhi'), AIMessage(content='<|python_tag|>{"name": "get_current_weather", "parameters": {"location": "Chicago", "format": "fahrenheit"}}; {"name": "get_flight_info_func", "parameters": {"origin": "LHR", "Destination": "DEL"}}')], 'User_prompt': 'What is my name', 'output': "I don't have a specific function available to determine your name. The functions provided are for getting flight information and current weather. If you'd like to use one of these functions, I can help with that!", 'intermediate_steps': []} ```_

…

On Tue, Sep 17, 2024 at 2:19 AM dosubot[bot] ***@***.***> wrote: The chat history is not being considered when using RunnableWithMessageHistory with your custom agent and tools because you need to set the output_messages_key property correctly when initializing the wrapper. Additionally, ensure that the input_messages_key and history_messages_key are set appropriately. Here is the relevant code snippet: from langchain_core.chat_history import InMemoryChatMessageHistoryfrom langchain_core.runnables.history import RunnableWithMessageHistoryfrom langchain.agents import AgentExecutor, create_tool_calling_agentfrom langchain_core.prompts import ChatPromptTemplatefrom langchain_openai import ChatOpenAIfrom langchain_core.tools import tool model = ChatOpenAI(model="gpt-4o")memory = InMemoryChatMessageHistory(session_id="test-session")prompt = ChatPromptTemplate.from_messages( [ ("system", "You are a helpful assistant."), ("placeholder", "{chat_history}"), ("human", "{input}"), ("placeholder", "{agent_scratchpad}"), ] ) @tooldef magic_function(input: int) -> int: """Applies a magic function to an input.""" return input + 2 tools = [magic_function] agent = create_tool_calling_agent(model, tools, prompt)agent_executor = AgentExecutor(agent=agent, tools=tools) agent_with_chat_history = RunnableWithMessageHistory( agent_executor, lambda session_id: memory, input_messages_key="input", history_messages_key="chat_history", output_messages_key="output" # Ensure this key matches the output of your agent ) config = {"configurable": {"session_id": "test-session"}}print( agent_with_chat_history.invoke( {"input": "Hi, I'm polly! What's the output of magic_function of 3?"}, config )["output"] )print("---")print(agent_with_chat_history.invoke({"input": "Remember my name?"}, config)["output"])print("---")print( agent_with_chat_history.invoke({"input": "what was that output again?"}, config)[ "output" ] ) Make sure that the output_messages_key matches the key used by your agent to output messages and that the input_messages_key and history_messages_key are correctly set to handle the conversation history. Additionally, consider using LangSmith for better inspection and debugging of your chain or agent. LangSmith allows you to log traces and inspect the internal workings of your application, which can be particularly useful for complex multi-step applications. To enable LangSmith, set the appropriate environment variables: os.environ["LANGCHAIN_TRACING_V2"] = "true"if not os.environ.get("LANGCHAIN_API_KEY"): os.environ["LANGCHAIN_API_KEY"] = getpass.getpass() This will help you ensure that the chat history is being correctly managed and utilized in your application [1] <https://github.com/langchain-ai/langchain/blob/master/docs/docs/how_to/chatbots_tools.ipynb> [2] <https://github.com/langchain-ai/langchain/blob/master/docs/docs/how_to/agent_executor.ipynb> [3] <https://github.com/langchain-ai/langchain/blob/master/docs/docs/how_to/message_history.ipynb> [4] <https://github.com/langchain-ai/langchain/blob/master/docs/docs/how_to/migrate_agent.ipynb> [5] <https://github.com/langchain-ai/langchain/blob/master/docs/docs/tutorials/qa_chat_history.ipynb> . *To continue talking to Dosu <https://dosu.dev>, mention @dosu <https://github.com/dosu>.* — Reply to this email directly, view it on GitHub <#26567 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BG72HMOZF7NX36YJBGPUXODZW7CYJAVCNFSM6AAAAABOKXGXI2VHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTANRWGY3TKOA> . You are receiving this because you authored the thread.Message ID: ***@***.*** com>

[dosubot[bot]]

The issue with the chat history not being considered by RunnableWithMessageHistory in your custom agent setup likely stems from how the chat history is being managed and passed between invocations. Here are a few key points to ensure proper handling:

1. Update Chat History After Each Invocation:
   Ensure that the chat history is updated with both the user input and the agent's response after each invocation. This step is crucial for maintaining the context across multiple interactions.

2. Correct Key Settings:
   Verify that the input_messages_key, history_messages_key, and output_messages_key are correctly set and match the keys used in your agent's input and output.

Here is an updated version of your code with these considerations:

from langchain_core.messages import AIMessage, HumanMessage
from langchain_core.chat_history import InMemoryChatMessageHistory
from langchain_core.runnables.history import RunnableWithMessageHistory
from langchain.agents import AgentExecutor, create_tool_calling_agent
from langchain_core.prompts import ChatPromptTemplate
from langchain_openai import ChatOpenAI
from langchain_core.tools import tool
from pydantic import BaseModel, Field
import json

# Define custom tools
class get_flight_info_input(BaseModel):
    origin: str = Field(description="origin airport code for flight info, e.g. DFW")
    Destination: str = Field(description="departure airport code for flight info, e.g. JFK")

def get_flight_info_tool(origin, Destination):
    flight_info = {
        "Origin": origin,
        "Destination": Destination,
        "Airline": "KLM"
    }
    return json.dumps(flight_info)

get_flight_info = StructuredTool.from_function(
    func=get_flight_info_tool,
    name="get_flight_info_func",
    description="get flight info between two locations",
    args_schema=get_flight_info_input
)

class get_current_weather_input(BaseModel):
    location: str = Field(description="The city and state, e.g. San Francisco, CA")
    format: str = Field(description="The temperature unit to use. Select celsius or fahrenheit by Inferring this from the users location. e.g. fahrenheit")

def get_current_weather(location, format):
    get_weather_location = {
        "format": format,
        "Destination": location,
    }
    return json.dumps(get_weather_location)

get_current_weather = StructuredTool.from_function(
    func=get_current_weather,
    name="get_current_weather",
    description="Get the current weather",
    args_schema=get_current_weather_input,
    handle_tool_error=True
)

tools = [get_flight_info, get_current_weather]

# Create session history
store = {}

def get_session_history(session_id: str) -> BaseChatMessageHistory:
    if session_id not in store:
        store[session_id] = InMemoryChatMessageHistory()
    return store[session_id]

# Create agent
prompt = ChatPromptTemplate.from_messages(
    [
        ("system", "You are a helpful assistant"),
        ("user", "{User_prompt}"),
        MessagesPlaceholder(variable_name="chat_history", optional=True),
        MessagesPlaceholder("agent_scratchpad"),
    ]
)
llm_with_tools = llm.bind_tools(tools)

agent = (
    {
        "User_prompt": lambda x: x["User_prompt"],
        "agent_scratchpad": lambda x: format_to_openai_tool_messages(x["intermediate_steps"]),
    }
    | prompt
    | llm_with_tools
    | OpenAIToolsAgentOutputParser()
)

agent_executor = AgentExecutor(agent=agent, tools=tools, verbose=True, handle_parsing_errors=False, return_intermediate_steps=True, max_iterations=4)
Agent_with_message_history = RunnableWithMessageHistory(agent_executor, get_session_history, input_messages_key="User_prompt", history_messages_key="chat_history", output_messages_key="output")

config = {"configurable": {"session_id": "abc2"}}

# First invocation
User_prompt = "My name is XYZ. I am in Chicago. Hows the weather there. Also flight info from london to delhi"
response = Agent_with_message_history.invoke({"User_prompt": User_prompt}, config=config)
chat_history = [
    HumanMessage(content=User_prompt),
    AIMessage(content=response["output"])
]
print(response)

# Second invocation
User_prompt = "What is my name"
response = Agent_with_message_history.invoke({"chat_history": chat_history, "User_prompt": User_prompt}, config=config)
chat_history.extend([
    HumanMessage(content=User_prompt),
    AIMessage(content=response["output"])
])
print(response)

Ensure that the chat_history is correctly updated and passed between invocations. This should help the agent consider the chat history in its responses ^[1][2][3][4].

To continue talking to Dosu, mention @dosu.

[arun1980in]

It is still the same issue.

output of 2nd invocation. the Chat_history has the Human_message and AI_message. it has the information about the name XYZ. which is what it should respond back with when the question is asked "What is my name"

> Entering new AgentExecutor chain...
I don't have a specific function available to determine your name. The functions provided are for getting flight information and current weather. If you'd like to use one of these functions, I can help with that!

> Finished chain.
{'User_prompt': 'What is my name', 'chat_history': [HumanMessage(content='My name is XYZ. I am in Chicago. Hows the weather there. Also flight info from london to delhi'), AIMessage(content='<|python_tag|>{"name": "get_current_weather", "parameters": {"location": "Chicago", "format": "fahrenheit"}}; {"name": "get_flight_info_func", "parameters": {"origin": "LHR", "Destination": "DEL"}}')], 'output': "I don't have a specific function available to determine your name. The functions provided are for getting flight information and current weather. If you'd like to use one of these functions, I can help with that!", 'intermediate_steps': []}