there is some errors. when use chat_history with image_url (AgentExecutor with tools)

[chanyong-owl]

Checked other resources

• I added a very descriptive title to this question.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.

Commit to Help

• I commit to help with one of those options 👆

Example Code

chat_historys=[SystemMessage(content='You are a helpful assistant.'), HumanMessage(content=[{'type': 'image_url', 'image_url': {'url': '*****'}}, {'type': 'text', 'text': 'who is he'}]), AIMessage(content="I can't identify individuals, but I can tell you that this is a business card for someone named ****, who is a Project Manager. The card includes contact information such as phone numbers, an email address, a website, and a physical address."), HumanMessage(content='what is phone number'), AIMessage(content='The phone numbers listed on the business card are:\n\n- *********')]

agent = create_openai_functions_agent(llm, tools, prompt)
# tools : google search , weather search and so on.

agent_executor = AgentExecutor(agent=agent, tools=tools,verbose=True)

response = agent_executor.invoke({
                "input":params.input,

                "chat_history": chat_historys
})

Description

my chat history is like [chat_historys]. it's comibined common text, and image_url. If there is only common text without image_url, it's always ok.
when there is message historys with image_url, and common msg, and then if i ask about some question that dosen't use tools like qustion "who are you", there is no error.

when there is message historys with image_url, and common msg, and then if i ask about some question that use tools like qustion "how is the weather", there is error.

the error is like below.

Error code: 400 - {'error': {'code': 'BadRequest', 'message': "'function' is not an allowed role. The allowed roles are ['system', 'user', 'assistant'].", 'param': None, 'type': None}}

is there anyone who has experience like that?

Help me T.T

System Info

nothing

[dosubot[bot]]

Hey there, @chanyong-owl! I'm here to help you out with any bugs, questions, or contributions you have in mind. Let's dig into this issue together and figure out a solution.

The error you're encountering is due to the presence of unsupported roles in the chat_history. The allowed roles for messages in the chat_history when using the AgentExecutor in LangChain are SystemMessage and HumanMessage ^[1].

To resolve this issue, ensure that your chat_history only includes messages with the allowed roles. Here is an example of how you can structure your chat_history correctly:

chat_historys = [
    SystemMessage(content='You are a helpful assistant.'),
    HumanMessage(content=[{'type': 'image_url', 'image_url': {'url': '*****'}}, {'type': 'text', 'text': 'who is he'}]),
    HumanMessage(content='what is phone number')
]

agent = create_openai_functions_agent(llm, tools, prompt)
agent_executor = AgentExecutor(agent=agent, tools=tools, verbose=True)

response = agent_executor.invoke({
    "input": params.input,
    "chat_history": chat_historys
})

Make sure that all messages in chat_history are either SystemMessage or HumanMessage. If you need to include responses from the AI, they should be handled separately and not included in the chat_history as AIMessage.

Details

About Dosu
This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

To continue the conversation, mention @dosu.

[MaximilianCroissant]

This is still an issue. Any time a function message is in history and an image is included in the user message, this error pops up.