sending images as input to a react agent

[wisefinger]

Checked other resources

• I added a very descriptive title to this question.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.

Commit to Help

• I commit to help with one of those options 👆

Example Code

from app.config import Config
from app.llm.utils import get_current_dialogue
from app.state import memory
from langchain_openai import ChatOpenAI
from langchain.agents import AgentExecutor, create_react_agent
from app.llm.react_template import get_react_prompt_template
from langchain_community.agent_toolkits import GmailToolkit


def graph_agent(session_id: str, question: str) -> tuple[str, list]:
    """
    This function is the entry point for the graph agent.
    It will take the user message and the session id and return the response from the agent.
    """
    Config.load_config()
    # Define the LLM
    llm = ChatOpenAI(model="gpt-4o-mini",temperature=0)
    # Get the react prompt template
    prompt_template = get_react_prompt_template()
    # define the tools
    toolkit = GmailToolkit()
    tools = toolkit.get_tools()
    # Construct the ReAct agent
    agent = create_react_agent(llm, tools, prompt_template)
    # Create an agent executor by passing in the agent and tools
    agent_executor = AgentExecutor(agent=agent, tools=tools, verbose=True)
    # Get the current dialogue
    current_dialogue = get_current_dialogue(session_id, limit=10)
    # invoke the agent to get the result
    result = agent_executor.invoke({"input": question})


    return result, current_dialogue

Description

I have created a working react agent using langchain. I was wondering how I would be able to send input to the agent based on a local image. For example, upload an image of an invoice. I would then add instructions to prompt on mailing it to a specific email address. When you define a custom ChatGPT it comes out of the box, but I can't find clear documentation on how to do it using langchain and langraph.

System Info

langchain==0.2.15
langchain-chroma==0.1.1
langchain-community==0.2.14
langchain-core==0.2.36
langchain-experimental==0.0.57
langchain-openai==0.1.3
langchain-pinecone==0.1.1
langchain-text-splitters==0.2.2
langchainhub==0.1.15

[dosubot[bot]]

To send an image as input to a React agent using LangChain, you can use the HumanMessage class to create a message that includes both the image and the text prompt. Here is an example of how you can set this up to upload an image of an invoice and prompt it to mail to a specific email address:

import base64
import httpx
from langchain_core.messages import HumanMessage
from langchain_openai import ChatOpenAI

# Load and encode the image
image_path = "path/to/your/invoice.jpg"
with open(image_path, "rb") as image_file:
    image_data = base64.b64encode(image_file.read()).decode("utf-8")

# Define the message with an image and text
message = HumanMessage(
    content=[
        {"type": "text", "text": "Please send this invoice to example@example.com."},
        {
            "type": "image_url",
            "image_url": {"url": f"data:image/jpeg;base64,{image_data}"},
        },
    ],
)

# Initialize the model
model = ChatOpenAI(model="gpt-4o")

# Invoke the model with the message
response = model.invoke([message])
print(response.content)

This code sets up a message that includes an image and a user message asking to send the invoice to a specified email address. The ChatOpenAI class from the langchain_openai package is used to interact with OpenAI's models, which support image inputs. The image is encoded in base64 and included in the message content ^[1][2][3][4].

To continue talking to Dosu, mention @dosu.

[rpgeddam]

I'm trying to solve the same thing. I tried sending the image as a base64 encoded string similar to @dosu 's example, but that just returns nonsense. The Messages formatting doesn't seem to work with react agent.

[wisefinger]

Hi there, you need to use another agent type agent = create_structured_chat_agent(llm, tools, prompt_template)
and then add the image to your message like this:
`def invoke_image( image_data: str) -> tuple[str, list]:
"""
This function is the entry point sending images to the Patrick graph agent.
"""
Config.load_config()
# Define the LLM
llm = ChatOpenAI(model="gpt-4o",temperature=0)

    # Define the message with an image and text
    message = HumanMessage(
        content=[
            {"type": "text", "text": """This is an image of a receipt. Please provide the following details: the date, description, amount, payment method, and vendor.
            Reply with the following format:
            Do you want to create this new expense?
             - Date: [date]
             - Description: [description]
             - Amount: [amount]
             - Payment Method: [payment method]
             - Vendor: [vendor]
             
             Do not add any other information.
             """},

            {
                "type": "image_url",
                "image_url": {"url": f"data:image/jpeg;base64,{image_data}"},
            },
        ],
    )

` I tested it and for me this works fine

[sahill3]

Is there any article or any other resource that I can refer to? Langchain Docs do not have examples that include image input to "agents".

[rahul-08-11]

To feed an image to a react agent the following code worked for me. Once i backtrack their library code i came to realize that the input invoke method could accept either Dict or Any value according to its type annotation 'inputs: Union[dict[str, Any], Any]' (attached image) Despite this passing any value other then dict without key "message" throws an error.

##1.  create llm instance
llm = AzureChatOpenAI(... )

##2. create and setup langchain compatible tool
@tool
def add_tool(a  : int , b : int) -> int:
    """Add two numbers"""
    return a + b

tools = [
    Tool(
        name="add",
        func=add_tool,
        description="Add two numbers",
    )
]

##3.  create a react agent with basic config
agent = create_react_agent(
                llm,
              tools
            )
##4. prepare your message
message = HumanMessage(content=[
            {
                "type":"text",
                "text": "Describe the image"
            },
            {
                "type":"image_url",
                "image_url": {"url": "YOUR_PUBLIC_IMAGE_URL"},

            }
        ])

##5. pass correct argument in invoke method as shown
result = agent.invoke(input={"messages": [message]})

Additional note:
langchain-openai==0.3.28
langgraph==0.5.3
langchain-community==0.3.27
langchain-mcp-adapters==0.1.9