docs/how_to/multimodal_inputs/

[giscus[bot]]

docs/how_to/multimodal_inputs/

Here we demonstrate how to pass multimodal input directly to models.

https://python.langchain.com/docs/how_to/multimodal_inputs/

[VpkPrasanna]

HI is there any lcel way of implementation of image input ?

[adithya04dev]

is there a way to have chat history? and also automatic memory management using chains,runnables?

[syntaxtrash]

use RunnableWithMessageHistory.

[jvlinsta]

LCEL way:

messages | llm

[MohamadRezaei0]

what about audio input?

[Hamza5]

If anyone wants to allow the agent to use multimodal output from a tool call (common scenario in complex tasks), for example, let the model capture a screenshot, check this discussion on how to do it:
#25881

[Albaeld]

This simply does not work with openai:gpt-4.1. I get:
Error code: 400 - {'error': {'message': "Missing required parameter: 'messages[0].content[1].file'.", 'type': 'invalid_request_error', 'param': 'messages[0].content[1].file', 'code': 'missing_required_parameter'}}

[gutbash]

yeah but does it work with memory is the question

[brunoshine]

Hi all, there is an open issue for this here #31505 with some working examples for OpenAI and Gemini, and still failing code for Azure OpenAI

[gutbash]

Hi all, there is an open issue for this here #31505 with some working examples for OpenAI and Gemini, and still failing code for Azure OpenAI

it doesnt test with memory. i tried a similar implementation and it didnt work with memory.

[brunoshine]

But it works as I have an agent implemented in langgraph that uses postgres as the node state / memory

[dayvidborges]

Hey guys, gutbash the second option i showed you can perfectly fine use with memory

[brunoshine]

Hi all,

I'm also facing the same issue that @Albaeld reported.
I have a langgraph agent with several nodes and Im calling the graph via agent.astream_events(**kwargs, version="v2").

On the llm node I have:

 messages = prompt + state["messages"]
 chat = ChatPromptTemplate.from_messages(messages)
logger.debug("-------------------------------------------------------------------------------->> invoking LLM with config")
logger.debug(f"{chat}")
logger.debug("-------------------------------------------------------------------------------->> invoking LLM...")
response = await model.ainvoke(chat.format_prompt(), config)

this outputs:

== APP == -------------------------------------------------------------------------------->> invoking LLM with config
== APP == input_variables=[] input_types={} partial_variables={} messages=[SystemMessage(content="You are a concierge agent. You are very helpful and friendly. You are very polite and respectful. You are always ready to help.\n\ncurrent date/time: 2025-06-05 19:32:40\n\nuser context: {'id': '123123123', 'name': 'Bruno Figueiredo', 'avatar': ''}", additional_kwargs={}, response_metadata={}), HumanMessage(content='olá', additional_kwargs={}, response_metadata={}, id='8fa5db18-69b0-4085-9d4e-aa16847da243'), AIMessage(content='Olá, Bruno! 😊 Como posso ajudar você hoje?', additional_kwargs={}, response_metadata={'finish_reason': 'stop', 'model_name': 'gpt-4o-2024-11-20', 'system_fingerprint': 'fp_ee1d74bde0'}, id='run-2df83ba9-a843-4029-a020-0d41465fddf3'), HumanMessage(content=[{'type': 'text', 'text': 'resume'}, {'type': 'file', 'source_type': 'url', 'url': 'https://www.w3.org/WAI/ER/tests/xhtml/testfiles/resources/pdf/dummy.pdf'}], additional_kwargs={}, response_metadata={}, id='363b871a-d9bc-4f40-ab56-5138486e7f07')]
== APP == -------------------------------------------------------------------------------->> invoking LLM...
== APP == Error invoking model: Error code: 400 - {'error': {'message': "Missing required parameter: 'messages[3].content[1].file'.", 'type': 'invalid_request_error', 'param': 'messages[3].content[1].file', 'code': 'missing_required_parameter'}}

any thoughts? thanks

[ccurme]

Hi @brunoshine, it looks like you're passing in a PDF via a URL, from what I can tell that's not supported by OpenAI, see their relevant docs here.

See LangChain docs here on working with PDFs and OpenAI.

[gutbash]

bro i dont understand why multimodality is still not really supported and definitely not well documented. they should've added multimodal handling before LCEL even existed.

[cividanieltorres]

I am having a bit of a misunderstanding. I am following the guidelines of this page to build the following input message:

{
  "messages": [
    {
      "type": "human",
      "content": [
         {"type": "text", "text": "what do you see?"},
         {
          "type": "image",
          "source_type": "url",
          "url": "https://xyz.com/example-image.png"
         }
      ]
    }
  ]
}

However, I am getting the following error both with latest langchain-openai and langchain-google-vertexai library versions:

• OpenAI: openai.BadRequestError: Error code: 400 - {'error': {'message': "Invalid value: 'image'. Supported values are: 'text', 'image_url', 'input_audio', 'refusal', 'audio', and 'file'.", 'type': 'invalid_request_error', 'param': 'messages[1].content[1].type', 'code': 'invalid_value'}}
• VertexAI: ValueError: Only text, image_url, and media types are supported!

[gutbash]

I am having a bit of a misunderstanding. I am following the guidelines of this page to build the following input message:

{
  "messages": [
    {
      "type": "human",
      "content": [
         {"type": "text", "text": "what do you see?"},
         {
          "type": "image",
          "source_type": "url",
          "url": "https://xyz.com/example-image.png"
         }
      ]
    }
  ]
}

However, I am getting the following error both with latest langchain-openai and langchain-google-vertexai library versions:

• OpenAI: openai.BadRequestError: Error code: 400 - {'error': {'message': "Invalid value: 'image'. Supported values are: 'text', 'image_url', 'input_audio', 'refusal', 'audio', and 'file'.", 'type': 'invalid_request_error', 'param': 'messages[1].content[1].type', 'code': 'invalid_value'}}
• VertexAI: ValueError: Only text, image_url, and media types are supported!

hey 👋 i see the problem with your specific error. try changing the type on the image to specifically image_url.