How to use ImagePromptTemplate

[Shuntw6096]

Checked other resources

• I added a very descriptive title to this question.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.

Commit to Help

• I commit to help with one of those options 👆

Example Code

How to use ImagePromptTemplate?

Description

How to use ImagePromptTemplate?
there is no info in the doc.
anyone has sample code?

System Info

Ubuntu 22.04

[dosubot[bot]]

To use ImagePromptTemplate in the LangChain framework, you can follow this example. This demonstrates initializing ImagePromptTemplate, formatting a prompt with an image name, using a direct URL, or a local path to an image:

from langchain_core.prompts.image import ImagePromptTemplate

# Initialize with a template if needed
image_prompt_template = ImagePromptTemplate(template={"url": "http://example.com/{image_name}.jpg"})

# Format the prompt with an image name
formatted_prompt = image_prompt_template.format(image_name="your_image_name")

# Using a direct URL
formatted_prompt_direct_url = image_prompt_template.format(url="http://example.com/your_image_name.jpg")

# Using a local path (converts image to a data URL)
formatted_prompt_path = image_prompt_template.format(path="/path/to/your/image.jpg")

print(formatted_prompt)
print(formatted_prompt_direct_url)
print(formatted_prompt_path)

This code snippet shows how to create an image prompt using ImagePromptTemplate by specifying an image through a template URL, a direct URL, or a local path. When using a local path, the image is converted to a data URL. For more details, you can refer to the ImagePromptTemplate class in the LangChain repository.

Details

Sources

• libs/core/langchain_core/prompts/image.py

About Dosu
This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

To continue the conversation, mention @dosu.

[chriss1245]

How would you invoke an llm with this prompt. I have seen it does not contain the method invoke?

[thdesc]

@chriss1245 Indeed but maybe you can use the ImagePromptTemplate with a HumanMessagePromptTemplate (or any _StringImageMessagePromptTemplate object). I haven't succeeded to make it work though...

[ryanoley]

I have not been able to get ImagePromptTemplate template to work after lots of trying, but @thdesc you are right, you can use HumanMessagePromptTemplate.

Here is how I did it. You can break up the messages also, but that does not work with all LLMs right now (gemini in particular requires a "text" type in the request)

import base64
from mimetypes import guess_type

# Function to encode a local image into data URL 
def local_image_to_data_url(image_path):
    mime_type, _ = guess_type(image_path)
    # Default to png
    if mime_type is None:
        mime_type = 'image/png'

    # Read and encode the image file
    with open(image_path, "rb") as image_file:
        base64_encoded_data = base64.b64encode(image_file.read()).decode('utf-8')

    # Construct the data URL
    return f"data:{mime_type};base64,{base64_encoded_data}"


prompt_template =  HumanMessagePromptTemplate.from_template(
            template=[
                {"type": "text", "text": "Summarize this image"},
                {
                    "type": "image_url",
                    "image_url": "{encoded_image_url}",
                },
            ]
        )

summarize_image_prompt = ChatPromptTemplate.from_messages([prompt_template])

gpt4_turbo_0409_llm = <initialize-a-langchain-AzureChatOpenAI-object-or-other-llm-here>
gpt4_image_chain = summarize_image_prompt | gpt4_turbo_0409_llm 

img_file = "local/path/to/page_3.png"
page3_encoded = local_image_to_data_url(img_file)

response = gpt4_image_chain.invoke(input={"encoded_image_url":page3_encoded})

[nathanjones4323]

@ryanoley Thank you!

I ended up using this inside of an Agent workflow. Here is an example in case anyone wants to see how to use your snippet with an Agent and using the image_url directly instead of an encoded image

from langchain.agents import AgentExecutor, create_openai_tools_agent
from langchain.schema import SystemMessage
from langchain_community.tools import CopyFileTool
from langchain_core.prompts import (
    ChatPromptTemplate,
    HumanMessagePromptTemplate,
    MessagesPlaceholder,
)
from langchain_openai import ChatOpenAI

# Define the prompt template with the image_url
prompt_messages = [
    SystemMessage(content="Describe the image provided"),
    HumanMessagePromptTemplate.from_template(
        template=[
            {"type": "image_url", "image_url": {"url": "{image_url}"}},
        ]
    ),
    MessagesPlaceholder("agent_scratchpad"),
]

prompt = ChatPromptTemplate(messages=prompt_messages)

# Define the LLM model
llm = ChatOpenAI(model="gpt-4o")

# Define a dummy tool
tools = [CopyFileTool()]

# Create an Agent that can be used to call the tools we defined
agent = create_openai_tools_agent(llm, tools, prompt)

# Define the AgentExecutor properly
agent_executor = AgentExecutor(
    agent=agent,
    tools=tools,
)

# Example image URL you want to pass in at runtime
image_url = "https://fastly.picsum.photos/id/450/200/200.jpg?hmac=DluUYibC-zBoNHLOHsO6aHIuiA3pDhholFjiR5KcwR0"

# Execute the Agent with the image URL
response = agent_executor.invoke({"image_url": image_url, "agent_scratchpad": []})

[batjko]

Using the HumanMessagePromptTemplate worked for me as well.

As the template property takes a List, you can also pass multiple image URLs, e.g.:

        HumanMessagePromptTemplate.from_template(
            template=[
                {"type": "image_url", "image_url": {"url": url}} for url in image_urls
            ]
        ),

[thdesc]

After some investigation, I think some changes must be made to the following file: libs/core/langchain_core/prompts/chat.py, to ensure ImagePromptTemplate works with HumanMessagePromptTemplate.

Suggested Changes:

1. Change input_variables = None to input_variables = []:

   • Line 453
   • Line 465

2. Change formatted = prompt.format(**inputs) to formatted = prompt.format(**kwargs):

   • Line 547

Code Example:

If you apply these changes, the following code should work:

from langchain_core.prompts import HumanMessagePromptTemplate, ChatPromptTemplate
from langchain_core.messages import SystemMessage

chat_prompt_template = ChatPromptTemplate.from_messages(
    messages=[
        SystemMessage(content='Describe the following image very briefly.'),
        HumanMessagePromptTemplate.from_template(
            # This line is an ugly hack. It will trigger the ImagePromptTemplate internally (see [Line 455](https://github.com/langchain-ai/langchain/blob/d8a1f1114d839b2a902ff9bd4ae668751257414a/libs/core/langchain_core/prompts/chat.py#L455C1-L455C64))
            [{'image_url': ""}]
        )
    ]
)

url = 'https://fastly.picsum.photos/id/237/200/300.jpg?hmac=TmmQSbShHz9CdQm0NkEjx1Dyh_Y984R9LpNrpvH2D_U'

detail = 'low'

print(chat_prompt_template)
# Output:
# ChatPromptTemplate(input_variables=[], messages=[SystemMessage(content='Describe the following image very briefly.'), HumanMessagePromptTemplate(prompt=[ImagePromptTemplate(input_variables=[], template={'url': ''})])])

result = chat_prompt_template.invoke(input={'url': url, 'detail': detail})
print(result)
# Output:
# ChatPromptValue(messages=[SystemMessage(content='Describe the following image very briefly.'), HumanMessage(content=[{'type': 'image_url', 'image_url': {'url': 'https://fastly.picsum.photos/id/237/200/300.jpg?hmac=TmmQSbShHz9CdQm0NkEjx1Dyh_Y984R9LpNrpvH2D_U', 'detail': 'low'}}])])

This also works with a local path instead of a URL.

Chain Example with an LLM:

from langchain_core.prompts import HumanMessagePromptTemplate, ChatPromptTemplate
from langchain_core.messages import SystemMessage
from langchain_openai import ChatOpenAI
from langchain_core.output_parsers import StrOutputParser

chat_prompt_template = ChatPromptTemplate.from_messages(
    messages=[
        SystemMessage(content='Describe the following image very briefly.'),
        HumanMessagePromptTemplate.from_template(
            [{'image_url': ""}]
        )
    ]
)

url = 'https://fastly.picsum.photos/id/237/200/300.jpg?hmac=TmmQSbShHz9CdQm0NkEjx1Dyh_Y984R9LpNrpvH2D_U'
detail = 'low'

llm = ChatOpenAI(model_name="gpt-4o")
output_parser = StrOutputParser()

chain = chat_prompt_template | llm | output_parser

result = chain.invoke(input={"url": url, "detail": detail})

print(result)
# Output:
# The image shows a black Labrador Retriever puppy lying on a wooden surface.

[michalzan]

thanks for this. Did you create a pull request, so we can track?

[thdesc]

No I haven't. I am not sure if a pull request could be created from this since it involves an ugly hack to trigger the ImagePromptTemplate ([{'image_url': ""}]). I just wanted to share some thoughts I had in order to use the ImagePromptTemplate...

[joegoldin]

Opened a PR here: #22519

[thdesc]

Opened the second one : #22559

[thdesc]

Closed this PR : #22559 and opened this one: #22613

[yulisunny]

human_template = []
    for i, base64_image in enumerate(base64_images):
        human_template.append({"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{base64_image}"}})

I was able to get ImagePromptTemplate to work straight up with the latest release using the above.

[thdesc]

Yes indeed, this works but I think this doesn't take advantages of the ImagePromptTemplate. I think the idea behind is to be able to give either an URL hosting an image or to give a base64 encoded image as you did.

The ImagePromptTemplate does that. You can pass an URL or a local path to an image. See below:

from langchain_core.prompts.image import ImagePromptTemplate

image_prompt_template = ImagePromptTemplate()

output = image_prompt_template.invoke(input={"url": "https://example.com/image.jpg", "detail": "auto"})
print(output)
# Output:
# ImagePromptValue(image_url={'detail': 'auto', 'url': 'https://example.com/image.jpg'})

path = "path/to/image.png"
output = image_prompt_template.invoke(input={"path": path, "detail": "auto"})
print(output.image_url['url'][:100])
# Output:
# data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAABQAAABThCAYAAAAEVaCqAAAAAXNSR0IArs4c6QAAIABJREFUeJzs3X

[thdesc]

Hi everyone !

This PR has been merged #22613.

Now you can directly pass a local path to an image without needing to extract, encode, and format the content of the image file yourself!

Here how it works:

from langchain_core.prompts import HumanMessagePromptTemplate, ChatPromptTemplate
from langchain_core.messages import HumanMessage

image_path = 'path/to/image.png'
detail_parameter = 'high'

chat_prompt_template = ChatPromptTemplate.from_messages(
    messages=[
        HumanMessage(content='Describe the following image.'),
        HumanMessagePromptTemplate.from_template(
            [{'image_url': {'path': '{image_path}', 'detail': '{detail_parameter}'}}]
        )
    ]
)

prompt = chat_prompt_template.format(image_path=image_path, detail_parameter=detail_parameter)

print(prompt[:150])
# Output:
# Human: Describe the following image.
# Human: [{'type': 'image_url', 'image_url': {'url': 'data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAABQAAABMXCAYAAAA