ChatOllama with Images

[justas145]

Checked other resources

• I added a very descriptive title to this question.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.

Commit to Help

• I commit to help with one of those options 👆

Example Code

llm = ChatOllama(base_url="URL", model=model)

Description

I am hosting llm on a server and it is a Llava model that is able to see the images. I want to send an image or a base64 code to it. How can I do it?

System Info

langchain==0.1.0
langchain-anthropic==0.1.4
langchain-community==0.0.20
langchain-core==0.1.50
langchain-experimental==0.0.57
langchain-google-genai==1.0.1
langchain-groq==0.1.3
langchain-openai==0.0.2.post1
langchain-text-splitters==0.0.1
langchainhub==0.1.15

[dosubot[bot]]

To send an image or a base64 encoded image to the Llava model using the ChatOllama class, you can follow the example code below. This demonstrates how to convert an image to a base64 string and send it along with a text prompt to the model.

First, ensure you have the necessary imports and helper functions to handle the image conversion:

import base64
from io import BytesIO
from PIL import Image

def convert_to_base64(pil_image):
    """
    Convert PIL images to Base64 encoded strings

    :param pil_image: PIL image
    :return: Base64 string
    """
    buffered = BytesIO()
    pil_image.save(buffered, format="JPEG")  # You can change the format if needed
    img_str = base64.b64encode(buffered.getvalue()).decode("utf-8")
    return img_str

Next, use the ChatOllama class to send the image and text prompt:

from langchain_community.chat_models import ChatOllama
from langchain_core.messages import HumanMessage
from langchain_core.output_parsers import StrOutputParser

# Initialize the ChatOllama model
llm = ChatOllama(base_url="URL", model="model")

# Function to create the prompt with text and image
def prompt_func(data):
    text = data["text"]
    image = data["image"]

    image_part = {
        "type": "image_url",
        "image_url": f"data:image/jpeg;base64,{image}",
    }

    content_parts = []

    text_part = {"type": "text", "text": text}

    content_parts.append(image_part)
    content_parts.append(text_part)

    return [HumanMessage(content=content_parts)]

# Create the chain with the prompt function, model, and output parser
chain = prompt_func | llm | StrOutputParser()

# Load and convert the image to base64
file_path = "path_to_your_image.jpg"
pil_image = Image.open(file_path)
image_b64 = convert_to_base64(pil_image)

# Invoke the chain with the text and image data
query_chain = chain.invoke(
    {"text": "What is the Dollar-based gross retention rate?", "image": image_b64}
)

print(query_chain)

This code will send the image and text prompt to the Llava model hosted on your server and print the response ^[1].

To continue talking to Dosu, mention @dosu.

[MichaelCurrin]

I tried this out and it solved my use case. :)

---

The official in the docs uses a different approach with llm.bind(images=[image_b64])

https://python.langchain.com/v0.2/docs/integrations/llms/ollama/#multi-modal

But this does not work. It hallucinates consistently about what is in the image, even if I use the same image as on that example. I tried both with OllamaLLM as in the example there and ChatOllama (which worked above).

I suggest the docs be updated to use the answer provided on this thread which is longer but actually works.