[GROK 4] Multimodal input

[tradanghi1999]

At the mean time, it seems that it is not avaliable for grok 4 to input images at the same syntax as OpenAI.

Anyone faces the same issue, please help me on this.

from langchain_openai import ChatOpenAI
from langchain_xai import ChatXAI

from langchain_core.messages import (
    SystemMessage,
    HumanMessage,
    AIMessage,
    ToolMessage,
    AnyMessage,
)

# Same error
# llm = ChatOpenAI(
#     base_url="https://api.x.ai/v1",
#     model="grok-4-0709",
#     api_key=os.getenv("XAI_API_KEY"),
# )

llm = ChatXAI(
    api_key=os.getenv("XAI_API_KEY"),
    model="grok-2-vision-1212",
)

messages = [
    HumanMessage(
        content=[
            "What is this image about?",
            {
                "type": "image_url",
                "image_url": {
                    "url": "https://res.cloudinary.com/dipsb8rj7/image/upload/v1747846127/6394272d-2fc1-4a45-b848-6a554ac1de32.png",
                    "detail": "high"
                },
            },
        ]
    )
]

response = llm.invoke(messages)

UnprocessableEntityError: Failed to deserialize the JSON body into the target type: messages[0]: data did not match any variant of untagged enum Content at line 1 column 235