Does it make sense to do few-shot prompting with image-to-text?

[LuchiLucs]

Checked other resources

• I added a very descriptive title to this question.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.

Commit to Help

• I commit to help with one of those options 👆

Example Code

system = f"""
I wish to describe some images using text, ... custom detailed prompt

Below some examples of interactions:
"""

example_1_input = HumanMessage(
    content=[
        {
            "type": "image_url",
            "image_url": {
                "url": "https://custom_url_..."
            },
        }
    ]
)

example_1_output = SystemMessage(
    MyPyndaticModel(...).model_dump_json()
)

...

prompt = ChatPromptTemplate.from_messages(
    [
        ("system", system),
        example_1_input,
        example_1_output,
        example_2_input,
        example_2_output,
        ("human", "{image_input}"),
    ]
)


        llm = AzureChatOpenAI(...)

        structured_llm = llm.with_structured_output(MyPyndaticModel)
        few_shot_structured_llm = prompt | structured_llm

        # input setup
        base64_image = base64.b64encode(image_content).decode("utf-8")
        image_content = [
            {
                "type": "image_url",
                "image_url": {"url": f"data:image/png;base64,{base64_image}"},
            }
        ]

        # invoke
        img_description: MyPyndaticModel= few_shot_structured_llm.invoke({"image_input": image_content})

        # output
        text = img_description.my_text_output_field

Description

• I have a custom prompt with structured output to describe an image input to text output. I was wondering if it is reasonable with respect to performance to include few shot prompting to add edge case and normal examples.

• In case it is reasonable, how should I implement it? I was trying to create a system message prompt including the few shot examples but I don't know how to include the image input. The output are displayed in the system prompt as the json dump of the pyndatic model. I searched for this online and I found few examples here and there but never a complete example.

• The provided example is not working because it seems returning the pyndatic model of the first few-shot examples. It was a try, trying to mix HumanMessage and SystemMessage to implement a sort of few-shot prompting "interactions".

System Info

python 3.12.6
langchain-openai>=0.3.8