Sometimes validation error with with_structured_output(Class)

[leoniewgnr]

Im prompting my model with structured outputs:

class Act(BaseModel):
    action: Plan | FinalResponse

planner_prompt = ChatPromptTemplate.from_messages(
        [
            ("system", PLANNER_PROMPT),
            ("placeholder", "{messages}"),
        ]
    )

model = ChatOpenAI(model=OPENAI_MODEL)
structured_model = model.with_structured_output(Act)
chain = planner_prompt | structured_model
output = chain.invoke({"messages": [('user', 'do something')]})

This sometimes working and sometimes failing with the error:

Error in planner node: 2 validation errors for Act
action.Plan.steps
  Field required [type=missing, input_value={'query': '...'}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.9/v/missing
action.FinalResponse.response
  Field required [type=missing, input_value={'query': '...'}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.9/v/missing

It seems like the model is not responding with the correct JSON format, even though I thought this is guaranteed when using with_structured_output?

[dosubot[bot]]

Hey @leoniewgnr! I'm here to help you with any bugs, questions, or contributions you have. Let's tackle this issue together!

I found a similar solved discussion that might be relevant to your issue: Unexpected string validation error in Pydantic output parser. This discussion addresses validation errors related to the PydanticOutputParser expecting a specific format ^[1].

The with_structured_output(Act) method sometimes fails with validation errors in the Act class because the input provided to the model does not conform to the specified input_schema. The with_structured_output method ensures that the model's output adheres to the specified schema, and if the output does not match, validation errors will occur.

Here is an example of how to use the with_structured_output method with a Pydantic class:

from pydantic import BaseModel, Field
from typing import Union, Optional

class Plan(BaseModel):
    steps: str

class FinalResponse(BaseModel):
    response: str

class Act(BaseModel):
    action: Union[Plan, FinalResponse]

planner_prompt = ChatPromptTemplate.from_messages(
    [
        ("system", "You are a planner."),
        ("placeholder", "{messages}"),
    ]
)

model = ChatOpenAI(model="gpt-4")
structured_model = model.with_structured_output(Act)
chain = planner_prompt | structured_model
output = chain.invoke({"messages": [('user', 'do something')]})

In your case, the validation errors indicate that the model's response does not include the required fields steps for Plan and response for FinalResponse. This can happen if the model's output does not match the expected schema.

To mitigate this, you can:

1. Ensure that your prompt is clear and guides the model to produce the expected structured output.
2. Use the strict parameter in the with_structured_output method to enforce strict validation of the structured output, which can help catch issues early.

Example with strict parameter:

structured_model = model.with_structured_output(Act, strict=True)

Additionally, you can handle raw outputs and parse them manually if the model struggles with generating outputs that match complex schemas perfectly ^{[2][3][4][5][6]}.

To continue talking to Dosu, mention @dosu.

[leoniewgnr]

Great, this worked! I also had to

model_config = ConfigDict(extra="forbid")

to the Classes