[BUG] Structured outputs fail with lite agents

[investindata]

Description

Structured outputs often fail when using lite agents inside of a flow. It has been impossible for me to get reliable structured outputs, despite trying several models. My understanding is that the output validation in a lite agent is, well, lite, compared to full-fledged agent from a crew.

The reason I'm using lite agents is because I'm using Flows, which are supposed to be more reliable. But this behavior makes my flow less reliable.

Below is an example based on what I'm working on. I'm using gemini-2.0-flash, but I ran into the same issue with other models.

from crewai import Agent, LLM
from pydantic import BaseModel, Field
from typing import Optional, List, Annotated
import asyncio
import os

os.environ["OPENAI_API_KEY"] = "<your-key>"

async def get_founders_names():
    class FounderNames(BaseModel):
        names: Optional[Annotated[
            List[str],
            Field(description="List of the full names of the company founders")
        ]] = None
    
    llm = LLM(
        model="gpt-4o",
        temperature=0.5
    )
    
    researcher_agent = Agent(
        role="Data Extraction Research Specialist",
        goal=(
            "Extract relevant information from sources and structure it into "
            "clean, organized data formats"
        ),
        backstory=(
            "You are a detail-oriented researcher who specializes in finding "
            "key information and organizing it into clear, structured outputs."
        ),
        llm=llm,
        allow_delegation=False,
        verbose=True
    )

    query = (
        "Extract the names of the founders of Dunder Mifflin company from the "
        "text below:\n\n"
        "Willy Wonka founded the Wonka Chocolate Factory. Dunder Mifflin (The "
        "Office) was founded in 1949 by Robert Dunder and Robert Mifflin. Elon "
        "Musk founded SpaceX in 2002."
    )

    result = await researcher_agent.kickoff_async(
        query, response_format=FounderNames
    )

    print("\nresult:\n")
    print(f"{result.pydantic=}")
    print(f"{type(result.pydantic)=}\n")

async def main():
    await get_founders_names()

if __name__ == "__main__":
    asyncio.run(main())

Steps to Reproduce

1. use the code above with gemini-2.0-flash
2. try variations and see for yourself that it often fails

Expected behavior

Valid pydantic structured outputs

Screenshots/Code snippets

Check this thread for additional information, with examples of outputs: https://community.crewai.com/t/having-a-very-hard-time-getting-structured-outputs-to-work/7016

An example would be:
'‘‘‘ json {“names”: [“<founder 1>”, “<founder 2>”] }’’’

Rather than something clean like:
{“names”: [“<founder 1>”, "<founder 2>”] }

Operating System

Other (specify in additional context)

Python Version

3.12

crewAI Version

0.159.0

crewAI Tools Version

0.62.0

Virtual Environment

Venv

Evidence

Already shared above.

Possible Solution

Apply the same output validation for full-fledged agents to like agent.

Additional context

See this forum thread: https://community.crewai.com/t/having-a-very-hard-time-getting-structured-outputs-to-work/7016/2

[Vidit-Ostwal]

Hi @investindata, I see you are not using response_format as a parameter.
Can you try with that once?

[investindata]

I am using the response_format as a parameter in the code above. here's the snippet:

result = await researcher_agent.kickoff_async(
query, response_format=FounderNames
)

You can also see the full project I'm working on here: https://github.com/investindata/diligence-agent/blob/feat/flows/src/diligence_agent/flow.py

I defined three flows and, as you can see, I created a function to parse the outputs since I'm not getting a structured output.