Structured Output (response_format) - LiteLLM vs. CrewAI

[mouramax]

Hello everyone,

I've encountered an issue where the call method of a CrewAI LLM, despite being a wrapper around litellm.completion, doesn't seem to correctly handle the response_format parameter. The example code below demonstrates the problem:

from litellm import get_supported_openai_params, completion
from pydantic import BaseModel
from crewai import LLM
import os

PROVIDER = 'openrouter'
MODEL = 'google/gemini-2.0-flash-lite-preview-02-05:free'   # I'm poor, gimme free version

# First, let's confirm that 'response_format' is supported by the chosen provider and model.

supported_params = get_supported_openai_params(
    model=MODEL,
    custom_llm_provider=PROVIDER
)

if 'response_format' in supported_params:
    print('Yeah, we got response_format!')
else:
    print('No response_format, baby!')

# Set up our tests

os.environ['OPENROUTER_API_KEY'] = 'YOUR_KEY_NOT_MINE'

class CalendarEvent(BaseModel):
    name: str
    date: str
    participants: list[str]

# Test if litellm.completion can handle response_format correctly.

messages = [
    {'role': 'system', 'content': 'Extract the event information.'},
    {'role': 'user', 'content': 'Alice and Bob are going to a science fair on Friday.'},
]

litellm_response = completion(
    model=f'{PROVIDER}/{MODEL}',
    messages=messages,
    response_format=CalendarEvent,
)

print(f'\nLiteLLM Response:\n\n{litellm_response}')

# Test if crewai.LLM.call can handle response_format correctly.

gemini_llm = LLM(
    model=f'{PROVIDER}/{MODEL}',
    response_format=CalendarEvent,
)

crewai_response = gemini_llm.call(
    "Extract the event information:\n\n"
    "Alice and Bob are going to a science fair on Friday."
)

print(f'\nCrewAI Response:\n\n{crewai_response}')

When running the code above, I get the following error:

ValueError: The model openrouter/google/gemini-2.0-flash-lite-preview-02-05:free does not support response_format for provider 'openrouter'. Please remove response_format or use a supported model.

[mouramax]

Update

Digging deeper, I identified that the problem is that the crewai.LLM._validate_call_params method depends on the validation done in litellm.utils.supports_response_schema which, in turn, depends on the model_prices_and_context_window.json file being updated, which is unlikely to happen given the increasing number of models and providers. This is an architectural problem with the LiteLLM library.

In this case, I propose that crewai.LLM checks the response_format directly in the result of litellm.utils.get_supported_openai_params, as in the following modified/refactored version:

def _validate_call_params(self) -> None:
    """
    Validate parameters before making a call. Currently this only checks if
    a response_format is provided and whether the model supports it.
    The custom_llm_provider is dynamically determined from the model:
      - E.g., "openrouter/deepseek/deepseek-chat" yields "openrouter"
      - "gemini/gemini-1.5-pro" yields "gemini"
      - If no slash is present, "openai" is assumed.
    """
    provider = self._get_custom_llm_provider()
    if (
        self.response_format is not None
        and not self.supports_response_format()
    ):
        raise ValueError(
            f"The model {self.model} does not support response_format for provider '{provider}'. "
            "Please remove response_format or use a supported model."
        )

def supports_response_format(self) -> bool:
    try:
        supported_params = get_supported_openai_params(model=self.model)
        return (
            supported_params is not None
            and "response_format" in supported_params
        )
    except Exception as e:
        logging.error(f"Failed to get supported params: {str(e)}")
        return False

[golimix]

ValueError: The model deepseek/deepseek-chat does not support response_format for provider 'deepseek'. Please remove response_format or use a supported model.

the same problem

[Baukebrenninkmeijer]

@mouramax did you work on a PR to integrate this by any chance? Is there a related issue?

[mouramax]

@Baukebrenninkmeijer,

I shared my take on this over in this other issue.

My suggestion is to just use the LLM class the way CrewAI expects it to be used, and structured output works just fine.

[Baukebrenninkmeijer]

@mouramax yeah I found that shortly after. Agree that this is still strange behaviour, since the issue is also present with other providers like Gemini, Nvidia, ollama, etc. Anyways, thanks for diving into this!