[Bug]: Vercel AI SDK OpenAI provider fails with claude-3-7-sonnet when thinking mode is enabled ("Expected 'id' to be a string")

[jerry-intrii]

What happened?

Description

When using LiteLLM v1.61.16 with claude-3-7-sonnet and enabling the thinking mode, the Vercel AI SDK's OpenAI Provider fails to process the response due to missing tool_calls fields, resulting in an "Expected 'id' to be a string" error.

Environment

• LiteLLM version: 1.61.16
• Claude model: claude-3-7-sonnet
• Integration: Self-hosted LiteLLM with Vercel AI SDK OpenAI Provider

Steps to Reproduce

1. Set up LiteLLM with claude-3-7-sonnet
2. Enable thinking mode in litellm_params:

{
  "thinking": {
    "type": "enabled",
    "budget_tokens": 32000
  }
}

3. Make a request through Vercel AI SDK's OpenAI Provider to the LiteLLM endpoint

Current Behavior

The request fails in browser's console with error: "Expected 'id' to be a string"

Expected Behavior

The request should process successfully, handling the thinking mode response format appropriately.

Additional Context

1. Response format without thinking mode:

ChatCompletionMessage(
    content="...",
    refusal=None,
    role='assistant',
    audio=None,
    function_call=None,
    tool_calls=None
)

2. Response format with thinking mode enabled:

ChatCompletionMessage(
    content="...",
    reasoning_content="..."
)

Possible Solution

The LiteLLM implementation may need to ensure that responses maintain compatibility with OpenAI's response format when thinking mode is enabled, particularly regarding the tool_calls field structure expected by Vercel AI SDK.

Relevant log output

(node:internal/webstreams/transformstream:573:10)\\n    at node:internal/webstreams/transformstream:378:16\\n    at writableStreamDefaultControllerProcessWrite (node:internal/webstreams/writablestream:1129:5)\\n    at writableStreamDefaultControllerAdvanceQueueIfNeeded (node:internal/webstreams/writablestream:1244:5)\\n    at writableStreamDefaultControllerWrite (node:internal/webstreams/writablestream:1118:3)\\n    at writableStreamDefaultWriterWrite (node:internal/webstreams/writablestream:1008:3)\\n    at [kChunk] (node:internal/webstreams/readablestream:1586:31)\\n    at readableStreamFulfillReadRequest (node:internal/webstreams/readablestream:2119:24)\\n    at readableStreamDefaultControllerEnqueue (node:internal/webstreams/readablestream:2311:5)\\n    at transformStreamDefaultControllerEnqueue (node:internal/webstreams/transformstream:508:5)\\n    at TransformStreamDefaultController.enqueue (node:internal/webstreams/transformstream:324:5)\\n    at Object.transform (node:internal/webstreams/encoding:142:22)\"\n}"

Are you a ML Ops Team?

No

What LiteLLM version are you on ?

v1.61.16

Twitter / LinkedIn details

No response

[krrishdholakia]

Hey @jerry-intrii trying to understand this better - what exactly is the param we're returning incorrectly?

I can see us returning tool_calls even on the thinking response

[Saran33]

It seems to be returning empty tool calls when reasoning is enabled, even when the request should not trigger any tool call from the model.

tool_calls = [{'index': -1, 'id': None, 'function': {'arguments': '{}', 'name': None}, 'type': 'function'}]

I'm using the Langchain ChatOpenAI client and the above is what is being set as additional_kwargs on the Langchain chunk. The additional_kwargs attr is where Langchain stores the raw provider-specific tool calls before standardising them to the Langchain format.

[krrishdholakia]

Oh - is response_format set @Saran33

if so, can you share a script to repro?

[Saran33]

@krrishdholakia no, I'm not setting response_format. I just tried it without including any tools in the request and attempting a normal reasoning response, and that also includes empty tool call chunks in the response. The repo is an internal one but it's using Langchain's ChatOpenAI along with LangGraph, and streaming responses to the client in the Vercel AI Datastream Protocol.

In my custom parser for Vercel protocol I can handle it by ignoring chunks with index -1, but the empty tool calls are causing validation errors in LangGraph tool nodes:

  langgraph/prebuilt/tool_node.py", line 357, in _arun_one
    if invalid_tool_message := self._validate_tool_call(call):
...
pydantic_core._pydantic_core.ValidationError: 1 validation error for ToolMessage
tool_call_id
  Input should be a valid string [type=string_type, input_value=None, input_type=NoneType]

[Saran33]

  - model_name: claude-3-7-sonnet-reasoning
    litellm_params:
      model: claude-3-7-sonnet-20250219
      api_key: "os.environ/ANTHROPIC_API_KEY"
      thinking: {"type": "enabled", "budget_tokens": 1024}
      # additional_drop_params: ["temperature"]
      drop_params: True

The chunks look ok:

2025-03-06 04:15:14 04:15:14 - LiteLLM Proxy:DEBUG: proxy_server.py:3019 - async_data_generator: received streaming chunk - ModelResponseStream(id='chatcmpl-b66b85ac-ff00-4f90-92ad-92619ce4eab9', created=1741234514, model='claude-3-7-sonnet-20250219', object='chat.completion.chunk', system_fingerprint=None, choices=[StreamingChoices(finish_reason=None, index=0, delta=Delta(reasoning_content=' today.', thinking_blocks=[{'type': 'thinking', 'thinking': ' today.', 'signature_delta': None}], provider_specific_fields={'thinking_blocks': [{'type': 'thinking', 'thinking': ' today.', 'signature_delta': None}]}, content='', role=None, function_call=None, tool_calls=None, audio=None), logprobs=None)], provider_specific_fields=None, stream_options=None, citations=None)

Then the complete response contains:

2025-03-06 04:15:15 04:15:15 - LiteLLM Proxy:DEBUG: proxy_track_cost_callback.py:32 - kwargs stream: True + complete streaming response: ModelResponse(id='chatcmpl-b66b85ac-ff00-4f90-92ad-92619ce4eab9', created=1741234513, model='claude-3-7-sonnet-20250219', object='chat.completion', system_fingerprint=None, choices=[Choices(finish_reason='stop', index=0, message=Message(content="Hello! It's nice to meet you. I'm Claude, an AI assistant made by Anthropic. How can I help you today? I'm here to answer questions, assist with tasks, provide information, or just chat. What's on your mind?", role='assistant', tool_calls=[ChatCompletionMessageToolCall(function=Function(arguments='{}', name=None), id='c1d25261-9aea-495d-8f1b-0dc8f74e360a', type='function')], function_call=None, provider_specific_fields=None))], usage=Usage(completion_tokens=109, prompt_tokens=265, total_tokens=374, completion_tokens_details=None, prompt_tokens_details=None, cache_creation_input_tokens=0, cache_read_input_tokens=0))

[Saran33]

Could this be the issue here if tool calls is being passed as an empty list instead of None?

https://github.com/BerriAI/litellm/blob/main/litellm/litellm_core_utils/prompt_templates/factory.py#L1522-L1535

If instead we do something like this could it fix it?

assistant_tool_calls = assistant_content_block.get("tool_calls")
if assistant_tool_calls: # Only processes non-empty tool calls
    assistant_content.extend(convert_to_anthropic_tool_invoke(assistant_tool_calls))

assistant_function_call = assistant_content_block.get("function_call")
if assistant_function_call:  # Only processes non-empty function calls
    assistant_content.extend(convert_function_to_anthropic_tool_invoke(assistant_function_call))

(running docker image so haven't tried it locally)

[krrishdholakia]

hey @Saran33 are you able to share the debug log with the raw request that triggers this?

Specifically this log -

litellm/litellm/proxy/proxy_server.py

Line 3521 in f6535ae

"Request received by LiteLLM:\n{}".format(json.dumps(data, indent=4)),

[Saran33]

Testing it with minimal chat prompt without tools or structured output etc.

{
    "messages": [
        {
            "content": "You are a helpful AI Assistant based on the \"anthropic/claude-3-7-sonnet-reasoning\" architecture that will provide responses based on the current conversation history,\n    as well as your extensive knowledge base unless explicitly restricted by the conversation's context.\n    The user will provide previous messages as context. Use this information to inform your responses,\n    but you can also draw upon your general knowledge to answer questions or complete tasks not directly addressed in the conversation history.\n    If a request is made for which you have no specific information in the conversation history and it falls outside your general knowledge or capabilities,\n    say \"Sorry, I don't have any information on ...\" and provide a general advice if possible.\n    When asked for tasks or information that require creation or explanation (like writing code, explaining concepts, etc.),\n    use your best judgment to provide a helpful response based on your training and knowledge.\n    System time: 2025-03-06T04:51:34.295528+00:00\n    ",
            "role": "system"
        },
        {
            "content": "Hi",
            "role": "user"
        }
    ],
    "model": "claude-3-7-sonnet-reasoning",
    "stream": true,
    "user": "<USER_ID>"
}

[krrishdholakia]

able to repro this

[krrishdholakia]

This issue was caused by not handling the new anthropic thinking_delta content block -

litellm/litellm/llms/anthropic/chat/handler.py

Line 623 in f6535ae

if is_empty: