[Bug]: parent ticket - gemini tool calling / response format errors

[krrishdholakia]

What happened?

Creating a parent ticket to track the errors for gemini tool calling / response format.

Case 1

{
    "tools": [
        {
            "function_declarations": [
                {
                    "name": "agents_agenttoolservice_get_table_column_names",
                    "description": "Get the column names of a table. This is helpful for building a WHERE clause in a PostgreSQL query.",
                    "parameters": {
                        "type": "object",
                        "properties": {
                            "table": {
                                "type": "string",
                                "description": "Name of the table"
                            }
                        },
                        "required": [
                            "table"
                        ]
                    }
                }
            ]
        }
    ]
}

Returns:

{
    "json": {
        "tool_calls": [
            {
                "id": "7466be7d-8604-4342-841d-cccb8457c372",
                "type": "function",
                "function": {
                    "name": "agents_agenttoolservice_get_table_column_names",
                    "arguments": {
                        "error": "Failed to parse arguments: unexpected token at end of stream '{\"table\": \"users\"}'",
                        "raw_args": "{\"table\": \"users\"}{\"table\": \"users\"}"
                    }
                }
            }
        ],
        "tool_call_id": "7466be7d-8604-4342-841d-cccb8457c372",
        "function_call": null
    }
}

Case 2

Request:

{
    "messages": [
        {
            "content": "\n    You are a helpful assistant who can help with questions on customers business or personal finances.\n    Use the results from the available tools to answer the question.\n    ",
            "role": "system"
        },
        {
            "content": "Hello",
            "role": "user"
        }
    ],
    "max_completion_tokens": 1000,
    "stream": false,
    "temperature": 0.0,
    "tools": [
        {
            "type": "function",
            "function": {
                "name": "test_agent",
                "description": "This tool helps find relevant help content",
                "parameters": {
                    "properties": {
                        "state": {
                            "properties": {
                                "messages": {
                                    "items": {},
                                    "type": "array"
                                },
                                "conversation_id": {
                                    "type": "string"
                                }
                            },
                            "required": [
                                "messages",
                                "conversation_id"
                            ],
                            "type": "object"
                        },
                        "config": {
                            "description": "Configuration for a Runnable.",
                            "properties": {
                                "tags": {
                                    "items": {
                                        "type": "string"
                                    },
                                    "type": "array"
                                },
                                "metadata": {
                                    "type": "object"
                                },
                                "callbacks": {
                                    "anyOf": [
                                        {
                                            "items": {},
                                            "type": "array"
                                        },
                                        {},
                                        {
                                            "type": "null"
                                        }
                                    ]
                                },
                                "run_name": {
                                    "type": "string"
                                },
                                "max_concurrency": {
                                    "anyOf": [
                                        {
                                            "type": "integer"
                                        },
                                        {
                                            "type": "null"
                                        }
                                    ]
                                },
                                "recursion_limit": {
                                    "type": "integer"
                                },
                                "configurable": {
                                    "type": "object"
                                },
                                "run_id": {
                                    "anyOf": [
                                        {
                                            "format": "uuid",
                                            "type": "string"
                                        },
                                        {
                                            "type": "null"
                                        }
                                    ]
                                }
                            },
                            "type": "object"
                        },
                        "kwargs": {
                            "default": null,
                            "type": "object"
                        }
                    },
                    "required": [
                        "state",
                        "config"
                    ],
                    "type": "object"
                }
            }
        }
    ]
}

Response:

{\n \"error\": {\n \"code\": 400,\n \"message\": \"Invalid value at 'tools[0].function_declarations[2].parameters.properties[1].value.any_of[0].properties[2].value.any_of[2].type'.

Relevant log output

Are you a ML Ops Team?

No

What LiteLLM version are you on ?

v1.67.0

Twitter / LinkedIn details

No response

[krrishdholakia]

For 2., this is the working tool input

            {
                "type": "function",
                "function": {
                    "name": "test_agent",
                    "description": "This tool helps find relevant help content",
                    "parameters": {
                        "properties": {
                            "state": {
                                "properties": {
                                    "messages": {
                                        "items": {
                                            "type": "object"
                                        },
                                        "type": "array"
                                    },
                                    "conversation_id": {
                                        "type": "string"
                                    }
                                },
                                "required": [
                                    "messages",
                                    "conversation_id"
                                ],
                                "type": "object"
                            },
                            "config": {
                                "description": "Configuration for a Runnable.",
                                "properties": {
                                    "tags": {
                                        "items": {
                                            "type": "string"
                                        },
                                        "type": "array"
                                    },
                                    "metadata": {
                                        "type": "object"
                                    },
                                    "callbacks": {
                                        "anyOf": [
                                            {"type": "array"},
                                            {"type": "object"},
                                            {"type": "null"}
                                        ],
                                    },
                                    "run_name": {
                                        "type": "string"
                                    },
                                    "max_concurrency": {
                                        "anyOf": [
                                            {
                                                "type": "integer"
                                            },
                                            {
                                                "type": "null"
                                            }
                                        ]
                                    },
                                    "recursion_limit": {
                                        "type": "integer"
                                    },
                                    "configurable": {
                                        "type": "object"
                                    },
                                    "run_id": {
                                        "anyOf": [
                                            {
                                                "format": "uuid",
                                                "type": "string"
                                            },
                                            {
                                                "type": "null"
                                            }
                                        ]
                                    }
                                },
                                "type": "object"
                            },
                            "kwargs": {
                                "default": None,
                                "type": "object"
                            }
                        },
                        "required": [
                            "state",
                            "config"
                        ],
                        "type": "object"
                    }
                }
            }
        ]

[tanvithakur94]

@krrishdholakia I tried using the same above payload I shared before and now Im getting this error, can you check on your side?
{ "message": "Error code: 400 - {'error_message': 'Chat completion failed', 'cause': 'BadRequestError: VertexAIException BadRequestError - {\\n \"error\": {\\n \"code\": 400,\\n \"message\": \"Unable to submit request because list_customersfunctionDeclarationparameters.config schema specified other fields alongside any_of. When using any_of, it must be the only field set. Learn more: https://cloud.google.com/vertex-ai/generative-ai/docs/multimodal/function-calling\",\\n \"status\": \"INVALID_ARGUMENT\"\\n }\\n}\\n'}" }

[krrishdholakia]

@tanvithakur94 "list_customers" isn't in the prior example.

Can you please share your complete input

[tanvithakur94]

@krrishdholakia Here is the payload. Can you take a look?

fail.txt