[Bug]: Knowledge Base Call returning error

[apanagos]

What happened?

We are running LiteLLM in AWS.
Using other functionality successfully.
Set up Bedrock Knowledge Base and added it.
I am able to run it through the "Test Key" with the model and the Vector Store without issue.

When I run it from an application I get an error.

Using Curl:

curl https://myurl/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer sk-mykey" \
  -d '{
    "model": "us.anthropic.claude-3-5-sonnet-20241022-v2:0",
    "messages": [{"role": "user", "content": "Who is on the Platform Engineering team?"}],
    "tools": [
        {
            "type": "file_search",
            "vector_store_ids": ["QZTOGKY5HL"]
        }
    ]
  }'

ERROR:

{
    "error": {
        "code": "400",
        "message": "litellm.BadRequestError: BedrockException - {\"message\":\"3 validation errors detected: Value '' at 'toolConfig.tools.1.member.toolSpec.name' failed to satisfy constraint: Member must have length greater than or equal to 1; Value '' at 'toolConfig.tools.1.member.toolSpec.name' failed to satisfy constraint: Member must satisfy regular expression pattern: [a-zA-Z0-9_-]+; Value '' at 'toolConfig.tools.1.member.toolSpec.description' failed to satisfy constraint: Member must have length greater than or equal to 1\"}. Received Model Group=us.anthropic.claude-3-5-sonnet-20241022-v2:0Available Model Group Fallbacks=None",
        "param": null,
        "type": null
    }
}

I get the same error when using OpenAI lib

# Get env variables
OPENAI_API_KEY = os.getenv("OPENAI_API_KEY")
OPENAI_API_BASE = os.getenv("OPENAI_API_BASE")
MODEL = os.getenv("MODEL")
KB_ID="QZTOGKY5HL"

# set up OpenAI client
client = openai.OpenAI(
    api_key=OPENAI_API_KEY,
    base_url=OPENAI_API_BASE,
)

# send a chat message to the model
messages = [
    {
        "role": "user",
        "content": "Who is on the Platform Engineering Team?"
    }
]

tools = [
    {
        "type": "file_search",
        "vector_store_ids": [KB_ID]
    }
]

response = client.chat.completions.create(
    model=MODEL,
    messages = messages,
    tools=tools,
)

If I remove the tools the call runs successfully (of course without KB) so I know settings are correct.

If I do a curl to get a list of Vector Stores I see the one I am using.

Relevant log output

{
  "status": "failure",
  "batch_models": null,
  "usage_object": null,
  "user_api_key": "3f1c7c212629bacf3fc183b9dc1dd5b9f569d2105dc2ded4f2bb15ef06cc1dbd",
  "error_information": {
    "traceback": "  File \"/usr/lib/python3.13/site-packages/litellm/proxy/proxy_server.py\", line 3537, in chat_completion\n    return await base_llm_response_processor.base_process_llm_request(\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n    ...<16 lines>...\n    )\n    ^\n  File \"/usr/lib/python3.13/site-packages/litellm/proxy/common_request_processing.py\", line 391, in base_process_llm_request\n    responses = await llm_responses\n                ^^^^^^^^^^^^^^^^^^^\n  File \"/usr/lib/python3.13/site-packages/litellm/router.py\", line 990, in acompletion\n    raise e\n  File \"/usr/lib/python3.13/site-packages/litellm/router.py\", line 966, in acompletion\n    response = await self.async_function_with_fallbacks(**kwargs)\n               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/lib/python3.13/site-packages/litellm/router.py\", line 3536, in async_function_with_fallbacks\n    raise original_exception\n  File \"/usr/lib/python3.13/site-packages/litellm/router.py\", line 3350, in async_function_with_fallbacks\n    response = await self.async_function_with_retries(*args, **kwargs)\n               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/lib/python3.13/site-packages/litellm/router.py\", line 3728, in async_function_with_retries\n    raise original_exception\n  File \"/usr/lib/python3.13/site-packages/litellm/router.py\", line 3619, in async_function_with_retries\n    response = await self.make_call(original_function, *args, **kwargs)\n               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/lib/python3.13/site-packages/litellm/router.py\", line 3737, in make_call\n    response = await response\n               ^^^^^^^^^^^^^^\n  File \"/usr/lib/python3.13/site-packages/litellm/router.py\", line 1129, in _acompletion\n    raise e\n  File \"/usr/lib/python3.13/site-packages/litellm/router.py\", line 1088, in _acompletion\n    response = await _response\n               ^^^^^^^^^^^^^^^\n  File \"/usr/lib/python3.13/site-packages/litellm/utils.py\", line 1492, in wrapper_async\n    raise e\n  File \"/usr/lib/python3.13/site-packages/litellm/utils.py\", line 1353, in wrapper_async\n    result = await original_function(*args, **kwargs)\n             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/lib/python3.13/site-packages/litellm/main.py\", line 531, in acompletion\n    raise exception_type(\n          ~~~~~~~~~~~~~~^\n        model=model,\n        ^^^^^^^^^^^^\n    ...<3 lines>...\n        extra_kwargs=kwargs,\n        ^^^^^^^^^^^^^^^^^^^^\n    )\n    ^\n  File \"/usr/lib/python3.13/site-packages/litellm/litellm_core_utils/exception_mapping_utils.py\", line 2239, in exception_type\n    raise e\n  File \"/usr/lib/python3.13/site-packages/litellm/litellm_core_utils/exception_mapping_utils.py\", line 939, in exception_type\n    raise BadRequestError(\n    ...<4 lines>...\n    )\n",
    "error_code": "400",
    "error_class": "BadRequestError",
    "llm_provider": "bedrock",
    "error_message": "litellm.BadRequestError: BedrockException - {\"message\":\"3 validation errors detected: Value '' at 'toolConfig.tools.1.member.toolSpec.name' failed to satisfy constraint: Member must have length greater than or equal to 1; Value '' at 'toolConfig.tools.1.member.toolSpec.name' failed to satisfy constraint: Member must satisfy regular expression pattern: [a-zA-Z0-9_-]+; Value '' at 'toolConfig.tools.1.member.toolSpec.description' failed to satisfy constraint: Member must have length greater than or equal to 1\"}. Received Model Group=us.amazon.nova-lite-v1:0\nAvailable Model Group Fallbacks=None LiteLLM Retried: 1 times, LiteLLM Max Retries: 2"
  },
  "applied_guardrails": null,
  "user_api_key_alias": "ai-en",
  "user_api_key_org_id": null,
  "requester_ip_address": "",
  "user_api_key_team_id": "2fee7bea-c133-46df-9979-ede094b333aa",
  "user_api_key_user_id": "default_user_id",
  "guardrail_information": null,
  "model_map_information": null,
  "mcp_tool_call_metadata": null,
  "additional_usage_values": {},
  "user_api_key_team_alias": "Platform Engineering",
  "vector_store_request_metadata": null
}

Are you a ML Ops Team?

Yes

What LiteLLM version are you on ?

v1.71.1

Twitter / LinkedIn details

No response

[ishaan-jaff]

Fixed here @apanagos

#11467

can you test again and confirm it's fixed

[apanagos]

ok. I will test it out today and report back.

[germanilia]

I don't think the issue is fixed with this version, it fails to even with the regular completion with the lates docker version

[apanagos]

I tested locally but had issues. I just updated with 1.72.2 (ghcr.io/berriai/litellm:main-stable) but I don't think the fix is in there. Once I deploy this to prod for some of my users to use I will circle back to test v1.72.1.dev8 in dev.

[ishaan-jaff]

Please try this image: ghcr.io/berriai/litellm:main-v1.72.2.rc

[apanagos]

ghcr.io/berriai/litellm:main-stable is ghcr.io/berriai/litellm:main-v1.72.2.rc and that is what I have deployed. (swagger shows 1.72.2)

I assumed the dev8 fix is not in there. If it is then I am getting the same issue when testing.

[apanagos]

So I looked and saw that #11404 is supposed to be addressed in main-stable a.k.a. v1.72.2.rc which is the versions I have deployed.

This is the code I am using in my test:

import openai
import os
from dotenv import load_dotenv

# Load the environment variables from the .env file
load_dotenv()

# Get env variables
OPENAI_API_KEY = os.getenv("OPENAI_API_KEY")
OPENAI_API_BASE = os.getenv("OPENAI_API_BASE")
MODEL = os.getenv("MODEL")
KB_ID="QZTOGKY5HL"


# set up OpenAI client
client = openai.OpenAI(
    api_key=OPENAI_API_KEY,
    base_url=OPENAI_API_BASE,
)

# send a chat message to the model
messages = [
    {
        "role": "user",
        "content": "Who is Justin Ranta?"
    }
]

tools = [
    {
        "type": "file_search",
        "vector_store_ids": [KB_ID]
    }
]

response = client.chat.completions.create(
    model=MODEL,
    messages = messages,
    tools=tools,
)

# print out the response
print(f"Response: {response}")

The models I tested with were:

• MODEL="us.amazon.nova-lite-v1:0"
• MODEL="us.anthropic.claude-3-5-sonnet-20241022-v2:0"

If I comment out tools i get a response BUT if I include it I get the following error:

{
  "status": "failure",
  "batch_models": null,
  "usage_object": null,
  "user_api_key": "3f1c7c212629bacf3fc183b9dc1dd5b9f569d2105dc2ded4f2bb15ef06cc1dbd",
  "error_information": {
    "traceback": "  File \"/usr/lib/python3.13/site-packages/litellm/proxy/proxy_server.py\", line 3619, in chat_completion\n    return await base_llm_response_processor.base_process_llm_request(\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n    ...<16 lines>...\n    )\n    ^\n  File \"/usr/lib/python3.13/site-packages/litellm/proxy/common_request_processing.py\", line 397, in base_process_llm_request\n    responses = await llm_responses\n                ^^^^^^^^^^^^^^^^^^^\n  File \"/usr/lib/python3.13/site-packages/litellm/router.py\", line 994, in acompletion\n    raise e\n  File \"/usr/lib/python3.13/site-packages/litellm/router.py\", line 970, in acompletion\n    response = await self.async_function_with_fallbacks(**kwargs)\n               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/lib/python3.13/site-packages/litellm/router.py\", line 3542, in async_function_with_fallbacks\n    raise original_exception\n  File \"/usr/lib/python3.13/site-packages/litellm/router.py\", line 3356, in async_function_with_fallbacks\n    response = await self.async_function_with_retries(*args, **kwargs)\n               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/lib/python3.13/site-packages/litellm/router.py\", line 3734, in async_function_with_retries\n    raise original_exception\n  File \"/usr/lib/python3.13/site-packages/litellm/router.py\", line 3625, in async_function_with_retries\n    response = await self.make_call(original_function, *args, **kwargs)\n               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/lib/python3.13/site-packages/litellm/router.py\", line 3743, in make_call\n    response = await response\n               ^^^^^^^^^^^^^^\n  File \"/usr/lib/python3.13/site-packages/litellm/router.py\", line 1133, in _acompletion\n    raise e\n  File \"/usr/lib/python3.13/site-packages/litellm/router.py\", line 1092, in _acompletion\n    response = await _response\n               ^^^^^^^^^^^^^^^\n  File \"/usr/lib/python3.13/site-packages/litellm/utils.py\", line 1494, in wrapper_async\n    raise e\n  File \"/usr/lib/python3.13/site-packages/litellm/utils.py\", line 1355, in wrapper_async\n    result = await original_function(*args, **kwargs)\n             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/lib/python3.13/site-packages/litellm/main.py\", line 543, in acompletion\n    raise exception_type(\n          ~~~~~~~~~~~~~~^\n        model=model,\n        ^^^^^^^^^^^^\n    ...<3 lines>...\n        extra_kwargs=kwargs,\n        ^^^^^^^^^^^^^^^^^^^^\n    )\n    ^\n  File \"/usr/lib/python3.13/site-packages/litellm/litellm_core_utils/exception_mapping_utils.py\", line 2270, in exception_type\n    raise e\n  File \"/usr/lib/python3.13/site-packages/litellm/litellm_core_utils/exception_mapping_utils.py\", line 970, in exception_type\n    raise BadRequestError(\n    ...<4 lines>...\n    )\n",
    "error_code": "400",
    "error_class": "BadRequestError",
    "llm_provider": "bedrock",
    "error_message": "litellm.BadRequestError: BedrockException - {\"message\":\"3 validation errors detected: Value '' at 'toolConfig.tools.1.member.toolSpec.name' failed to satisfy constraint: Member must satisfy regular expression pattern: [a-zA-Z0-9_-]+; Value '' at 'toolConfig.tools.1.member.toolSpec.name' failed to satisfy constraint: Member must have length greater than or equal to 1; Value '' at 'toolConfig.tools.1.member.toolSpec.description' failed to satisfy constraint: Member must have length greater than or equal to 1\"}. Received Model Group=us.amazon.nova-lite-v1:0\nAvailable Model Group Fallbacks=None LiteLLM Retried: 1 times, LiteLLM Max Retries: 2"
  },
  "applied_guardrails": null,
  "user_api_key_alias": "ai-en",
  "user_api_key_org_id": null,
  "requester_ip_address": "",
  "user_api_key_team_id": "2fee7bea-c133-46df-9979-ede094b333aa",
  "user_api_key_user_id": "default_user_id",
  "guardrail_information": null,
  "model_map_information": null,
  "mcp_tool_call_metadata": null,
  "additional_usage_values": {},
  "user_api_key_team_alias": "Platform Engineering",
  "vector_store_request_metadata": null
}

I confirmed the correct VectorDB ID.
I wrote code that went directly to Bedrock KB and got the documents and then used LiteLLm to query it. So I can do the work manually.

I am using AWS Bedrock.

[apanagos]

@ishaan-jaff I updated with the latest version of LiteLLM 1.72.6.post1
and I am still getting the same issue. Please reopen this ticket.

{
  "status": "failure",
  "batch_models": null,
  "usage_object": null,
  "user_api_key": "3f1c7c212629bacf3fc183b9dc1dd5b9f569d2105dc2ded4f2bb15ef06cc1dbd",
  "error_information": {
    "traceback": "  File \"/usr/lib/python3.13/site-packages/litellm/proxy/proxy_server.py\", line 3637, in chat_completion\n    return await base_llm_response_processor.base_process_llm_request(\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n    ...<16 lines>...\n    )\n    ^\n  File \"/usr/lib/python3.13/site-packages/litellm/proxy/common_request_processing.py\", line 399, in base_process_llm_request\n    responses = await llm_responses\n                ^^^^^^^^^^^^^^^^^^^\n  File \"/usr/lib/python3.13/site-packages/litellm/router.py\", line 997, in acompletion\n    raise e\n  File \"/usr/lib/python3.13/site-packages/litellm/router.py\", line 973, in acompletion\n    response = await self.async_function_with_fallbacks(**kwargs)\n               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/lib/python3.13/site-packages/litellm/router.py\", line 3550, in async_function_with_fallbacks\n    raise original_exception\n  File \"/usr/lib/python3.13/site-packages/litellm/router.py\", line 3364, in async_function_with_fallbacks\n    response = await self.async_function_with_retries(*args, **kwargs)\n               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/lib/python3.13/site-packages/litellm/router.py\", line 3742, in async_function_with_retries\n    raise original_exception\n  File \"/usr/lib/python3.13/site-packages/litellm/router.py\", line 3633, in async_function_with_retries\n    response = await self.make_call(original_function, *args, **kwargs)\n               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/lib/python3.13/site-packages/litellm/router.py\", line 3751, in make_call\n    response = await response\n               ^^^^^^^^^^^^^^\n  File \"/usr/lib/python3.13/site-packages/litellm/router.py\", line 1136, in _acompletion\n    raise e\n  File \"/usr/lib/python3.13/site-packages/litellm/router.py\", line 1095, in _acompletion\n    response = await _response\n               ^^^^^^^^^^^^^^^\n  File \"/usr/lib/python3.13/site-packages/litellm/utils.py\", line 1494, in wrapper_async\n    raise e\n  File \"/usr/lib/python3.13/site-packages/litellm/utils.py\", line 1355, in wrapper_async\n    result = await original_function(*args, **kwargs)\n             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/lib/python3.13/site-packages/litellm/main.py\", line 544, in acompletion\n    raise exception_type(\n          ~~~~~~~~~~~~~~^\n        model=model,\n        ^^^^^^^^^^^^\n    ...<3 lines>...\n        extra_kwargs=kwargs,\n        ^^^^^^^^^^^^^^^^^^^^\n    )\n    ^\n  File \"/usr/lib/python3.13/site-packages/litellm/litellm_core_utils/exception_mapping_utils.py\", line 2271, in exception_type\n    raise e\n  File \"/usr/lib/python3.13/site-packages/litellm/litellm_core_utils/exception_mapping_utils.py\", line 971, in exception_type\n    raise BadRequestError(\n    ...<4 lines>...\n    )\n",
    "error_code": "400",
    "error_class": "BadRequestError",
    "llm_provider": "bedrock",
    "error_message": "litellm.BadRequestError: BedrockException - {\"message\":\"3 validation errors detected: Value '' at 'toolConfig.tools.1.member.toolSpec.name' failed to satisfy constraint: Member must have length greater than or equal to 1; Value '' at 'toolConfig.tools.1.member.toolSpec.name' failed to satisfy constraint: Member must satisfy regular expression pattern: [a-zA-Z0-9_-]+; Value '' at 'toolConfig.tools.1.member.toolSpec.description' failed to satisfy constraint: Member must have length greater than or equal to 1\"}. Received Model Group=us.amazon.nova-lite-v1:0\nAvailable Model Group Fallbacks=None LiteLLM Retried: 1 times, LiteLLM Max Retries: 2"
  },
  "applied_guardrails": null,
  "user_api_key_alias": "ai-en",
  "user_api_key_org_id": null,
  "requester_ip_address": "",
  "user_api_key_team_id": "2fee7bea-c133-46df-9979-ede094b333aa",
  "user_api_key_user_id": "default_user_id",
  "guardrail_information": null,
  "model_map_information": null,
  "mcp_tool_call_metadata": null,
  "additional_usage_values": {},
  "user_api_key_team_alias": "Platform Engineering",
  "vector_store_request_metadata": null
}

[ishaan-jaff]

re-opened