'A conversation must start with a user message' with Bedrock Guardrails & Converse

[athewsey]

I'm getting the below error message when using Bedrock Guardrails + boto3 Converse through the gateway, even though the same request works fine directly against Bedrock.

Error message

botocore.exceptions.ClientError: An error occurred (400) when calling the Converse operation: {'error': 'Error from LiteLLM endpoint: {"error":{"message":"litellm.BadRequestError: BedrockException - {\\"message\\":\\"A conversation must start with a user message. Try again with a conversation that starts with a user message.\\"}\\n. Pass in default user message via `completion(..,user_continue_message=)` or enable `litellm.modify_params=True`.\\nFor Proxy: do via `litellm_settings::modify_params: True` or user_continue_message under `litellm_params`. Received Model Group=us.anthropic.claude-3-5-sonnet-20241022-v2:0\\nAvailable Model Group Fallbacks=None","type":null,"param":null,"code":"400"}}'}

Minimal reproducible example

The below code assumes you have the gateway deployed and an Amazon Bedrock Guardrail configured in your same active AWS region - and yields the error:

import boto3
from botocore import UNSIGNED
from botocore.config import Config

# (br client setup begins)
proxy_url = "https://YOUR-GATEWAY-URL/bedrock"
proxy_token = "sk-YOUR-GATEWAY-API-TOKEN"
br = boto3.client(
    "bedrock-runtime",
    config=Config(signature_version=UNSIGNED),
    endpoint_url=proxy_url,
)
def add_authorization_header(request, **_):
    request.headers["Authorization"] = f"Bearer {proxy_token}"
br.meta.events.register("request-created.*", add_authorization_header)
# (br client setup ends)

resp = br.converse(
    **{
        'modelId': 'us.anthropic.claude-3-5-sonnet-20241022-v2:0',
        'messages': [
            {
                'role': 'user',
                'content': [
                    {
                        'guardContent': {
                            'text': {
                                'text': "How's the weather?"
                            }
                        }
                    }
                ]
            }
        ],
        'system': [
            {
                'text': 'You are a cool cat'
            }
        ],
        'inferenceConfig': {'maxTokens': 4096, 'temperature': 1.0},
        'additionalModelRequestFields': {},
        'guardrailConfig': {
            'guardrailIdentifier': 'YOUR-BEDROCK-GUARDRAIL-ID',
            'guardrailVersion': 'DRAFT',
            'trace': 'enabled_full',
        }    
    }
)

print(resp)

...However, replacing the br client setup section with just br = boto3.client("bedrock-runtime") (not using the proxy/gateway) seems to work just fine.

Additional info

Tried upgrading to LiteLLM v1.69.0-stable and current main branch code of this solution, and still seeing same issue.

[athewsey]

It looks to me like this solution's Bedrock middleware is mangling messages on which the guardContent wrapper is set:

By temporarily turning on additional logging in the LiteLLM config.yaml:

general_settings:
  store_model_in_db: true
  store_prompts_in_spend_logs: true

...I was able to see the below "request" in LiteLLM admin console's "Logs" panel:

{
  "model": "us.anthropic.claude-3-5-sonnet-20241022-v2:0",
  "stream": true,
  "messages": [
    {
      "role": "system",
      "content": "You are a cool cat"
    },
    {
      "role": "user",
      "content": ""
    }
  ],
  "max_tokens": 4096,
  "temperature": 1
}

---

There's probably some interesting debate to be had on what the correct behaviour should be for Bedrock guardrails in this scenario:

• Per [Bug]: Bedrock guardrails analysing (and blocking) old+system messages BerriAI/litellm#9882, there's a question over what messages should be assumed in scope when a Bedrock Guardrail is enforced at the LiteLLM config (global or, in Enterprise Edition, API-key) level
• The underlying Converse API provides full flexibility not just for which messages in the history the enabled guardrail should apply to, but even qualifiers of how different text content elements should be treated by the guardrail (e.g. as a grounding source vs a user message).
• Since this middleware is mapping boto3 requests to OpenAI-style requests for LiteLLM, which then converts back to boto3 - it seems practically certain that the full functionality couldn't be preserved and we'd need to define what basic subset should actually work?

For context, our team is currently trying to apply guardrails at the client/converse layer rather than the LiteLLM configuration layer because 1/ the above-mentioned litellm bug is blocking, but also 2/ LiteLLM OSS Edition only supports a global guardrail configuration at this point - We couldn't set separate guardrails for different models or API keys without upgrading to Enterprise Edition.