Help with passthrough endpoints?

[Dima-Mediator]

Hi, the documentation to pass-through endpoints is rather scarce. Can someone help?

1. Is there a way to enable load-balancing (with different back-end provider api keys) for "built-in" pass-through endpoints (ex: /anthropic/v1/messages)? As per the documentation examples it seems to rely on $ANTHROPIC_API_KEY and that's it?

2. Can Custom Pass-Through Endpoints completely replace built-in endpoints, and is load balancing possible with custom endpoints?

3. Built-in passthrough endpoints (/anthripic, /gemini etc) support LiteLLM virtual keys auth by default, but the same for Custom Pass-Through Endpoints requires setting " auth: true " and is an Enterprise feature - is this correct?

4. What exactly is the "use_client_credentials_pass_through_routes" setting doing? Is it for custom pass-through endpoint or all (including built-in)? Is it entperise-only?

Thanks!

[Dima-Mediator]

Following up to @krrishdholakia response in #361 (comment)

1. At the very least: improve documentation (https://docs.litellm.ai/docs/proxy/pass_through) with the answers on the above...

2. In the ideal world if I were designing this, and wanted to preserve maximum backwards compatibility and inherit current configs structure, the end-to-end config would look like below..

general_settings:

  master_key: sk-1234

  pass_through_endpoints:
    
     # Existing "Custom" endpoint specification as documented https://docs.litellm.ai/docs/proxy/pass_through
     
    - path: "/v1/rerank"
      type: custom                                       ## New optional param (defaults to "custom" for backward compatibility)                        
      target: "https://api.cohere.com/v1/rerank"          
      headers:                                            
        Authorization: "bearer os.environ/COHERE_API_KEY" 
        content-type: application/json                    
        accept: application/json
      forward_headers: True                     

    - path "/anthropic"
      type: anthropic               # Shall work as Anthropic for billing purposes, LiteLLM keys, etc.
      litellm_params:
        api_base: <...>
        api_key: <your-api-key>
        <any other parameters that might be applicable for pass-through endpoints of a certain type.>

    - path: "/anthropic"            # Endpoint path is repeated which indicates load-balanced deployment (similar to models)
      type: anthropic          
      litellm_params:
        api_base: <...>
        api_key: <your-api-key>
        (any other parameters that might be applicable)=

    - path: "/anthropic-fallback" 
      type: anthropic
      litellm_params:
        api_base: <...>
        api_key: <your-api-key>

    - path: "/vertex_ai"
      type: vertex_ai
      enabled: false               # Optional param "enabled" (default true). This is how users could explicitly disable a standard endpoint.
                       # Default behavior (when no config is given) keep as it works today (all standard endpoints available)

# Fallback config for pass-through endpoints.  Arguably, supporting fallback is more important then supporting load-balancing.

router_settings:
  pass_through_endpoints_fallbacks: [{"/anthropic": ["/anthropic-fallback"]}]

[krrishdholakia]

Hey @Dima-Mediator this is great. Does type refer to our mapped passthrough routes?

[krrishdholakia]

Curious, what is the problem you're trying to solve with this?

[Dima-Mediator]

type refer to our mapped passthrough routes?

That was my thought, yes. Whatever is supported already - Anthropic, Bedrock, Vertex_AI, etc.

Curious, what is the problem you're trying to solve with this?

Failover for a hypothetical situation that a major provider unexpectedly blocks/bans the key or an account due to something that users may have done which evaded guardrails (if any), or any other reasons. Like providers may (suddenly) revoke the API key that leaked and was found in the open. Or credits run out (and auto-top-up did not go through). In other words, resiliency. Not having a single point of failure; Which in this case is Anthropic's or Gemini's upstream API key - something that we don't even fully control. For normal models deployment (/chat/completions) we can configure it resilient with different API keys on different provider's accounts - but not for provider-specific endpoints.

Some other users may also think of also getting higher effective rate limits by distributing workload across accounts; But frankly I'm not sure if that is really important. Tier 5 is easy to reach, and provides very very generous limits.