API fallback

[sevenlayercookie]

Is your feature request related to a problem? Please describe.
I have multiple LLM sources (local, openai, openrouter, gemini, etc.). I always prefer my local LLM to handle requests when it can, but it's not always available.

Describe the solution you'd like
It would be nice if there were API fallback function. i.e. you add multiple API's and API keys, arrange them in a priority list, and then fallback to the next if no response or 400 response from each server.

Alternatives
Support for an already baked router such as Portkey

[clusterzx]

This issue has been marked as stale due to inactivity. Please respond to keep it open.

[nichtdu]

as a suggestion: i use litellm as a proxy for that. it works with paperless-ai for me

[gavinmcfall]

@nichtdu how do you specify the model from LiteLLM?

As an example, here is one of the models in my model_config.yaml in LiteLLM

  - model_name: "Google: gemini-1.5-pro"
    provider: google
    litellm_params:
      model: gemini/gemini-1.5-pro
    api_key: ${GOOGLE_API_KEY}
    model_info:
      id: gemini-1.5-pro
      mode: completion
      input_cost_per_token: 0.00000125  # USD $1.25 per million tokens (input)
      output_cost_per_token: 0.00000500  # USD $5.00 per million tokens (output)
      max_tokens: 32768  # Maximum context window: 32,768 tokens
    aliases: [general_gemini, vision_gemini]

Should I be using the model_name or id from Model_info? or something else entirely