Could we support to configure router strategy based on each model group

[Colstuwjx]

It seems like the router setting is the global setting and apply to all model groups, how could we set it against per model groups ?

[krrishdholakia]

hey @Colstuwjx can you help me understand the scenario a bit better?

[Colstuwjx]

For example, when we have 2 groups of gpt-4o model, each group has 2 gpt-4o model endpoints, I just want to separate them as 2 individual model groups, with individual route strategy and ratelimit / api-key / retry settings. However, it seems like it's hard to configure this in litellm-proxy, am I wrong ?

[krrishdholakia]

hey @Colstuwjx open to this - can you share a mock config of how you would want to define this?

[Colstuwjx]

For example, a mock config would be like:

model_groups:
- name: hotel-model-group
  models:
  - name: gpt-4o-01
    model: openai/gpt-4o
    api_key: xxx
  - name: gpt-4o-02
    model: openai/gpt-4o
    api_key: xxx
  - name: deepseek-01
    model: openai/DeepSeek-R1-671B
    api_key: xxx
  router:
     strategy: static-weight
     strategy_settings:
         gpt-4o-01: 50%
         gpt-4o-02: 50%
     fallback: ["deepseek-01"]
- name: flight-model-group
...

I know it's a little bit complicate, just represent the requirements to configure model endpoint and route strategy per model group rather than globally setting.

[krrishdholakia]

hey @Colstuwjx acknowledging this. If you're just trying to set weights for a specific model group, you can already do that, it'll be respected by the default (simple-shuffle) routing strategy