Proposal: Allow a model with no model ("langfuse"), and per-provider keys for langfuse/other prompt management backends.

[radiosilence]

Currently if I want to use langfuse I have to define it like this somehow:

  - model_name: prompt-default
    litellm_params:
      model: langfuse/openai/gpt-4o
      api_key: os.environ/OPENAI_API_KEY

However, I might be defining model in my langfuse config, so the model param becomes moot:

  - model_name: prompt-default
    litellm_params:
      api_key: os.environ/OPENAI_API_KEY

However, if I now want to try out claude in using langfuse config, it breaks because the API key is the wrong one.

One way to solve this is to store the api_key in the langfuse prompt config, which is a terrible idea and bad.

Proposal

  - model_name: prompt-default
    api_keys:
       anthropic: os.environ/ANTHROPIC_API_KEY
       openai: os.environ/OPENAI_API_KEY
    litellm_params:
      model: langfuse

You can sooorta do this with custom handlers I think, however, it isn't great or formalized and a bit of a hack. I will have a go at doing this after standup and breakfast.

[radiosilence]

Ok after some experimentation, it seems that if you don't pass the API key to the config, it picks up the correct one from your environment anyway!

So you can just do:

  - model_name: dynamic
    litellm_params:
      model: langfuse/
  - model_name: dynamic-claude-fallback
    litellm_params:
      model: langfuse/anthropic/claude-sonnet-4-20250514

and as long as you have ANTHROPIC_API_KEY and OPENAI_API_KEY in your env, it seems to just work. Is this documented anywhere.