Feature Proposal: Implement LLM API Configuration Fallback and Rotation System for Enhanced Reliability and Cost Optimization

[Godvvs]

1. Introduction / Problem Statement:

Currently,Roo code relies on a single, pre-configured Large Language Model (LLM) API key/configuration for its requests. If this specific API configuration becomes unavailable due to various reasons (e.g., rate limiting, temporary service disruption, key invalidation, network issues), the associated functionality fails until the configuration is manually updated or the issue resolves. This creates a single point of failure.

Furthermore, users often have access to multiple LLM API keys, potentially across different service tiers (including free or lower-cost options). The current system doesn't efficiently leverage these multiple keys to ensure service continuity or to optimize costs by prioritizing the use of less expensive options.

2. Proposed Solution:

We propose the implementation of an automated LLM API configuration fallback and rotation system. This system would allow users to define a pool of multiple saved API configurations (as previously discussed, e.g., "Gemini1", "Gemini2", ... , "default"). When initiating an LLM request, the system would attempt to use a designated primary or the last known working configuration.

If a request fails due to a detectable, potentially temporary issue (like rate limits or timeouts) with the currently selected configuration, the system should:

Temporarily Mark: Flag the failing configuration as temporarily unavailable.
Rotate/Fallback: Automatically select the next available configuration from the user-defined pool and retry the request.
Cooldown: Implement a configurable cooldown period for the flagged configuration. After this period expires, the configuration should be considered available again for subsequent attempts.
3. Key Features & Mechanism:

Configuration Pool Management: Allow users to manage a list/pool of saved LLM API configurations within the application settings. Optionally, allow users to set a priority order for fallback attempts.
Failure Detection: The system should identify specific error types that warrant a fallback (e.g., HTTP 429 Too Many Requests, HTTP 5xx Server Errors, connection timeouts). Critical errors like HTTP 401 Unauthorized (invalid key) might warrant permanently disabling the key or requiring manual intervention, distinct from temporary issues.
Rotation Logic: Define the sequence for trying configurations (e.g., follow priority order, round-robin through available configurations). Specify behavior if all configurations in the pool fail consecutively.
Cooldown Implementation: Introduce a configurable setting for the duration a configuration remains marked as unavailable after a failure (e.g., 5 minutes, 30 minutes).
Status Tracking: (Optional) Provide visibility to the user regarding the current status of each configuration in the pool (Active, Cooldown, Failed).
4. Benefits:

Enhanced Reliability & Resilience: Reduces downtime by automatically switching to a working configuration if the primary one fails.
Cost Optimization: Enables users to strategically place free or lower-cost API keys higher in the fallback order, maximizing their usage and potentially reducing reliance on more expensive tiers.
Improved Efficiency: Automates the process of switching keys, reducing the need for manual intervention during temporary outages or rate-limiting events.
Better Resource Utilization: Makes effective use of all available API keys provisioned by the user.

[petiteoffense]

This would be awesome.