Auto Router for LLM depending on complexity of the task.

[tomahawk000]

The Auto Router facilitates LLM usage by placing an LLM at the forefront to determine routing based on task complexity. This approach could save a significant number of tokens.

For example, telling the LLM to 'npm run dev' or run xyz tool doesn't need to call the most expensive API you're utilizing, but a request concerning Firebase authentication or another complex issue would escalate to the top.

I understand this may not align well with caching.

[tomahawk000]

I believe OpenRouter is working on this type of feature.

[raymondclowe]

Openrouter has a "openrouter/auto" fake model that does this, it routes to a real model depending on the query. I can make it work in other apps but sometimes it doesn't with Roo, perhaps because it has some strange values in the model definition like "pricing" is set to "-1". I've seen it work, but also fail saying I didn't have enough credits.

{"id":"openrouter/auto","name":"Auto Router","created":1699401600,"description":"Your prompt will be processed by a meta-model and routed to one of dozens of models (see below), optimizing for the best possible output.\n\nTo see which model was used, visit [Activity](/activity), or read the model attribute of the response. Your response will be priced at the same rate as the routed model.\n\nThe meta-model is powered by [Not Diamond](https://docs.notdiamond.ai/docs/how-not-diamond-works). Learn more in our [docs](/docs/model-routing).\n\nRequests will be routed to the following models:\n- [openai/gpt-4o-2024-08-06](/openai/gpt-4o-2024-08-06)\n- [openai/gpt-4o-2024-05-13](/openai/gpt-4o-2024-05-13)\n- [openai/gpt-4o-mini-2024-07-18](/openai/gpt-4o-mini-2024-07-18)\n- [openai/chatgpt-4o-latest](/openai/chatgpt-4o-latest)\n- [openai/o1-preview-2024-09-12](/openai/o1-preview-2024-09-12)\n- [openai/o1-mini-2024-09-12](/openai/o1-mini-2024-09-12)\n- [anthropic/claude-3.5-sonnet](/anthropic/claude-3.5-sonnet)\n- [anthropic/claude-3.5-haiku](/anthropic/claude-3.5-haiku)\n- [anthropic/claude-3-opus](/anthropic/claude-3-opus)\n- [anthropic/claude-2.1](/anthropic/claude-2.1)\n- [google/gemini-pro-1.5](/google/gemini-pro-1.5)\n- [google/gemini-flash-1.5](/google/gemini-flash-1.5)\n- [mistralai/mistral-large-2407](/mistralai/mistral-large-2407)\n- [mistralai/mistral-nemo](/mistralai/mistral-nemo)\n- [deepseek/deepseek-r1](/deepseek/deepseek-r1)\n- [meta-llama/llama-3.1-70b-instruct](/meta-llama/llama-3.1-70b-instruct)\n- [meta-llama/llama-3.1-405b-instruct](/meta-llama/llama-3.1-405b-instruct)\n- [mistralai/mixtral-8x22b-instruct](/mistralai/mixtral-8x22b-instruct)\n- [cohere/command-r-plus](/cohere/command-r-plus)\n- [cohere/command-r](/cohere/command-r)","context_length":2000000,"architecture":{"modality":"text->text","input_modalities":["text"],"output_modalities":["text"],"tokenizer":"Router","instruct_type":null},"pricing":{"prompt":"-1","completion":"-1"},"top_provider":{"context_length":null,"max_completion_tokens":null,"is_moderated":false},"per_request_limits":null}

[BradKML]

I absolutely want this to work ngl, but at the same time the setup would have been complex, plus what if all we want is support "open weight" or "low price" models whenever possible? https://github.com/lm-sys/RouteLLM https://github.com/SomeOddCodeGuy/WilmerAI

[RyzeNGrind]

+1 this auto router feature from notdiamond is definitely worth having similar functionality in roocode. This way lower cost open weight models could be used for smaller tasks where their strengths and context could be maximized while the costlier API priced, or expensive closed source ones could be used for specifically big tasks like codebase refactoring or changes to ADR/ architectural model.

[BradKML]

The trouble is that we would need a router tool that knows when a problem is "hard" and work accordingly, or know which kinds of task it is handling. What do you have in mind in that regard?