[Feature]: LiteLLM MCP Roadmap

[ishaan-jaff]

The Feature

Starting this to discuss LiteLLM MCP Roadmap

• Allow setting mcp servers on LiteLLM Keys, Teams
• Allow adding, editing, deleting mcp servers on litellm ui
• Allow segmenting tools on /mcp/list using /mcp/list/{server_name}
• Expose a way for users to make an LLM API request and litellm calls /list tools, call tools and sends that to the LLM API
• Add support for certificate auth to MCP SSE servers

Are you a ML Ops Team?

No

Twitter / LinkedIn details

No response

[ishaan-jaff]

cc @msabramo @marty-sullivan @jeromeroussin

[krrishdholakia]

Can this be made into a discussion? @ishaan-jaff

it'll be easier to track

[ishaan-jaff]

done

[olafgeibig]

I think a very common problem is that very quickly the number of MCP servers you have installed explodes. I think that makes prompt engineering much more difficult because you have to carefully tune your prompt that it's not choosing the wrong tools for the task you plan to do. So I think it would be a great feature if the proxy would have several endpoints to offer different subsets of MCP servers that are installed so that you can have one endpoint for your IDE focused on software development tools and another for a general chat client. Just a mechanism to have different profiles with different sets of MCP servers available.

[krrishdholakia]

Could we leverage something like tags here? https://docs.litellm.ai/docs/proxy/tag_routing

[msabramo]

I wonder if you should leave room for the possibility of doing this, but not invest in it right away. LLMs are pretty smart and I don't know that picking the wrong tools would happen that often unless you have a lot of very similar tools (maybe some people have that use case; I don't see that being an issue for me). Maybe try it and then solve if that pops up a lot? But make sure whatever you do API-wise leaves room for that possibility. Maybe you can throw all_tools or something into the path of the request so you have a place where you can specify a tool collection if needed?

[msabramo]

Another thought off the top of my head - what if there was a way to specify which tools are automatically used based on prompt language and which aren't (and have to be explicitly requested)?

That lets you specify that say the "check deployment status" tool can be invoked automatically but the "do deployment tool" has to be explicitly requested so there are no accidental deploys.

Could allow that on the tool level or if you end up doing tool collections, it could be specified on the tool collection level - i.e.: idempotent_tools and non_idempotent_tools.

[olafgeibig]

@ishaan-jaff Yes, now I get it. That would probably do the trick. At least on a client level. I think many users would like to remove the necessity to manage multiple sets MCPs for different clients. But have one central place to manage and maintain the MCP installations while still be able to have different sets of MCPs for different applications.

[marty-sullivan]

I like the "segmentation" idea too. If you can narrow down the types of tools you want to expose to the LLM, that can only help.

Allow setting mcp servers on LiteLLM Keys, Teams

This is also a good one. It would definitely be good to be able to limit access to certain MCPs by org/team membership.

[wagnerjt]

Would love to tackle this one for the initial proxy features:

• CRUD operations of MCP servers isolated by team and virtual key
• UI exploring of MCP server and list tools
• Litellm SDK MCP client for streamable http transport
• Health checks on MCP servers (similar to health check on models via the ping request)

---

One thing to consider is restricting litellm to only use HTTP streamable transport since SSE requires a constant connection. If we wanted to support SSE, we would need to either stay connected consistently or only connect on tool runs and periodically listen for new changes.

---

I think the authentication is an important discussion and would be curious to see where you guys are going with that requirement.

It makes sense to include a pass through of a token from the litellm proxy to the MCP servers for specific authn/authz at the server. Do we also want litellm proxy to be able store it's own cert to generate jwt like the new feature?

[wagnerjt]

I made a bunch of progress on the backend parts for this.

Essentially the db and config for hosting the servers will look like

model LiteLLM_MCPServerTable {
  server_id    String         @id @default(uuid())
  alias        String?
  description  String?
  url          String
  transport    String         @default("http")
  spec_version String         @default("2025-03-26")
  auth_type    String?        
  created_at   DateTime?      @default(now()) @map("created_at")
  created_by   String?
  updated_at   DateTime?      @default(now()) @updatedAt @map("updated_at")
  updated_by   String?
}

and there will be the following defaults:

transport: "http", # one of (http|sse) - defaults to sse
spec_version: "2025-03-26", # one of (2024-11-05|2025-03-26) - defaults to 2025-03-26
auth_type: API_KEY, # one of (API_KEY|BASIC|BEARER_TOKEN) defaults to no auth

Now just waiting for the streamable http to come out this week to implement the last parts of the backend changes here. For context why I am pushing for streamable http support for litellm's proxy instead of SSE is the constant connection needed and other gunicorn and workers issues when running as explained here.

In the meantime, I will start working towards the frontend and tests this week. If @ishaan-jaff or @krrishdholakia has time to connect, that would be great since there will be breaking changes to the proxy with this PR and I want to validate that is okay :)!

---

As for the duplicate tools defined in #10404, I am hoping the team + virtual key additions are enough, but if not we can look towards prefixing the tools with the server's alias i.e. alias/tool_name and add a best practice to keep the aliases unique.