-
Notifications
You must be signed in to change notification settings - Fork 36
Open
Description
The MCP catalog evaluation script currently extracts and stores two overlapping configurations for each MCP server:
- archestra_config.client_config_permutations.mcpServers - Contains multiple ways to run a server (docker, uvx, npx, etc.). This is displayed in the "Configuration" block on our website catalog (example:
https://www.archestra.ai/mcp-catalog/upstash__context7) - server.mcp_config - Contains one DXT-formatted configuration for running the server, where env variables reference values from user_config. This is what Archestra app uses to actually start MCP servers:

Both are extracted independently from READMEs using AI (via separate prompts) and located in /app/app/mcp-catalog/scripts/evaluate-catalog.ts.
Problem
- mcpServers extraction fails - LLM non-deterministic output causes incorrect config names, missed commands, or wrong parsing
- server.mcp_config extraction fails - LLM mixes different configurations, misses required env variables, or adds incorrect ones. MCP servers need exact env variables as documented, otherwise they crash
Solution
- Use regex patterns to extract archestra_config.client_config_permutations.mcpServers from documentation instead of relying on LLM interpretation, or propose your own approach
1.1. Scan all markdown files in the repository, not just README (example: https://github.com/korotovsky/slack-mcp-server/blob/master/docs/03-configuration-and-usage.md#using-npx) - Try to extract server.mcp_config from the original dxt manifest if available (https://github.com/korotovsky/slack-mcp-server/blob/master/manifest-dxt.json) with fallback to generating one from one of the
archestra_config.client_config_permutations.mcpServers
Test
We should be able to use generated configs in Archestra or other MCP client to run MCP servers
Metadata
Metadata
Assignees
Labels
No labels