Reference Architecture Tooling | Draft

[omrishiv]

Git

	Self-Hosted/Community Edition	SaaS/Enterprise	External Community	Features	Gaps	Agentic Community Use
Gitea / forgejo	Y	Y	Stars - 49.1K	- Low memory use- Supports Actions- Includes container registry
Github	N	Y
Gitlab	Y	Y	Stars - 100

Container Registry

	Self-Hosted/Community Edition	SaaS/Enterprise	External Community	Features	Gaps	Agentic Community Use
Harbor	Y	N	Stars - 25.8K	- Image Scanning- Access Control- Image Signing
Docker/Hub	Y	Y	Stars - 9.6K*
Nexus	Y	Y	Stars - 2.2K
JFrog Container Registry	Y	Y
Quay	Y	Y	Stars - 2.6K
Github Container Registry	N	Y
Gitlab Container Registry	Y	Y

CI/CD

	Self-Hosted/Community Edition	SaaS/Enterprise	External Community	Features	Gaps	Agentic Community Use
ArgoCD	Y	Y (Akuity)	Stars - 19.9K
Flux	Y	N	Stars - 7.2K
Argo Workflows	Y	N	Stars - 15.7K
Tekton	Y	N	Stars - 8.7K

Visualization

	Self-Hosted/Community Edition	SaaS/Enterprise	External Community	Features	Gaps	Agentic Community Use
Grafana	Y	Y	Stars - 68.6K

Tracing

	Self-Hosted/Community Edition	SaaS/Enterprise	External Community	Features	Gaps	Agentic Community Use
Jaegar	Y		Stars - 21.5K
Zipkin	Y		Stars - 17.2K
Tempo	Y	Y	Stars - 4.6K

Logging - Collector

	Self-Hosted/Community Edition	SaaS/Enterprise	External Community	Features	Gaps	Agentic Community Use
FluentBit	Y		Stars - 7K

Logging - Storage

	Self-Hosted/Community Edition	SaaS/Enterprise	External Community	Features	Gaps	Agentic Community Use
OpenSearch	Y	Y	Stars - 11K

Metrics

	Self-Hosted/Community Edition	SaaS/Enterprise	External Community	Features	Gaps	Agentic Community Use
Prometheus	Y		Stars - 59.1K

Agent Observability

Tool	Self-Hosted/Community Edition	SaaS/Enterprise	External Community	Features	Gaps	Agentic Community Use
Langfuse	✅ Yes	✅ Yes	⭐ 13.3K stars	Open-source LLM engineering platform with tracing, evals, prompt management, metrics, and Langfuse-k8s deployment charts	Self-hosting complexity for scale, enterprise features require commercial license, recent v3 stability concerns
OpenLLMetry	✅ Yes	✅ Yes (Traceloop)	⭐ 6K stars	Open-source observability platform built on OpenTelemetry for comprehensive LLM application monitoring and tracing	OpenTelemetry dependency complexity, limited built-in evaluation metrics, requires additional tooling for analysis
Arize Phoenix	✅ Yes	✅ Yes (Arize)	⭐ 6.2K stars	Open-source AI observability platform for experimentation, evaluation, and troubleshooting LLM applications	Limited integration ecosystem, newer project with smaller community, primarily notebook-based workflows
Helicone	✅ Yes	✅ Yes	⭐ 4.1K stars	One line integration, LLM observability, monitoring, evaluation, experimentation, supports 20+ LLM providers	Proxy-based architecture can add latency, limited offline/air-gapped deployment options, cost tracking focus over evaluation
Comet Opik	✅ Yes	✅ Yes	⭐ 10.5K stars	Comprehensive tracing, automated evaluations, production-ready dashboards, LLM as a Judge metrics, dataset experiments	Relatively new to LLM space, complex pricing model, heavy integration with Comet ecosystem
Langsmith	❌ No	✅ Yes	N/A - Proprietary	LLM observability platform with tracing, prompt evaluation, and performance monitoring for LangChain applications	Vendor lock-in, LangChain ecosystem dependency, no self-hosting option, limited customization

Agent Evaluations

Tool	Self-Hosted/Community Edition	SaaS/Enterprise	External Community	Features	Gaps	Agentic Community Use
Trulens	✅ Yes	❌ No	⭐ 2.6K stars	Systematic evaluation and tracking for LLM experiments using feedback functions to assess quality, relevance, and safety	Limited real-time monitoring, complex setup for production use, primarily evaluation-focused rather than observability
HAL - Holistic Agent Leaderboard	✅ Open source evaluation harness available	✅ Third-party standardized leaderboard platform	✅ Very Strong - Princeton's SAgE team with active community contributions, multiple benchmark integrations - ⭐ 93 stars	Standardized, cost-aware evaluation, unified evaluation harness, multiple benchmarks (SWE-bench, AgentHarm, AppWorld, Cybench), research-backed methodology	No real-time observability, evaluation-only focus, requires technical setup for harness
Agent Evaluation (AWS Labs)	✅ Open source, self-deployable	❌ Not offered as SaaS	⚠️ Limited - recently launched (2024), small contributor base - ⭐ 258 stars	LLM-powered evaluator agent, concurrent multi-turn conversations, AWS service integration (Bedrock, Q Business, SageMaker), CI/CD pipeline support	Very new project, limited community adoption, AWS-centric approach, narrow focus on conversational testing
RAGAS	✅ Fully open source, self-deployable	❌ Not offered as managed service (community-driven)	✅ Extremely Strong - Active GitHub community, academic research backing, published framework - ⭐ ~9,800 stars	Reference-free RAG evaluation, LLM application optimization, test dataset generation, comprehensive metric suite, agent evaluation capabilities	Originally RAG-focused, agent evaluation is newer addition, requires technical setup

LLM Gateway

LLM Gateway	Self-Hosted/Community Edition	SaaS/Enterprise	External Community	Features	Gaps	Agentic Community Use
LiteLLM	✅	✅	⭐ 24.8K stars	100+ LLM APIs in OpenAI format, unified interface, spend tracking, rate limiting, retry/fallback logic, authentication, load balancing, proxy server management, automatic retries, real-time cost analytics	Limited to OpenAI format standardization
Kong Gateway	✅	✅	⭐ 41.2K stars	Lightweight, fast, flexible cloud-native API gateway, extensive API management capabilities, plugin-based extensibility, credential management, usage monitoring, governance, AI Gateway functionality	General-purpose API gateway, not AI-specific
Portkey Gateway	✅	✅	⭐ 8.7K stars	200+ LLMs, 50+ AI Guardrails, blazing fast, enterprise-grade reliability, security, and scale, seamless AI model switching, rate-limiting, caching, scalability for large-scale AI applications	Newer in market compared to others
KGateway	✅	❌	⭐ 4.6K stars	Kubernetes-native gateway solution, generally available with 2.0 release	Limited enterprise/SaaS offering, smaller ecosystem
Higress Gateway	✅	✅	⭐ 5.6K stars	AI Native API gateway with 100+ model support, MCP Server hosting and marketplace, streaming processing, Wasm plugin extensibility, production-grade (handles hundreds of thousands of requests/second), millisecond configuration updates	Learning curve for advanced features, primarily developed by Alibaba

MCP Gateway & Registry

Solution	Self-Hosted/Community Edition	SaaS/Enterprise	External Community	Key Features	Current Gaps	Agentic Community Use
mcp-gateway-registry	✅ Yes	⚠️ Partial	✅ Yes	• Unified access to governed list of MCP servers • Dynamic tool discovery using semantic search with FAISS indexing • Fine-Grained Access Control (FGAC) for MCP servers and tools • Amazon Cognito integration with dual auth modes • Web UI for service management • Health monitoring & real-time updates • Both SSE and Streamable HTTP transport support • Authentication patterns: user-behalf flows and agent identity flows	• Multi-level registry federation (#37) • Usage metrics and analytics (#38) • Tool popularity scoring (#39) • KeyCloak IdP support (#5) • Token vending capability (#18)	✅ AI agents can autonomously discover and execute specialized tools beyond their initial capabilities • Supports both user-behalf and autonomous agent flows • Integration with Strands Agents SDK • Programmatic API for agent integration

Async Communication

	Self-Hosted/Community Edition	SaaS/Enterprise	External Community	Features	Gaps	Agentic Community Use
Kafka	Y	Y	Stars - 30.3K	Well tested for large scale usage	Requires more management
RabbitMQ	Y	Y	Stars - 12.9K	Easier to get started and manage than Kafka
ActiveMQ	Y		Stars - 2.4K

Service Mesh

	Self-Hosted/Community Edition	SaaS/Enterprise	External Community	Features	Gaps	Agentic Community Use
istio	Y		Stars - 37K	mTLSsupports a large array of use cases	More difficult to operaterequires sidecars
LinkerD	Y	Y	Stars - 11K	mTLS
Cillium	Y		Stars - 21.9K	mTLSno sidecars

Service Discovery

	Self-Hosted/Community Edition	SaaS/Enterprise	External Community	Features	Gaps	Agentic Community Use
Consul	Y	Y	Stars - 29.1K

IDP

	Self-Hosted/Community Edition	SaaS/Enterprise	External Community	Features	Gaps	Agentic Community Use
Okta	N	Y
Cognito	N	Y
Keycloak	Y	N	Stars - 27.9K

LLM Hosting

	Self-Hosted/Community Edition	SaaS/Enterprise	External Community	Features	Gaps	Agentic Community Use
Ray	Y	Y	Stars - 37.7K
vLLM	Y		Stars - 50.5K

Agent Memory

	Self-Hosted/Community Edition	SaaS/Enterprise	External Community	Features	Gaps	Agentic Community Use
mem0	Y	Y	Stars - 35.3K
letta	Y		Stars - 17K
langmem	Y	Y	Stars - 808

Databases - Vector

	Self-Hosted/Community Edition	SaaS/Enterprise	External Community	Features	Gaps	Agentic Community Use
Marqo	Y		Stars - 4.9K
Weaviate	Y	Y	Stars - 13.7K
Milvus	Y	Y	Stars - 35.6K	LF AI&Data
qdrant	Y	Y	Stars - 24.3K
pgvector	Y		Stars - 16.3K
faiss	Y		Stars - 35.7K

Databases - Relational

	Self-Hosted/Community Edition	SaaS/Enterprise	External Community	Features	Gaps	Agentic Community Use
postgres	Y		Stars - 17.9K (mirror)	supports pgvector extension

[omrishiv]

Trying to put together some of the tools and I have some observations/questions. Do we want a completely self-container kubernetes deployment that can run with minimal outside services? We need to think through:

• certificates - if we don't want a domain, we need a self-signed certificate which creates issues with docker push/pull from the node
• local by default - if we want to keep the cluster self-contained, we might not want to deal with domains/load balancers
• read only instances - these add security, but make image building harder

[omrishiv]

I would like to propose the following architecture for discussion to start with for v0. This keeps most things that don't require SSL/domains internal to reduce complexity, while being mostly self contained and transportable across infrastructures. We will leverage cloud load balancers/certificates for the connection in, but should otherwise be movable.

[mzayats]

For the service sesh discussion today: we use Istio, but that's mostly because we use Istio for all other service mesh needs as well... It would be interesting to see if others found other service meshes better for agents.

[omrishiv]

That was my inclination as well as that's what I've used before