Conduit LLM's First Release!

[nickna]

Conduit v0.1.0 Release Notes

We're excited to announce the inaugural release of Conduit, a unified API gateway for multiple LLM providers with OpenAI-compatible endpoints.

🚀 Highlights

Conduit provides a robust platform for working with multiple LLM providers through a single, consistent interface:

• OpenAI-compatible API endpoints for seamless integration with existing tools
• Multi-provider support for leading AI models (OpenAI, Anthropic, Google, Azure, etc.)
• Admin dashboard for configuration, monitoring, and management
• Docker-ready deployment for easy setup in any environment

🧩 Core Features

LLM Provider Support

• Multiple Provider Integration: Connect to OpenAI, Anthropic, Google Gemini, Azure OpenAI, Mistral, Cohere, Groq, Fireworks, Hugging Face, Ollama, Vertex AI, Bedrock, and more
• Provider Health Monitoring: Automatically track and display the status of connected LLM providers
• Graceful Failover: Configure fallback mechanisms when primary providers are unavailable

Routing & Management

• Intelligent Request Routing: Route requests based on various strategies:
  • Round-robin distribution
  • Random allocation
  • Least latency/cost optimization
  • Direct provider mapping
• Model Mapping: Create virtual model names that map to specific provider implementations
• Budget Management: Track and limit spending across providers

API Security & Control

• Virtual API Keys: Create and manage Conduit-specific API keys (condt_...) with built-in controls
• Usage Tracking: Monitor requests, tokens, and costs per API key
• Rate Limiting: Implement usage restrictions on a per-key basis

Developer Experience

• Streaming Support: Real-time token streaming for responsive applications
• Tools & Function Calling: Pass function definitions and handle tool calls
• Response Caching: Optimize performance and reduce costs with Redis caching

🔧 Technical Foundation

• Modern .NET Stack: Built on .NET 9 with Blazor for the admin UI
• Containerized Architecture: Separate Docker images for API and admin UI components
• Database Support: Compatible with SQLite and PostgreSQL backends
• Redis Integration: Optional Redis support for caching and distributed scenarios
• Extensible Design: Well-structured codebase with clear interfaces for adding new providers

💾 Installation

Docker Compose (Recommended)

version: "3.9"

services:
  postgres:
    image: postgres:16
    environment:
      POSTGRES_USER: conduit
      POSTGRES_PASSWORD: conduitpass
      POSTGRES_DB: conduitdb
    volumes:
      - pgdata:/var/lib/postgresql/data

  redis:
    image: redis:alpine
    volumes:
      - redis-data:/data

  api:
    image: ghcr.io/knnlabs/conduit-http:latest
    environment:
      DATABASE_URL: postgresql://conduit:conduitpass@postgres:5432/conduitdb
      CONDUIT_CACHE_ENABLED: "true"
      CONDUIT_CACHE_TYPE: "Redis"
      CONDUIT_REDIS_CONNECTION_STRING: "redis:6379"
    ports:
      - "5000:8080"

  webui:
    image: ghcr.io/knnlabs/conduit-webui:latest
    environment:
      DATABASE_URL: postgresql://conduit:conduitpass@postgres:5432/conduitdb
      CONDUIT_MASTER_KEY: your-secure-key-here
      CONDUIT_API_BASE_URL: http://api:8080
      CONDUIT_CACHE_ENABLED: "true"
      CONDUIT_CACHE_TYPE: "Redis"
      CONDUIT_REDIS_CONNECTION_STRING: "redis:6379"
    ports:
      - "5001:8080"

volumes:
  pgdata:
  redis-data:

Direct Installation

# Clone the repository
git clone https://github.com/knnlabs/Conduit.git
cd Conduit

# Build and run with Docker
docker compose up -d

🔮 What's Next

This release lays the foundation for a comprehensive LLM gateway. Our upcoming roadmap includes:

• Enhanced Observability: Expanded metrics, logging, and monitoring
• Advanced Routing Strategies: More sophisticated routing algorithms
• Additional Provider Integration: Support for emerging LLM providers
• Extended Multimodal Support: Improved handling of images and other non-text inputs
• API Enhancements: Additional endpoints and compatibility improvements
• Realtime Audio: Support for RealTime Audio Providers

📄 Documentation

For comprehensive documentation, review the docs folder in the repository.

---

Conduit is developed and maintained by KNN Labs, Inc.

---

This discussion was created from the release Conduit LLM's First Release!.

[nickna]

Quick Update:
We added online documentation here: https://knnlabs.github.io/Conduit/docs/intro