DSPex

DSPex is a comprehensive Elixir implementation of DSPy (Declarative Self-improving Language Programs) that provides a unified interface for working with Large Language Models. It combines high-performance native Elixir implementations with seamless Python DSPy integration through Snakepit for complex ML workflows.

🚀 What We've Built: Revolutionary Bidirectional Integration

DSPex now features a revolutionary bidirectional tool bridge system that provides:

• ✅ Universal DSPy Access: Automatic discovery and exposure of all DSPy classes without manual wrappers
• ✅ Bidirectional Tool Calling: Python DSPy code can call back to Elixir functions during reasoning
• ✅ Real-time Schema Introspection: Automatically discovers DSPy modules, methods, and signatures
• ✅ Enhanced Metaprogramming: Advanced defdsyp macro with automatic tool registration
• ✅ Seamless Elixir Integration: Native Elixir modules that transparently call Python DSPy
• ✅ Production-Ready gRPC Bridge: High-performance communication with proper session management
• ✅ Zero-Config Discovery: No manual configuration needed - DSPy modules are auto-discovered

Schema Bridge Architecture

graph TB
    A[Elixir Application] --> B[DSPex.Bridge]
    B --> C[Universal call_dspy Tool]
    C --> D[Snakepit gRPC Bridge]
    D --> E[Python DSPy Adapter]
    E --> F[DSPy Library]
    
    G[Schema Discovery] --> H[discover_dspy_schema Tool]
    H --> D
    
    I[Auto-Generated Wrappers] --> J[defdsyp Macro]
    J --> B

Loading

🔧 Tool System Integration

DSPex implements a sophisticated multi-layer tool system:

Layer 1: Python Tools (17 Registered)

@tool(description="Universal DSPy function caller with introspection")
def call_dspy(module_path: str, function_name: str, args: List, kwargs: Dict)

@tool(description="Discover DSPy module schema with introspection") 
def discover_dspy_schema(module_path: str = "dspy")

@tool(description="Configure DSPy with a language model")
def configure_lm(model_type: str, **kwargs)

@tool(description="Check DSPy availability and version")
def check_dspy()

Layer 2: Elixir Bridge Tools

# Direct DSPy interaction
DSPex.Bridge.call_dspy("dspy.Predict", "__init__", %{"signature" => "question -> answer"})
DSPex.Bridge.call_method(instance_ref, "__call__", %{"question" => "What is DSPy?"})

# Schema discovery
{:ok, schema} = DSPex.Bridge.discover_schema("dspy")
# Discovers 70+ DSPy classes automatically

# Instance management
{:ok, {session_id, instance_id}} = DSPex.Bridge.create_instance("dspy.ChainOfThought", %{"signature" => "question -> reasoning, answer"})

Layer 3: High-Level Module API

# Migrated to use schema bridge internally
{:ok, predictor} = DSPex.Modules.Predict.create("question -> answer")
{:ok, result} = DSPex.Modules.Predict.execute(predictor, %{"question" => "What is Elixir?"})

{:ok, cot} = DSPex.Modules.ChainOfThought.create("question -> reasoning, answer") 
{:ok, result} = DSPex.Modules.ChainOfThought.execute(cot, %{"question" => "Explain quantum computing"})

🎯 Key Technical Achievements

1. Universal DSPy Bridge

• Problem Solved: Previously required manual wrapper creation for every DSPy class
• Solution: Universal call_dspy tool with automatic Python introspection
• Result: All DSPy classes are now accessible without writing Elixir wrappers

2. Schema-Driven Auto-Discovery

• Discovers 70+ DSPy classes automatically: Predict, ChainOfThought, ReAct, ProgramOfThought, BootstrapFewShot, MIPRO, ColBERTv2, etc.
• Method introspection: Automatically discovers method signatures, parameters, and docstrings
• Type-aware: Understands constructor requirements and parameter types

3. Production gRPC Integration

• High-performance: Uses Snakepit v0.4.1 gRPC bridge for fast Python communication
• Session management: Proper session affinity and instance storage
• Error handling: Comprehensive error parsing and meaningful error messages
• JSON parameter handling: Automatic serialization/deserialization of complex parameters

4. Metaprogramming System

The defdsyp macro generates complete DSPy wrapper modules:

defmodule MyApp.CustomPredictor do
  use DSPex.Bridge
  
  defdsyp __MODULE__, "dspy.Predict", %{
    execute_method: "__call__",
    result_transform: &MyApp.ResultTransforms.prediction_result/1,
    methods: %{
      "forward" => "forward_predict",
      "save" => "save_model"
    }
  }
end

# Generates:
# - create/2 function
# - execute/3 function  
# - call/3 function (stateless)
# - Custom methods: forward_predict/2, save_model/2

🏗️ System Architecture

User Request
    ↓
DSPex High-Level API (Predict, ChainOfThought, etc.)
    ↓
DSPex.Bridge (Universal Interface)
    ↓
Snakepit gRPC Bridge (Session Management)
    ↓
Python DSPy Adapter (Tool Registration)
    ↓
DSPy Library (70+ Classes)
    ↓
Language Models (Gemini, OpenAI, etc.)

Core Components

• DSPex.Bridge - Universal interface with call_dspy, discover_schema, create_instance
• DSPex.Modules.* - High-level Elixir modules using the schema bridge internally
• Python Adapter - 17 registered tools for DSPy interaction
• Schema Discovery - Automatic introspection of DSPy classes and methods
• Session Management - Proper instance lifecycle and session affinity

📊 Real Integration Status

✅ Fully Working Examples

All examples now use the new gRPC system with schema bridge:

1. 01_question_answering_pipeline.exs - ✅ Real Gemini API calls, working Predict and ChainOfThought
2. 02_code_generation_system.exs - ✅ Advanced reasoning, schema discovery (7 optimizers found)
3. 03_document_analysis_rag.exs - ✅ Document analysis pipeline with real LLM responses
4. 04_optimization_showcase.exs - ✅ DSPy optimizers and advanced features
5. 05_streaming_inference_pipeline.exs - ✅ Streaming inference demonstrations

🔍 Schema Discovery Results

# Actual results from our system:
$ DSPex.Bridge.discover_schema("dspy")
# Found 74 DSPy classes/functions:
#   - Predict (class)
#   - ChainOfThought (class)  
#   - ReAct (class)
#   - ProgramOfThought (class)
#   - BootstrapFewShot (class)
#   - MIPRO (class)
#   - ColBERTv2 (class)
#   - ... and 67 more

🚀 Quick Start

1. Basic DSPy Operations (Now Working!)

# Start applications
{:ok, _} = Application.ensure_all_started(:snakepit)
{:ok, _} = Application.ensure_all_started(:dspex)

# Configure Gemini
Snakepit.execute_in_session("session", "configure_lm", %{
  "model_type" => "gemini",
  "api_key" => System.get_env("GOOGLE_API_KEY"),
  "model" => "gemini-2.5-flash-lite"
})

# Create and use DSPy modules
{:ok, predictor} = DSPex.Modules.Predict.create("question -> answer")
{:ok, result} = DSPex.Modules.Predict.execute(predictor, %{"question" => "What is Elixir?"})
# Result: "Elixir is a dynamic, functional language designed for building maintainable and scalable applications."

{:ok, cot} = DSPex.Modules.ChainOfThought.create("question -> reasoning, answer")
{:ok, result} = DSPex.Modules.ChainOfThought.execute(cot, %{"question" => "If a train travels 120 miles in 2 hours, what is its average speed?"})
# Result includes step-by-step reasoning and final answer

2. Direct Bridge Usage

# Discover available DSPy modules
{:ok, schema} = DSPex.Bridge.discover_schema("dspy")
IO.puts("Found #{map_size(schema)} DSPy classes")

# Direct DSPy calls
{:ok, {session_id, instance_id}} = DSPex.Bridge.create_instance("dspy.ReAct", %{
  "signature" => "task -> thought, action, observation, answer"
})

{:ok, result} = DSPex.Bridge.call_method({session_id, instance_id}, "__call__", %{
  "task" => "Create a function to merge two sorted arrays"
})

3. Auto-Generated Wrappers

# Generate a custom DSPy wrapper
defmodule MyApp.Summarizer do
  use DSPex.Bridge
  
  defdsyp __MODULE__, "dspy.ChainOfThought", %{
    execute_method: "__call__",
    result_transform: fn result -> 
      # Custom result transformation
      %{"summary" => get_in(result, ["answer"])}
    end
  }
end

# Use the generated wrapper
{:ok, summarizer} = MyApp.Summarizer.create(%{"signature" => "document -> summary"})
{:ok, result} = MyApp.Summarizer.execute(summarizer, %{"document" => "Long document text..."})

📋 Installation & Setup

Dependencies

# mix.exs
def deps do
  [
    {:dspex, "~> 0.2.0"}
  ]
end

Python Dependencies

# Install DSPy and required packages
pip install dspy-ai>=2.6.0
pip install litellm  # For LLM provider support
pip install grpcio grpcio-tools  # For gRPC bridge

Configuration

# config/config.exs
config :snakepit,
  pooling_enabled: true,
  adapter_module: Snakepit.Adapters.GRPCPython,
  pool_config: %{
    pool_size: 4,
    adapter_args: ["--adapter", "dspex_adapters.dspy_grpc.DSPyGRPCHandler"]
  }

config :dspex,
  default_model: "gemini-2.5-flash-lite"

🔧 Advanced Features

Schema-Based Code Generation

# Generate documentation for all DSPy modules
{:ok, docs} = DSPex.Bridge.generate_docs("dspy")
File.write!("dspy_docs.md", docs)

# Discover specific module types
{:ok, schema} = DSPex.Bridge.discover_schema("dspy.teleprompt")  # Optimizers
{:ok, schema} = DSPex.Bridge.discover_schema("dspy.retrieve")   # Retrievers

Session Management

# Create session-aware instances
{:ok, predictor} = DSPex.Modules.Predict.create("question -> answer", 
  session_id: "my_session")

# All calls use the same session
{:ok, result1} = DSPex.Modules.Predict.execute(predictor, %{"question" => "First question"})
{:ok, result2} = DSPex.Modules.Predict.execute(predictor, %{"question" => "Second question"})

Error Handling & Debugging

# Comprehensive error parsing
{:error, error} = DSPex.Modules.Predict.create("invalid signature format")
# Returns: "Invalid signature format"

# Detailed tracebacks for debugging
{:error, error} = DSPex.Bridge.call_dspy("nonexistent.Module", "__init__", %{})
# Returns detailed Python traceback

📚 Examples Deep Dive

Real Working Examples (Updated for Schema Bridge)

# All examples now use real Gemini API calls
export GOOGLE_API_KEY=your-api-key

# Basic question answering with real LLM responses
mix run examples/dspy/01_question_answering_pipeline.exs

# Advanced code generation with reasoning
mix run examples/dspy/02_code_generation_system.exs  

# Document analysis RAG system
mix run examples/dspy/03_document_analysis_rag.exs

# DSPy optimization showcase
mix run examples/dspy/04_optimization_showcase.exs

# Streaming inference pipeline  
mix run examples/dspy/05_streaming_inference_pipeline.exs

Example Output (Real Results)

$ mix run examples/dspy/01_question_answering_pipeline.exs

=== Question Answering Pipeline Demo ===

1. Basic Prediction
-------------------
✓ Created Predict instance: {"pipeline_session", "predict_344934"}

Q: What is the capital of France?
A: The capital of France is Paris.

Q: What is 2 + 2?  
A: 4

Q: Who wrote Romeo and Juliet?
A: William Shakespeare wrote Romeo and Juliet.

2. Chain of Thought Reasoning
-----------------------------
✓ Created ChainOfThought instance: {"pipeline_session", "chainofthought_408431"}

Q: If a train travels 120 miles in 2 hours, and then 180 miles in 3 hours, what is its average speed for the entire journey?

Reasoning: To find the average speed for the entire journey, we need to calculate the total distance traveled and divide it by the total time taken.

The train travels 120 miles in the first part of the journey and 180 miles in the second part.
Total distance = Distance 1 + Distance 2 = 120 miles + 180 miles = 300 miles.

The train travels for 2 hours in the first part and 3 hours in the second part.
Total time = Time 1 + Time 2 = 2 hours + 3 hours = 5 hours.

Average speed = Total distance / Total time = 300 miles / 5 hours = 60 miles per hour.

Answer: The average speed for the entire journey is 60 miles per hour.

🔍 Technical Implementation Details

Constructor Parameter Binding (Fixed)

• Issue: Snakepit was serializing Elixir maps as JSON strings
• Solution: Automatic JSON parsing in Python adapter with DSPy-specific signature handling
• Result: Seamless parameter passing between Elixir and Python

Session Affinity & Instance Storage

• Sessions: Proper gRPC session management with worker affinity
• Instances: Python objects stored in _MODULE_STORAGE with unique IDs
• Lifecycle: Automatic cleanup and session-aware instance retrieval

Result Transformation Pipeline

• Raw DSPy Results → Python Serialization → Elixir Transformation → User-Friendly Format
• Handles: completions, prediction_data, reasoning/answer pairs, error states
• Extensible: Custom result transformers via defdsyp configuration

🚦 Testing

# Run all tests
mix test

# Run specific test layers  
mix test.fast        # Mock adapter tests (~70ms)
mix test.protocol    # Protocol tests
mix test.integration # Full integration tests with real DSPy

# Test schema bridge specifically
mix test test/dspex/bridge_test.exs

🎯 Performance & Monitoring

Metrics

• gRPC Bridge Latency: ~2-5ms per call
• DSPy Module Creation: ~50-100ms (includes LLM configuration)
• Inference Calls: Depends on LLM provider (Gemini: ~500-2000ms)
• Schema Discovery: ~200ms for full DSPy introspection (70+ classes)

Monitoring

# Built-in session and performance monitoring
Snakepit.pool_status()
DSPex.Bridge.discover_schema("dspy") |> map_size()  # Check available modules

🛣️ Roadmap

✅ Completed (Current Release)

• Universal DSPy bridge with automatic discovery
• Schema-driven code generation with defdsyp macro
• Production gRPC integration with Snakepit v0.4.1
• All 5 examples working with real API calls
• Comprehensive error handling and debugging
• Session management and instance lifecycle

🔄 In Progress

• Performance optimization and caching
• Additional DSPy module wrappers (ReAct, ProgramOfThought)
• Streaming inference support
• Advanced optimization techniques

📋 Planned

• Distributed DSPy execution
• Model management and versioning
• Comprehensive benchmarking suite
• Visual pipeline builder
• Advanced monitoring and observability

🤝 Contributing

DSPex is built with a sophisticated architecture that makes contributions straightforward:

1. Python Tools: Add new @tool decorated methods to the Python adapter
2. Elixir Modules: Create new modules using DSPex.Bridge.call_dspy
3. Auto-Generated Wrappers: Use the defdsyp macro for custom DSPy integrations
4. Examples: All examples use the schema bridge system

# Development setup
git clone https://github.com/nshkrdotcom/dspex.git
cd dspex
mix deps.get
mix test

# Interactive development
iex -S mix

📄 License

MIT

🙏 Acknowledgments

• DSPy - The revolutionary Python framework that inspired this implementation
• Snakepit - The high-performance Python integration bridge that makes this possible
• LiteLLM - Universal LLM API interface used for model access
• The Elixir Community - For building an amazing ecosystem that makes projects like this possible

---

DSPex - Bringing the power of DSPy to the Elixir ecosystem with production-ready performance and developer-friendly APIs.