Implement standardized provider interface and registry

Addresses issue #1534 by creating:
- Provider protocol and base class implementation
- Provider registry system for registration and discovery
- Sample implementation for OpenAI provider
- Documentation for provider system

This is the first step toward reorganizing the provider architecture.

Author: jxnl

docs/concepts/providers.md

@@ -0,0 +1,137 @@
+# Provider System
+
+Instructor supports multiple Large Language Model (LLM) providers through a standardized provider interface. This documentation explains how the provider system works and how to implement new providers.
+
+## Provider Interface
+
+All providers in Instructor implement a common interface that defines the expected behavior and capabilities. This interface is defined by the `ProviderProtocol` in `instructor.providers.base`.
+
+```python
+from instructor.providers.base import ProviderProtocol
+```
+
+The protocol requires the following properties and methods:
+
+- `supported_modes`: List of `Mode` values supported by the provider
+- `capabilities()`: Dictionary of provider capabilities (streaming, function_calling, etc.)
+- `create_client()`: Factory method to create an Instructor client for this provider
+- `create()`: Method to create structured output from messages
+- `create_stream()`: Method to stream structured output from messages
+
+## Base Provider Implementation
+
+A base implementation of the provider interface is available in `ProviderBase`, which handles common functionality like mode validation:
+
+```python
+from instructor.providers.base import ProviderBase
+```
+
+## Using Provider Registry
+
+Providers register themselves with the registry using the `register_provider` decorator:
+
+```python
+from instructor.providers import register_provider
+
+@register_provider("openai")
+class OpenAIProvider(ProviderBase):
+    # Implementation...
+```
+
+You can list all available providers and get a specific provider:
+
+```python
+from instructor.providers import list_providers, get_provider
+
+# List all providers
+providers = list_providers()
+print(providers)  # ['openai', 'anthropic', ...]
+
+# Get a specific provider
+openai_provider = get_provider("openai")
+```
+
+## Creating a New Provider
+
+To create a new provider, implement the `ProviderBase` class and register it:
+
+```python
+from typing import List, Dict, Any, Type, Union, Iterator, Optional
+from pydantic import BaseModel
+
+from instructor.mode import Mode
+from instructor.client import Instructor, AsyncInstructor
+from instructor.providers import register_provider
+from instructor.providers.base import ProviderBase
+
+
+@register_provider("my_provider")
+class MyProvider(ProviderBase):
+    """My custom LLM provider."""
+    
+    _supported_modes = [Mode.JSON]
+    
+    @classmethod
+    def create_client(
+        cls, 
+        client: Any, 
+        provider_id: Optional[str] = None,
+        **kwargs
+    ) -> Union[Instructor, AsyncInstructor]:
+        # Implementation...
+        
+    def create(
+        self,
+        response_model: Type[BaseModel], 
+        messages: List[Dict[str, Any]], 
+        mode: Mode,
+        **kwargs
+    ) -> BaseModel:
+        # Implementation...
+    
+    def create_stream(
+        self,
+        response_model: Type[BaseModel], 
+        messages: List[Dict[str, Any]],
+        mode: Mode,
+        **kwargs
+    ) -> Iterator[BaseModel]:
+        # Implementation...
+```
+
+## Provider Capabilities
+
+The `capabilities()` method returns a dictionary of provider capabilities:
+
+```python
+{
+    "streaming": True,           # Supports streaming responses
+    "function_calling": True,    # Supports function calling
+    "async": True,               # Supports async operations
+    "multimodal": False          # Supports multimodal inputs
+}
+```
+
+## Mode Support
+
+Each provider indicates which modes it supports through the `supported_modes` property. Common modes include:
+
+- `Mode.JSON`: Provider uses JSON mode for output format
+- `Mode.TOOLS`: Provider uses tools/functions for output format
+- `Mode.MARKDOWN`: Provider uses markdown for output format
+
+Providers validate that requested modes are supported:
+
+```python
+def validate_mode(self, mode: Mode) -> None:
+    if mode not in self.supported_modes:
+        supported_str = ", ".join(str(m) for m in self.supported_modes)
+        raise ValueError(
+            f"Mode {mode} not supported by this provider. "
+            f"Supported modes: {supported_str}"
+        )
+```
+
+## Provider-Specific Arguments
+
+Providers can accept additional arguments through `**kwargs` in the `create_client()`, `create()`, and `create_stream()` methods.

instructor/providers/__init__.py

@@ -0,0 +1,52 @@
+"""
+Provider registry and interface for Instructor.
+
+This module defines the provider protocol and registry for Instructor.
+Providers should be registered here to be discoverable by the auto_client.
+"""
+
+from typing import Optional, TypeVar
+
+# Type for provider base class
+T = TypeVar("T", bound="ProviderBase")
+
+# Global registry of providers
+_provider_registry: dict[str, type[T]] = {}
+
+
+def register_provider(name: str):
+    """
+    Decorator to register a provider class.
+
+    Args:
+        name: Unique provider identifier
+    """
+
+    def decorator(cls: type[T]) -> type[T]:
+        _provider_registry[name] = cls
+        return cls
+
+    return decorator
+
+
+def get_provider(name: str) -> Optional[type[T]]:
+    """
+    Get provider by name.
+
+    Args:
+        name: Provider name
+
+    Returns:
+        Provider class or None if not found
+    """
+    return _provider_registry.get(name)
+
+
+def list_providers() -> list[str]:
+    """
+    List all registered providers.
+
+    Returns:
+        List of provider names
+    """
+    return list(_provider_registry.keys())

instructor/providers/base.py

@@ -0,0 +1,129 @@
+"""
+Base provider interfaces for Instructor.
+
+This module defines the base provider protocols and classes that all providers should implement.
+"""
+
+from abc import abstractmethod
+from typing import Protocol, TypeVar, Generic, Any, Optional, Union, ClassVar
+from collections.abc import Iterator
+from pydantic import BaseModel
+
+from ..mode import Mode
+from ..client import Instructor, AsyncInstructor
+
+
+# Type variable for response model
+T = TypeVar("T", bound=BaseModel)
+
+
+class ProviderProtocol(Protocol, Generic[T]):
+    """Protocol defining what all providers must implement."""
+
+    @property
+    def supported_modes(self) -> list[Mode]:
+        """Modes supported by this provider."""
+        ...
+
+    @classmethod
+    def capabilities(cls) -> dict[str, bool]:
+        """Provider capabilities (streaming, function_calling, etc.)"""
+        ...
+
+    @classmethod
+    def create_client(
+        cls, client: Any, provider_id: Optional[str] = None, **kwargs
+    ) -> Union[Instructor, AsyncInstructor]:
+        """Create an Instructor client for this provider."""
+        ...
+
+    def create(
+        self,
+        response_model: type[T],
+        messages: list[dict[str, Any]],
+        mode: Mode,
+        **kwargs,
+    ) -> T:
+        """Create structured output from messages."""
+        ...
+
+    def create_stream(
+        self,
+        response_model: type[T],
+        messages: list[dict[str, Any]],
+        mode: Mode,
+        **kwargs,
+    ) -> Iterator[T]:
+        """Stream structured output from messages."""
+        ...
+
+
+class ProviderBase(Generic[T]):
+    """Base implementation for providers with common functionality."""
+
+    # Class variables that should be overridden by subclasses
+    _supported_modes: ClassVar[list[Mode]] = []
+
+    @property
+    def supported_modes(self) -> list[Mode]:
+        """Get modes supported by this provider."""
+        return self._supported_modes
+
+    @classmethod
+    def capabilities(cls) -> dict[str, bool]:
+        """
+        Get provider capabilities.
+
+        Returns a dictionary with keys:
+            - streaming: Whether the provider supports streaming
+            - function_calling: Whether the provider supports function calling
+            - async: Whether the provider supports async operations
+            - multimodal: Whether the provider supports multimodal inputs
+        """
+        return {
+            "streaming": hasattr(cls, "create_stream"),
+            "function_calling": hasattr(cls, "create_with_functions"),
+            "async": hasattr(cls, "acreate"),
+            "multimodal": hasattr(cls, "supports_multimodal")
+            and cls.supports_multimodal(),
+        }
+
+    @classmethod
+    def supports_multimodal(cls) -> bool:
+        """Whether this provider supports multimodal inputs."""
+        return False
+
+    @classmethod
+    @abstractmethod
+    def create_client(
+        cls, client: Any, provider_id: Optional[str] = None, **kwargs
+    ) -> Union[Instructor, AsyncInstructor]:
+        """
+        Create an Instructor client for this provider.
+
+        Args:
+            client: The provider's native client instance
+            provider_id: Optional provider identifier (e.g., URL or name)
+            **kwargs: Additional provider-specific arguments
+
+        Returns:
+            An instance of Instructor or AsyncInstructor
+        """
+        raise NotImplementedError()
+
+    def validate_mode(self, mode: Mode) -> None:
+        """
+        Validate that the mode is supported by this provider.
+
+        Args:
+            mode: The mode to validate
+
+        Raises:
+            ValueError: If the mode is not supported
+        """
+        if mode not in self.supported_modes:
+            supported_str = ", ".join(str(m) for m in self.supported_modes)
+            raise ValueError(
+                f"Mode {mode} not supported by this provider. "
+                f"Supported modes: {supported_str}"
+            )