Refactor: Provider registry system with URL-based matching and third-party registration

[jxnl]

Our current conditional import approach in __init__.py is becoming unwieldy as we add more providers. We should replace it with a registry pattern that allows providers to self-register and be discovered dynamically, including support for URL-based discovery to enable third-party providers without modifying core code.

Current problems:

• Complex conditional import logic in __init__.py that's difficult to maintain
• No easy way to query available/supported providers at runtime
• Third-party providers can't extend Instructor without modifying core code
• Import-time errors if dependencies aren't installed, even if unused
• URL matching for providers is hard-coded in auto_client.py

Proposed solution:

Implement an enhanced provider registry system with URL pattern matching:

# providers/__init__.py
from typing import Dict, Callable, Any, Optional, Type, TypeVar, Pattern, List
import re
from ..core.client import Instructor, AsyncInstructor

T = TypeVar('T', bound='ProviderBase')
_provider_registry: Dict[str, Type[T]] = {}
_url_patterns: Dict[str, List[Pattern]] = {}

def register_provider(name: str, url_patterns: Optional[List[str]] = None):
    """
    Decorator to register a provider class.
    
    Args:
        name: Unique provider identifier
        url_patterns: List of regex patterns for matching provider URLs
    """
    def decorator(cls: Type[T]) -> Type[T]:
        _provider_registry[name] = cls
        
        if url_patterns:
            _url_patterns[name] = [re.compile(pattern) for pattern in url_patterns]
            
        return cls
    return decorator

def get_provider_by_name(name: str) -> Optional[Type[T]]:
    """Get provider by name."""
    return _provider_registry.get(name)

def get_provider_by_url(url: str) -> Optional[tuple[str, Type[T]]]:
    """
    Get provider that matches the given URL.
    
    Returns:
        Tuple of (provider_name, provider_class) or None if no match
    """
    for provider_name, patterns in _url_patterns.items():
        for pattern in patterns:
            if pattern.match(url):
                return provider_name, _provider_registry[provider_name]
    return None

With provider registration:

# providers/oai_provider.py
@register_provider(
    name="openai",
    url_patterns=[
        r"^https://api\.openai\.com/.*$",
        r"^https://.*\.openai\.azure\.com/.*$"
    ]
)
class OpenAIProvider(ProviderBase):
    # Implementation...

# For arbitrary model providers:
# providers/ollama.py
@register_provider(
    name="ollama", 
    url_patterns=[
        r"^ollama/.*$",
        r"^http://localhost:11434/.*$"
    ]
)
class OllamaProvider(ProviderBase):
    # Implementation...

Additional enhancements:

1. Add entry point support for third-party providers
2. Support for provider model options
3. Dynamic capability discovery

Implementation steps:

1. Create the provider base class with URL pattern registration
2. Update each provider to register with URL patterns
3. Modify auto_client.py to use the registry and URL matching
4. Add entry point support for third-party extensions
5. Add utilities for dynamically discovering provider capabilities