feat: Add AI-powered Smart Test Runner system

[mickael-palma-wttj]

This PR extracts duplicate AI provider code into a shared module and introduces an AI-powered Smart Test Runner that intelligently selects tests based on code changes. The refactoring eliminates ~400 lines of duplicate code while adding comprehensive test selection capabilities.

Technical Changes

Shared AI Services Module

• Created .github/scripts/shared/ai_services.rb with centralized AI provider classes
• Extracted HTTPClient class with timeout handling and JSON response parsing
• Implemented AIProvider base class with standardized make_request interface
• Migrated AnthropicProvider with Claude API integration (API version 2023-06-01, claude-opus-4-20250514 model)
• Refactored DustProvider with enhanced logging including conversation IDs and URIs
• Added AIProviderFactory for provider instantiation based on configuration
• Created SharedLoggerFactory with configurable debug logging

Code Deduplication

• Removed duplicate AnthropicProvider class from pr_review.rb (131 lines)
• Removed duplicate DustProvider class from pr_review.rb (193 lines)
• Removed duplicate HTTPClient class from pr_review.rb (60 lines)
• Removed duplicate AIProviderFactory and LoggerFactory classes
• Updated method calls from request_review to make_request across codebase
• Modified provider initialization to use direct API keys instead of config objects

Smart Test Runner Implementation

• Added smart_test_runner.rb with AI-powered test selection logic
• Implemented SmartTestConfig class for environment variable management
• Created GitChangeAnalyzer for parsing git diffs and identifying changed files
• Built TestDiscoveryService for mapping test files to source files
• Developed AITestSelector using external prompt templates for test selection
• Added smart_test_selection_prompt.md with Ruby-specific test selection guidelines

GitHub Actions Integration

• Created .github/workflows/smart_tests.yml workflow for automated test selection
• Added environment variable configuration for multiple AI providers
• Implemented test execution logic with fallback to full test suite

Enhanced Logging

• Added emoji-based status indicators in DustProvider (✅, 🔗, ⏳, 🔍, 🔄, ⚠️, ❌)
• Included conversation URIs in log messages for debugging
• Enhanced retry mechanism logging with attempt counters and wait times

Test Coverage

• Updated pr_review_spec.rb to use new provider initialization patterns
• Added comprehensive test suite in smart_test_runner_spec.rb
• Fixed method expectations from request_review to make_request
• Updated provider mocking to match new constructor signatures

Documentation

• Added docs/SMART_TEST_RUNNER.md with detailed usage instructions
• Created scripts/test_smart_runner.rb for local testing
• Updated README.md with Smart Test Runner section

Rationale

The refactoring addresses code duplication where identical AI provider classes existed in multiple scripts, making maintenance difficult and error-prone. The Smart Test Runner solves the problem of inefficient CI execution by using AI to analyze code changes and select only relevant tests, potentially reducing test execution time by 50-80% for small changes while maintaining comprehensive coverage through intelligent dependency analysis.

Impact Analysis

Dependencies Affected

• AI Provider Classes: All scripts using AnthropicProvider or DustProvider now depend on shared module
• HTTP Client: Centralized HTTP handling affects all API communication
• Logger Factory: Standardized logging across all AI-powered scripts
• Test Execution: GitHub Actions workflows now include intelligent test selection

Modules Modified

• .github/scripts/pr_review.rb - Refactored to use shared services
• spec/github/scripts/pr_review_spec.rb - Updated test expectations
• New shared module affects all future AI-powered scripts

Configuration Changes

• Environment variables now support both Anthropic and Dust AI providers
• GitHub Actions secrets required for AI provider authentication
• Fallback mechanisms ensure continued operation if AI services fail

Performance Impact

• Reduced code duplication improves maintainability
• Smart Test Runner reduces CI execution time for incremental changes
• Enhanced logging provides better debugging capabilities for AI

[github-actions]

PR Code Review: AI Test Runner Implementation

🔍 Code Quality & Architecture

✅ Strengths

Strong Adherence to SOLID Principles:

• Single Responsibility: Each class has a clear, focused purpose (e.g., GitChangeAnalyzer for git operations, AITestSelector for test selection)
• Open/Closed: The AIProvider interface allows easy extension for new providers without modifying existing code
• Dependency Inversion: Proper use of dependency injection throughout, especially in constructors

Excellent Module Organization:

• Proper namespace usage under existing KanbanMetrics structure
• Clean separation between GitHub scripts and shared services
• Good use of mixins like RetryHandler and DustResponseProcessor

Design Patterns Implementation:

• Strategy Pattern: Well-implemented for different AI providers (AnthropicProvider, DustProvider)
• Factory Pattern: AIProviderFactory cleanly abstracts provider creation
• Template Method: External prompt template system is elegant

⚠️ Areas for Improvement

Large Method Complexity:

# DiffParser#parse_streaming method is quite long (40+ lines)
def parse_streaming(diff_output)
  # Consider breaking this into smaller methods
  changed_files = []
  current_file = nil
  # ... 40+ lines of logic
end

Recommendation: Extract helper methods like process_diff_line, extract_file_path, etc.

Configuration Coupling:

# AITestConfig has many responsibilities
class AITestConfig
  attr_reader :repo, :pr_number, :commit_sha, :base_ref, :github_token,
              :api_provider, :anthropic_api_key, :dust_api_key, :dust_workspace_id, :dust_agent_id

Recommendation: Consider splitting into provider-specific config objects.

🎨 Style & Maintainability

✅ Excellent Ruby Idioms

Proper Use of Ruby Features:

# Good use of safe navigation and keyword arguments
def initialize(logger, timeouts = {})
  @http_timeout = timeouts[:http_timeout] || 30
  @read_timeout = timeouts[:read_timeout] || 120
end

# Proper use of case statements
case config.api_provider
when 'anthropic'
  AnthropicProvider.new(config.anthropic_api_key, http_client, logger)
when 'dust'
  DustProvider.new(config.dust_api_key, config.dust_workspace_id, config.dust_agent_id, http_client, logger)
end

Clean Error Handling:

def analyze_changes(config)
  # ... implementation
rescue StandardError => e
  logger.error "Failed to analyze changes: #{e.message}"
  { diff: '', changed_files: [], analysis: {} }
end

⚠️ Minor Style Issues

Inconsistent String Literals:

# Mix of single and double quotes - should be consistent
logger.info '🚀 Starting AI Test Runner'
logger.error "Failed to analyze changes: #{e.message}"

Recommendation: Use double quotes consistently for interpolated strings, single quotes for literals.

Magic Numbers:

# These should be named constants
LARGE_DIFF_THRESHOLD = 10_000_000 # Good
max_lines = 100 # Should be a constant

🧪 Testing & Coverage

✅ Strong Test Quality

Excellent Four-Phase Pattern:

it 'selects relevant tests using AI' do
  # Arrange
  allow(mock_ai_provider).to receive(:make_request).and_return(ai_response)
  
  # Act
  result = selector.select_tests(changes, test_discovery)
  
  # Assert
  expect(result[:selected_tests]).to eq(['spec/lib/example_spec.rb'])
  expect(result[:reasoning]['risk_level']).to eq('low')
end

Comprehensive Edge Case Testing:

• Tests for invalid JSON responses
• Tests for API failures with fallbacks
• Tests for empty change sets
• Tests for configuration validation

Good Use of Test Doubles:

let(:mock_ai_provider) { instance_double(AnthropicProvider) }
let(:config) do
  instance_double(AITestConfig,
                  anthropic?: true,
                  dust?: false,
                  api_provider: 'anthropic',
                  anthropic_api_key: 'test_key')
end

⚠️ Testing Gaps

Missing Integration Tests:

• No tests for the actual git command execution
• Limited testing of the full workflow end-to-end
• No VCR cassettes for external API calls

Recommendation: Add integration tests with real git repositories and VCR for API interactions.

🔒 Security & Performance

✅ Security Strengths

Proper Input Validation:

def extract_retry_after(response)
  return nil unless response.respond_to?(:headers) || response.respond_to?(:header)
  retry_after = response.respond_to?(:headers) ? response.headers['Retry-After'] : response.header['Retry-After']
  return nil unless retry_after
  retry_after.to_i if retry_after.to_i.positive?
end

Safe Command Execution:

cmd = [
  'git', 'diff', '--no-color',
  '--no-renames',
  '--diff-filter=ACMRT',
  '--stat=1000',
  "#{base}...#{head}"
]
stdout, stderr, status = Open3.capture3(*cmd)

Proper Secret Handling:

• API keys stored as GitHub secrets
• No hardcoded credentials in code

⚠️ Security Concerns

File Path Validation:

# This could be strengthened
file_path = line.sub(%r{^ b/}, '').strip

Recommendation: Add explicit path traversal protection:

def sanitize_file_path(path)
  # Prevent directory traversal
  return nil if path.include?('..')
  return nil if path.start_with?('/')
  path.strip
end

🚀 Performance Considerations

Excellent Memory Management:

# Good use of streaming for large diffs
def parse_streaming(diff_output)
  io = StringIO.new(diff_output)
  io.each_line do |line|
    # Process line by line instead of loading all into memory
  end
ensure
  io&.close
end

Smart Thresholds:

LARGE_DIFF_THRESHOLD = 10_000_000 # 10MB
MEMORY_DIFF_THRESHOLD = 1_000_000 # 1MB
MAX_STREAMING_LINES = 100

Retry with Exponential Backoff:

def retry_with_backoff(max_retries: DEFAULT_MAX_RETRIES, base_delay: DEFAULT_BASE_DELAY)
  # Proper exponential backoff with jitter
  delay = base_delay * (backoff_factor**(attempt - 1))
  delay += rand * 0.1 * delay # Jitter to prevent thundering herd
end

📋 Summary & Actions

🎯 Key Recommendations

1. High Priority: Break down large methods in DiffParser and DustProvider
2. Medium Priority: Add integration tests with VCR cassettes
3. Low Priority: Standardize string quote usage and extract magic numbers

✅ Coding Standards Compliance

• Module Organization: ✅ Excellent
• Design Patterns: ✅ Well implemented
• SOLID Principles: ✅ Strong adherence
• Error Handling: ✅ Comprehensive
• Testing Standards: ✅ High quality with minor gaps
• Security: ✅ Good with minor improvements needed

🏆 Standout Features

1. Intelligent Fallback Strategy: The system gracefully degrades when AI analysis fails
2. Memory-Efficient Diff Processing: Smart handling of large diffs with streaming
3. Comprehensive Configuration: Flexible support for multiple AI providers
4. Excellent Documentation: Both inline and external documentation is thorough

📊 Overall Assessment

APPROVE ✅

This is a well-architected, thoroughly tested feature that follows Ruby best practices and the project's coding standards. The AI test runner adds significant value by optimizing CI performance while maintaining safety through comprehensive fallback mechanisms.

The code demonstrates:

• Strong architectural decisions
• Excellent error handling and resilience
• Good performance considerations
• Comprehensive testing strategy
• Security-conscious implementation

Minor improvements suggested above would enhance the already solid foundation, but they don't block approval of this valuable feature addition.

---

Review generated by Dust AI at 2025-07-08 08:52:59 UTC