Skip to content

Commit fe5e89f

Browse files
mickael-palma-wttjmickael-palma-wttj
authored
feat: Add AI-powered Smart Test Runner system (#10)
* feat: Add AI-powered Smart Test Runner system 🤖 Implements intelligent test selection based on code changes using AI Features: - AI-powered analysis of code changes to select relevant tests - Support for both direct and indirect test dependencies - Configurable AI providers (Anthropic Claude, Dust) - Smart fallback to full test suite if AI analysis fails - Comprehensive reporting and analysis output Components: - SmartTestRunner: Main orchestrator class - GitChangeAnalyzer: Analyzes git diffs and changed files - TestDiscoveryService: Maps source files to test files - AITestSelector: Uses AI to intelligently select tests - External prompt template for easy maintenance Files added: - .github/workflows/smart_tests.yml: GitHub Action workflow - .github/scripts/smart_test_runner.rb: Main Ruby implementation - .github/scripts/smart_test_selection_prompt.md: AI prompt template - spec/github/scripts/smart_test_runner_spec.rb: Comprehensive test suite - scripts/test_smart_runner.rb: Local testing script - docs/SMART_TEST_RUNNER.md: Complete documentation Benefits: - 50-80% reduction in test execution time for small changes - Significant CI cost savings from reduced compute time - Maintains comprehensive test coverage with intelligent selection - Ruby-specific optimizations for RSpec and Rails conventions * fix: Update upload-artifact action from v3 to v4 - Fixes deprecation warning for actions/upload-artifact@v3 - Updates to the current supported version @v4 - Maintains same functionality for test results artifact upload * refactor: Extract shared AI services module to eliminate code duplication - Create shared AI services module with HTTPClient, AIProvider interfaces, and factory - Extract AnthropicProvider and DustProvider from duplicated code - Refactor Smart Test Runner to use shared AI services - Update tests and local runner to use shared services - Reduce codebase by ~150 lines of duplicate AI service code - Improve maintainability and error handling consistency This consolidation allows both Smart Test Runner and PR Review to use the same robust, well-tested AI service implementations. * fix: Handle AI provider creation failure in SmartTestRunner spec * feat: Enhance Dust conversation ID visibility in log output - Upgrade conversation creation and response logging from debug to info level - Add conversation ID to all retry attempts and error messages - Include conversation ID in delay and timeout messages - Add emojis for better log readability and distinction - Ensure conversation ID is always visible for debugging and monitoring This improves troubleshooting by making Dust conversation IDs prominent in CI/CD logs without requiring debug mode activation. * Complete PR Review refactoring to use shared AI services - Removed all duplicate AnthropicProvider, DustProvider, HTTPClient, AIProviderFactory, and LoggerFactory classes from pr_review.rb - Updated PullRequestReviewer to use SharedLoggerFactory and AIProviderFactory from shared services - Fixed PR Review spec tests to use correct provider constructor signatures and method names - Updated test expectations to match shared provider interface (make_request method) - Added create_github_client method to PullRequestReviewer class to replace GitHubClientFactory - All 684 tests now pass with zero failures and zero warnings - All duplicate class definitions have been eliminated, resolving constant redefinition warnings * chore: Remove backup script for PR review process * Fix PR Review method name: use make_request instead of request_review - Updated PullRequestReviewer#run to call ai_provider.make_request(prompt) instead of ai_provider.request_review(prompt) - Updated PR Review test to expect make_request method call - All 684 tests still pass with zero failures - PR Review script now works correctly with shared AI services * fix: Update Dust conversation URI format for consistency * refactor: rename SmartTestRunner to AITestRunner and update file paths - Rename SmartTestRunner class to AITestRunner for consistency - Rename SmartTestConfig to AITestConfig - Update main script from smart_test_runner.rb to ai_test_runner.rb - Update prompt file from smart_test_selection_prompt.md to ai_test_selection_prompt.md - Move documentation from docs/SMART_TEST_RUNNER.md to doc/SMART_TEST_RUNNER.md - Update all references in code, specs, workflows, and documentation - Centralize AI provider logic in shared/ai_services.rb - Implement streaming diff parsing for large diffs - Optimize git diff commands for performance - Mock sleep in tests to improve test suite speed - All 684 tests pass in ~1.1 seconds This completes the refactoring for better maintainability, performance, and consistent naming convention throughout the AI test runner system. * feat: implement exponential backoff and rate limiting for AI API calls Major improvements to API reliability and performance: ## Rate Limiting & Retry Logic - Add RetryHandler module with exponential backoff algorithm - Implement configurable retry parameters (max_retries, base_delay, max_delay) - Add 10% jitter to prevent thundering herd problems - Support Retry-After header extraction for rate limit compliance - Handle different HTTP error codes appropriately (429, 5xx vs 4xx) ## Memory Management Improvements - Replace magic numbers with named constants (LARGE_DIFF_THRESHOLD, etc.) - Convert all diff processing from split('\n') to StringIO streaming - Add proper resource cleanup with ensure blocks - Prevent memory spikes on large diffs with consistent streaming approach ## API Provider Enhancements - HTTPClient: Smart retry logic for rate limits and server errors - DustProvider: Exponential backoff for conversation polling - AnthropicProvider: Benefits from HTTPClient retry improvements - Comprehensive error logging with retry attempt visibility ## Testing & Documentation - Update test expectations for new retry behavior - Mock sleep calls to maintain fast test execution (0.15s for 50 tests) - Rename documentation file to AI_TEST_RUNNER.md for consistency - Update all references to use proper 'AI Test Runner' naming ## Performance Benefits - Graceful handling of temporary API failures - Reduced failure rates through intelligent backoff - Optimal retry timing (1s → 2s → 4s → 8s → 16s, capped at 30s) - Fast recovery for transient issues All 50 tests pass. No breaking changes to existing functionality. * refactor: improve code structure by extracting large class components Break down GitChangeAnalyzer class complexity by extracting responsibilities: ## Class Size Reduction - GitChangeAnalyzer: 200+ lines → ~88 lines (56% reduction) - Single Responsibility Principle: Each class now has one clear purpose ## Extracted Components - DiffProcessingConfig: Centralized configuration with documented rationale - DiffParser: Handles all diff parsing logic with memory-efficient streaming - FileTypeClassifier: Dedicated file type detection service ## Magic Numbers → Configuration - Made constants configurable via environment variables - Added comprehensive documentation explaining threshold choices - LARGE_DIFF_THRESHOLD: 10MB (git performance limit) - MEMORY_DIFF_THRESHOLD: 1MB (streaming mode trigger) - MAX_STREAMING_LINES: 100 (memory vs context balance) ## Benefits - Improved maintainability with smaller, focused classes - Better testability through separation of concerns - Configurable thresholds for different environments - Clear documentation of design decisions - Preserved existing functionality and API All 50 tests pass. Code is now more modular and follows SOLID principles.
1 parent b8b32e6 commit fe5e89f

File tree

10 files changed

+2158
-354
lines changed

10 files changed

+2158
-354
lines changed

‎.github/scripts/ai_test_runner.rb‎

Lines changed: 716 additions & 0 deletions
Large diffs are not rendered by default.

‎.github/scripts/ai_test_selection_prompt.md‎

Lines changed: 78 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,78 @@
1+
# Smart Test Selection AI Prompt
2+
3+
You are an expert Ruby developer analyzing code changes to determine which tests should be run.
4+
5+
## CODE CHANGES ANALYSIS
6+
The following files have been changed:
7+
{{changed_files_summary}}
8+
9+
## CHANGE DETAILS
10+
```diff
11+
{{diff_content}}
12+
```
13+
14+
## AVAILABLE TEST FILES
15+
{{test_files_list}}
16+
17+
## TEST-TO-SOURCE MAPPING
18+
{{test_mapping}}
19+
20+
## INSTRUCTIONS
21+
Analyze the changes and select tests that should be run. Consider:
22+
23+
1. **Direct tests**: Tests that directly test the changed source files
24+
2. **Indirect tests**: Tests that might be affected by the changes through:
25+
- Inheritance or module inclusion
26+
- Dependency injection
27+
- Shared interfaces or contracts
28+
- Integration points
29+
- Configuration changes
30+
31+
3. **Risk assessment**: Consider the impact of changes:
32+
- Public API changes → Run more tests
33+
- Internal implementation → Focus on direct tests
34+
- Breaking changes → Run comprehensive tests
35+
36+
## OUTPUT FORMAT
37+
Respond with a JSON object in this exact format:
38+
```json
39+
{
40+
"selected_tests": [
41+
"spec/lib/kanban_metrics/example_spec.rb"
42+
],
43+
"reasoning": {
44+
"direct_tests": ["list of tests that directly test changed files"],
45+
"indirect_tests": ["list of tests that might be indirectly affected"],
46+
"risk_level": "low|medium|high",
47+
"explanation": "Detailed explanation of selection reasoning"
48+
}
49+
}
50+
```
51+
52+
Select tests intelligently - don't run everything, but don't miss important dependencies.
53+
54+
## RUBY-SPECIFIC CONSIDERATIONS
55+
56+
### File Mapping Conventions
57+
- Source files in `lib/` correspond to test files in `spec/`
58+
- `lib/foo/bar.rb` → `spec/lib/foo/bar_spec.rb`
59+
- Class names follow CamelCase → snake_case conversion
60+
61+
### Test Dependencies to Consider
62+
- **Module Inclusion**: Changes to modules affect all classes that include them
63+
- **Inheritance**: Changes to parent classes affect all subclasses
64+
- **Shared Examples**: Changes to shared examples affect all tests that use them
65+
- **Factory Dependencies**: Changes to factories affect tests that use them
66+
- **Configuration**: Changes to initializers, configs affect integration tests
67+
68+
### Risk Level Guidelines
69+
- **Low Risk**: Private method changes, internal refactoring, documentation
70+
- **Medium Risk**: Public method signature changes, new public methods, class structure changes
71+
- **High Risk**: Interface changes, breaking changes, dependency updates, configuration changes
72+
73+
### Test Selection Strategy
74+
- Always include direct tests for changed source files
75+
- For module/class changes, include tests for dependent classes
76+
- For configuration changes, include integration and system tests
77+
- For Gemfile/dependency changes, consider running full suite
78+
- For test-only changes, run the changed tests plus any that depend on shared examples/helpers

0 commit comments

Comments
 (0)