Optimize test suite performance: Reduce runtime from ~2 minutes

[kioku]

Problem Statement

The current test suite takes approximately 2 minutes to run, significantly impacting developer productivity and CI/CD pipeline efficiency. This creates friction in the development workflow and slows down feedback loops.

Investigation Summary

Test Suite Statistics

• 359 total tests across the codebase
  • 275 tests in tests/ directory (integration tests)
  • 84 tests in src/ modules (unit tests)
• Current runtime: ~2 minutes
• Target runtime: 45-60 seconds

Key Bottlenecks Identified

1. Binary Compilation Overhead (Major Impact)

   • ~361 invocations of Command::cargo_bin("aperture") across test files
   • Each call potentially triggers binary compilation/linking
   • Highest impact files: integration_tests.rs (112 calls), base_url_integration_tests.rs (48 calls)

2. Mock Server Startup Overhead (Significant Impact)

   • Extensive use of MockServer::start().await in integration tests
   • Network setup/teardown costs for each test
   • Found in 20+ test files

3. File System I/O Overhead (Moderate Impact)

   • Heavy use of TempDir::new() for test isolation
   • Repeated creation/cleanup of temporary directories and files
   • Binary cache file operations in many tests

4. Sleep-Based Tests (Minor but Fixed Impact)

   • 2-second sleep in response_cache_integration_tests.rs:230
   • Fixed delay regardless of system performance

5. Test Structure Issues (Moderate Impact)

   • Some test files have high test counts (e.g., config_manager_tests.rs with 36 tests)
   • Potential for redundant test scenarios

Optimization Tasks

Priority 1 (Critical - Major Impact)

• Implement Binary Caching Strategy

  • Use once_cell or similar to cache compiled binary path
  • Create shared test fixture for common binary invocations
  • Reduce repeated Command::cargo_bin() calls

• Enable Better Parallel Execution

  • Evaluate cargo nextest for improved parallelization
  • Configure optimal --test-threads settings
  • Address any test isolation issues preventing parallelization

Priority 2 (High - Significant Impact)

• Optimize Mock Server Usage

  • Implement mock server pooling/reuse across related tests
  • Share MockServer instances where possible
  • Consider lighter-weight mocking for simple response tests

• Optimize File I/O Operations

  • Use in-memory filesystem where appropriate
  • Share TempDir instances for related test groups
  • Batch file operations to reduce I/O calls

Priority 3 (Medium - Moderate Impact)

• Remove Sleep-Based Tests

  • Replace tokio::time::sleep(Duration::from_secs(2)) in response cache tests
  • Implement event-driven waiting or polling mechanisms
  • Use mock time for TTL-based cache tests

• Consolidate Redundant Tests

  • Review config_manager_tests.rs for potential test reduction/parameterization
  • Combine similar test scenarios across test files
  • Move appropriate unit tests closer to source code

Implementation Notes

Binary Caching Implementation

use once_cell::sync::Lazy;
use std::process::Command;

static APERTURE_BIN: Lazy<std::path::PathBuf> = Lazy::new(|| {
    assert_cmd::cargo::cargo_bin("aperture")
});

// Usage in tests:
Command::new(&*APERTURE_BIN)

Mock Server Pooling

• Create test fixtures that manage server lifecycle
• Use std::sync::Arc for shared server instances
• Implement cleanup in test teardown

Parallel Execution Configuration

• Add nextest.toml configuration
• Set appropriate --test-threads values
• Ensure tests don't conflict on shared resources

Success Criteria

• Test suite runtime reduced to 45-60 seconds (25-50% improvement)
• No test functionality lost during optimization
• Maintained test isolation and reliability
• Improved developer experience with faster feedback loops
• Better CI/CD performance

Related Files

High-Impact Test Files (by Command::cargo_bin usage):

• tests/integration_tests.rs (112 calls)
• tests/base_url_integration_tests.rs (48 calls)
• tests/config_secrets_integration_tests.rs (38 calls)
• tests/config_manager_tests.rs (36 tests total)

Key Implementation Areas:

• Binary caching strategy
• Mock server lifecycle management
• Temporary directory management
• Test parallelization configuration

References

• Current test execution documented at: cargo test (full suite)
• Integration test patterns use assert_cmd and wiremock
• ADR-003 documents test isolation challenges with parallel execution