debug windows stdio test flakiness #1158
Closed
+778
−0
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add PowerShell scripts to help debug test_stdio_context_manager_exiting flakiness on Windows with Python 3.11/3.12: - check-tee-command.ps1: Verify tee command availability - test-stdio-flakiness-200-runs.ps1: Run test 200 times for statistics - test-stdio-flakiness-until-failure.ps1: Run until failure occurs - test-stdio-verbose-debug.ps1: Single run with maximum debug output - README.md: Documentation for using the scripts
Add -n 0 flag to disable xdist parallel execution which was causing test collection failures. Also fix successful run count calculation.
- Use PYTEST_DISABLE_PLUGIN_AUTOLOAD environment variable (recommended in CLAUDE.md) - Use -p no:xdist flag to explicitly disable the plugin - Add --no-cov to avoid coverage plugin issues - Add test-stdio-simple.ps1 to try different methods of disabling xdist
- Add setup-environment.ps1 to configure Git for Windows tee in PATH - Simplify all scripts to use working -o addopts="" approach - Add tee availability checks to all scripts - Update README with clear setup instructions and troubleshooting - Remove complex xdist workarounds in favor of simple override
Based on deep analysis, the flakiness likely occurs due to: 1. Job Object handle race conditions when tests run in parallel 2. Windows handle inheritance between test processes 3. Timing sensitivity in the minimal test that just opens/closes immediately 4. Python 3.11/3.12 specific subprocess handling changes These scripts help reproduce the CI environment conditions: - test-stdio-parallel-flakiness.ps1: Runs with xdist parallel workers - test-stdio-stress-race.ps1: Rapidly creates processes to expose races
This script collects detailed information about: - Windows version and build - Python version and subprocess configuration - Job Objects support - Security software - Process creation timing - Handle inheritance settings This will help identify differences between local dev environments (where the test passes) and CI environments (where it flakes).
felixweinberger
force-pushed
the
debug-windows-stdio-test-flakiness
branch
4 times, most recently
from
aa7c2cc to
397303e
Compare
felixweinberger
force-pushed
the
debug-windows-stdio-test-flakiness
branch
from
397303e to
6be585d
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Motivation and Context
How Has This Been Tested?
Breaking Changes
Types of changes
Checklist
Additional context