Implement a new --failing-and-slow-first command line argument to test runner. #24624
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…t runner. This keeps track of results of previous test run, and on subsequent runs, failing tests are run first, then skipped tests, and last, successful tests in slowest-first order. Add support for --failfast in the multithreaded test suite. This improves parallelism throughput of the suite, and helps stop at test failures quickly.
sbc100
reviewed
Jul 7, 2025
| @@ -35,3 +35,6 @@ coverage.xml | |||
| # ...except the templates. | |||
| !/tools/run_python.ps1 | |||
| !/tools/run_python_compiler.ps1 | |||
|
|
|||
| # Test runner previous run results for sorting the next run | |||
| __previous_test_run_results.json | |||
There was a problem hiding this comment.
This should probably go in out/ along with existing last_test.txt file (used to implement --continue)
sbc100
reviewed
Jul 7, 2025
There was a problem hiding this comment.
Actually maybe I misunderstood. I use --failfast --continue when implementing new features and wanting to fix each test failure as I run into it.
How does this improve CI times on the bots? It seems like it would not effect the first run, but only subsequent runs, which the bots don't do, do they?
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This keeps track of results of previous test run, and on subsequent runs, failing tests are run first, then skipped tests, and last, successful tests in slowest-first order. This improves parallelism throughput of the suite.
Add support for --failfast in the multithreaded test suite to help stop suite runs at first test failures quickly.
These two flags
--failfastand--failing-and-slow-firsttogether can help achieve < 10 second test suite runs on a CI when the suite is failing.Example
core0runtime withtest/runner core0on a 16-core/32-thread system:Same suite runtime with
test/runner --failing-and-slow-first core0:Gaining a better throughput and a -20.37% test suite wall time.