Proposal: Track flaky tests as bugs #6595
flaneur2020
started this conversation in
Ideas
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
Background
the last 0.7.126-nightly build did not actually pass the sqlogictest.
currently we did not forced all test to pass before merging, this is mostly due to there're some flaky tests exists, we still can merge the Pull Request as we found some test failures are not caused by my pull request.
flaky tests harms our confidence about automated tests, if the wolf really comes (like this time), we'd likely to dis-trust the alert, and causes real broken build.
Proposal
maybe we can deal with the flaky test as the following procedure:
so we can force each pull request MUST pass all the tests before merging, and build a trust with the CI.
Beta Was this translation helpful? Give feedback.
All reactions