Supporting flaky tests - just thinking out loud here

[jeffwright13]

I am aware of the Pytest plugins for flakiness (pytest-flaky, and pytest-rerun). Those are great if I insist on a rigid determination of P or F for each test for a given run. But what would also be valuable is the ability to actually mark a test as Flaky (or maybe Indeterminate). Once marked as such, the test's outcome would occupy a new category, Flaky or Indeterminate or whatever it needs to be called. Tests marked Flaky would not count toward P or F or anything else; they would instead occupy their own category. And maybe they could be notated on the command line as a snowflake:

.F..❅.s...F....sXxFF....❅..

A good tester using this feature would always strive to improve their tests and systems-under-test so that the number of Flaky category tests is zero...but knowing that in the real world, having a zero-flake test run is something of a unicorn. :-)

I am guessing this would require work in the guts of Pytest, and not be available for someone to develop as a plugin, but I'm not sure.

What do you think? Is this a terrible idea? Would it be something the devs could do, or is it something a plugin might be able to implement?

[The-Compiler]

Definitely something a plugin could do, via the pytest_report_teststatus hook.

[jeffwright13]

OK, well maybe this will be my next project. I had better study the list of existing plugins, though...there seem to be similar plugins. For example, https://pypi.org/project/pytest-ignore-flaky/.

Id' welcome any one else's opinions, too.