Result Outcomes Regarding Skipped Tests

[eleanorjboyd]

Hello! I am working on gathering the results of tests so I can display them to the users. I currently have my plugin configured to hook into pytest via pytest_runtest_logreport(report) but am running into an issue with skip tests. I am have two configurations of skip tests shown below. I wanted to find a consistent way to report which tests were skipped. I know since these tests are skipped then report.when == "call" will not be called instead just report.when == "setup" and "teardown". I was observing the values of skipped tests in these two states and got a bit confused. I am seeing that during "setup" the value of report.skipped == true and report.outcome == skipped but then during "takedown" the values are report.skipped == false and report.outcome == passed. What is the reasoning behind this? If I want a consistent place to mark each test with the following literal outcome: Literal["success", "failure", "skipped"] what is the best pytest hook and report.when to do this at?

Thank you!

[Karamann]

You might wanna look at the pytest_runtest_protocol hook.
This hook controls the entire test protocol (setup, call, teardown) and it gives you access to the report for each phase.
You can check the reports for each phase to determine the outcome of the test and report it accordingly.

ie:

def pytest_runtest_protocol(item, nextitem):
    reports = []
    for report in nextitem.ihook.pytest_runtest_protocol(item=item, nextitem=nextitem):
        reports.append(report)
    if any(report.skipped for report in reports):
        test_outcome = "skipped"
    elif any(report.failed for report in reports):
        test_outcome = "failure"
    else:
        test_outcome = "success"
    # Now you can report the outcome of the test.
    return reports