How to collect test nodes/names, test outcomes and their associated markers?

[matijadurdek]

Hi, I would need some help :)

I want to collect 3 types of data from a test run:

• test method name/node
• test outcome
• the arguments of a specific custom marker associated with the test method

Is it possible to collect the custom markers associated to test methods/nodes and the corresponding test outcomes?

Here is what I currently have in my setup:

pytest.ini:

markers =
    mycustommark: this is my custom marker

test examples:

@pytest.mark.mycustommark('TEST-1')
def test_1():
    pass


@pytest.mark.mycustommark('TEST-2')
def test_2():
    pass

conftest.py:

def pytest_runtest_logreport(report: TestReport):
    """Collect test nodes, test outcomes and marker values"""
    if report.when == "call" and 'mycustommarker' in report.keywords.keys() and report.keywords['mycustommarker']:
        print("Test node: ", report.nodeid)
        print("Marker: ", report.keywords.get('mycustommarker'))
        print("Test outcome: ", report.outcome)

I tried to use the pytest_runtest_logreport hook and it gives me the expected nodes and their outcomes in the output, but I'm not able to fetch the argument values from the mycustommark markers. It returns an integer value:

Test node:  tests/test_example.py::TestExample::test_1
Marker:  1
Test outcome:  passed

Test node:  tests/test_example.py::TestExample::test_2
Marker:  1
Test outcome:  passed

The expected output that I want:

Test node:  tests/test_example.py::TestExample::test_1
Marker:  TEST-1
Test outcome:  passed

Test node:  tests/test_example.py::TestExample::test_2
Marker:  TEST-2
Test outcome:  passed

[EmptyRabbit]

Try using pytest_runtest_makereport hook which will be called before pytest_runtest_logreport , can get the expected output. TestReport object seems not contain mark details

@pytest.hookimpl(tryfirst=True, hookwrapper=True)
def pytest_runtest_makereport(item, call):
    outcome = yield
    rep = outcome.get_result()

    target_mark = next(filter(lambda x: x.name == 'mycustommark', item.own_markers), None)
    if call.when == 'call' and target_mark:
        print('\n')
        print("Test node: ", item.nodeid)
        print("Marker: ", target_mark.args[0])
        print("Test outcome: ", rep.outcome)

[matijadurdek]

First of all, thank you for the quick response and your help. I'm replying a little late as I was trying out some approaches and your solution.

TestReport object seems not contain mark details

Yes, you are right. TestReport does not contain markers, but I found a workaround on how to add them to TestReport. TestReport contains the user_properties attribute, and I used the pytest_collection_modifyitems hook to add markers to test items by appending them as user properties. Here is the code and I gave it a real-world example for more clarification to avoid using generic names:

def pytest_collection_modifyitems(session: pytest.Session, config: Config, items: List[Item]):
    """Add 'xray' marker as user property to test items"""
    for item in items:
        for mark in item.iter_markers("xray"):
            xray_test_key = mark.args[0]
            item.user_properties.append(("testKey", xray_test_key))

def pytest_runtest_logreport(report: TestReport):
    """Print test outcomes, test names and markers/test keys"""
    if report.when == "call" and 'xray' in report.keywords.keys() and report.keywords['xray']:
        xray_marker_value = report.user_properties[0][1]
        print("Test node/name: ", report.nodeid)
        print("Marker / Xray test keys: ", xray_marker_value)
        print("Test outcome: ", report.outcome)

I tried your solution and it does the trick. And it is simpler. So, thanks for that. 😄

But, I still have a problem with this which I'm not able to solve. I'm not able to get correct test outcomes for skipped and error tests, or to be precise - I'm not able to get the outcome for the whole test.
When I'm using report.when == "call" in the if-clause, I only get the correct outcomes for passed and failed tests, but I don't get them for skipped (as only the "setup" call is run and has the skipped status, while the "call" call is not run in case of skipped tests) and error tests.
If I remove report.when == "call" and only leave if 'xray' in report.keywords.keys() and report.keywords['xray']:, then I get the results for all 3 calls (setup, call, tearDown), but I don't need all 3 call results. What I need is the "final" test result/outcome for each test that was executed (like e.g. pytest prints the test results to the terminal for each test after it has been run - it only prints the "final" outcome, not for all 3 calls separately).

Is there a simple way how to get the "final" outcome of each test (passed, failed (and error - that can be interpreted as failure), skipped) and not just the outcome of individual calls? How would I have to modify your/my code to achieve that? 😄

[EmptyRabbit]

Sorry for late reply. user_properties attribute is useful, learned something new. I searched the document and tried to find a way get "final" test result in pytest_runtest_logreport, but failed.

I looked at "allure-python" code and tried implement your needs in the similar way. Hope it's useful for you.

@pytest.hookimpl(tryfirst=True, hookwrapper=True)
def pytest_runtest_makereport(item, call):
    """ Set final result 'item_result' in each item """
    outcome = yield
    rep = outcome.get_result()

    target_mark = next(filter(lambda x: x.name == "xray", item.own_markers), None)

    if target_mark:
        attr_key = "item_result"
        status = rep.outcome

        if rep.when in ("setup", "teardown") and status == "failed":
            status = "error"
        if hasattr(rep, "wasxfail"):
            status = "xfailed" if rep.skipped else "xpass"

        historic_status = getattr(item, attr_key, status)
        if rep.when == "setup":
            setattr(item, attr_key, status)

        if rep.when == "call":
            if historic_status == "passed":
                setattr(item, attr_key, status)

        if rep.when == "teardown":
            if historic_status == "passed":
                setattr(item, attr_key, status)


def pytest_sessionfinish(session, exitstatus):
    """ Show the final result """
    for item in session.items:
        for mark in item.iter_markers("xray"):
            print("\n")
            print("Test node: ", item.nodeid)
            print("Marker: ", mark.args[0])
            print("Test outcome: ", item.item_result)

[matijadurdek]

@EmptyRabbit, thank you very much for all the help and thorough work. Sorry for the late reply, it has been a couple of busy weeks.

The if statements for collecting test outcomes are very detailed in your solution and cover various cases. Thank you for that. I'll try it out as soon as I can. I will also try out the pytest_runtest_logreport hook as well with your approach, together with running tests using the pytest-xdist plugin for parallel test execution to check if the "final" test results and their markers are collected correctly (1 test item = 1 outcome). I'll get back to you with the feedback regarding this. 😄