How to add pytest collection errors to allure report

[cml8ds]

I run my test suite using pytest --continue-on-collection-errors option.

I don't see the failing tests at collection step in the report and it seems that allure-pytest is not creating any json files in alluredir for these tests.

Is there a way to change this behavior?

[delatrie]

Hi, @cml8ds

Allure-pytest follows the lifecycle of pytest and pytest doesn't run tests from modules with collection errors: if no tests are collected, there is nothing to run, hence, no results.

The best you can do is to create surrogate test results to show collection errors in the report. Something like this should do the trick:

import pytest
from allure_commons import plugin_manager
from allure_commons.model2 import Label
from allure_commons.model2 import Status
from allure_commons.model2 import StatusDetails
from allure_commons.model2 import TestResult
from allure_commons.utils import host_tag
from allure_commons.utils import md5
from allure_commons.utils import now
from allure_commons.utils import platform_label
from allure_commons.utils import thread_tag
from allure_commons.utils import uuid4
from allure_commons.types import LabelType
from allure_pytest.utils import allure_package

class ItemStub: # A kinda hacky way to reuse the logic from allure_pytest.utils.allure_package
    def __init__(self, nodeid):
        self.nodeid = nodeid


def pytest_collectreport(report: pytest.CollectReport):
    if report.failed:
        fullName = f"collection-failure:{report.nodeid}"
        testCaseId = md5(fullName)
        timestamp = now()

        result = TestResult(
            name=f"Collection failed for {report.fspath}",
            status=Status.FAILED,
            statusDetails=StatusDetails(
                message=report.longreprtext.splitlines()[-1],
                trace=report.longreprtext,
            ),
            start=timestamp,
            stop=timestamp,

            uuid=uuid4(),
            fullName=fullName,
            testCaseId=testCaseId,
            historyId=testCaseId,
            labels=[
                Label(name=LabelType.LANGUAGE, value=platform_label()),
                Label(name=LabelType.FRAMEWORK, value="pytest"),
                Label(name=LabelType.HOST, value=host_tag()),
                Label(name=LabelType.THREAD, value=thread_tag()),
                Label(name="package", value=allure_package(ItemStub(report.nodeid))),
            ],
        )

        plugin_manager.hook.report_result(result=result) # this will write the test result on disk if Allure is enabled

It's reasonable to implement something like that out-of-the-box though. We do something similar to report before feature hook failures in some cucumber-based frameworks, so why not do this for collection errors in pytest? Additionally, that's what pytest does when generating a JUnit XML report with --junit-xml=test-results.xml. I will create a feature request based on this question.

UPDATE: Here is the issue to track: allure-framework/allure-python#859.

[cml8ds]

Hi @delatrie,

Thank you for your response, it works like a charm!