Skip to content

cucumberjson compatibility with pytest-timeout #770

@jpwright

Description

@jpwright

This isn't truly an issue with pytest-bdd, but I thought this was worth mentioning given the popularity of the pytest-timeout plugin.

I noticed that if the pytest-timeout plugin is used, it can correctly fail a scenario, but all steps show as passed in the cucumberjson output. A minimal example:

test_timeout.py:

from pytest_bdd import scenario, when
import time

@scenario('timeout.feature', 'Long running operation')
def test_long_running():
    pass

@when('I wait for a really long time')
def wait_long_time():
    time.sleep(3)

timeout.feature:

Feature: Timeout demonstration
    Demonstration of a long-running operation

    Scenario: Long running operation
        When I wait for a really long time

then run pytest:

pytest -s tests/test_timeout.py --cucumberjson=output.json --timeout 1

to get the following logs:

=================================================================================================================================================== test session starts ====================================================================================================================================================
platform linux -- Python 3.12.7, pytest-8.3.5, pluggy-1.5.0
rootdir: /media/jason/e/pytest-bdd
configfile: pytest.ini
plugins: bdd-8.1.0, timeout-2.3.1
timeout: 1.0s
timeout method: signal
timeout func_only: False
collected 1 item                                                                                                                                                                                                                                                                                                           

tests/test_timeout.py OUTCOME: failed
F

========================================================================================================================================================= FAILURES =========================================================================================================================================================
____________________________________________________________________________________________________________________________________________________ test_long_running _____________________________________________________________________________________________________________________________________________________

fixturefunc = <function wait_long_time at 0x7130aa194360>, request = <FixtureRequest for <Function test_long_running>>, kwargs = {}

    def call_fixture_func(
        fixturefunc: _FixtureFunc[FixtureValue], request: FixtureRequest, kwargs
    ) -> FixtureValue:
        if is_generator(fixturefunc):
            fixturefunc = cast(
                Callable[..., Generator[FixtureValue, None, None]], fixturefunc
            )
            generator = fixturefunc(**kwargs)
            try:
                fixture_result = next(generator)
            except StopIteration:
                raise ValueError(f"{request.fixturename} did not yield a value") from None
            finalizer = functools.partial(_teardown_yield_fixture, fixturefunc, generator)
            request.addfinalizer(finalizer)
        else:
            fixturefunc = cast(Callable[..., FixtureValue], fixturefunc)
>           fixture_result = fixturefunc(**kwargs)

venv/lib/python3.12/site-packages/_pytest/fixtures.py:898: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

    @when('I wait for a really long time')
    def wait_long_time():
>       time.sleep(3)
E       Failed: Timeout >1.0s

tests/test_timeout.py:10: Failed
------------------------------------------------------------------------------------------------------------------------- generated json file: /media/jason/e/pytest-bdd/output.json -------------------------------------------------------------------------------------------------------------------------
================================================================================================================================================= short test summary info ==================================================================================================================================================
FAILED tests/test_timeout.py::test_long_running - Failed: Timeout >1.0s

but the following output.json shows the step passed:

[
  {
    "keyword": "Feature",
    "uri": "tests/timeout.feature",
    "name": "Timeout demonstration",
    "id": "tests/timeout.feature",
    "line": 1,
    "description": "Demonstration of a long-running operation",
    "language": "en",
    "tags": [],
    "elements": [
      {
        "keyword": "Scenario",
        "id": "test_long_running",
        "name": "Long running operation",
        "line": 4,
        "description": "",
        "tags": [],
        "type": "scenario",
        "steps": [
          {
            "keyword": "When",
            "name": "I wait for a really long time",
            "line": 5,
            "match": {
              "location": ""
            },
            "result": {
              "status": "passed",
              "duration": 0
            }
          }
        ]
      }
    ]
  }
]

pytest==8.3.5
pytest-bdd==8.1.0
pytest-timeout==2.3.1

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions