using pytest fixture and test collection outside of pytest command line

[movadaml]

Hi,
I've a separate test runner orthogonal to pytest (cocotb). But would like to import pytest tests and run them on pytest fixtures.

While there is some effort in to support separate test runners such as pytest in cocotb it's not there yet.

I've created a method to discover and run pytests but it's fragile and not generic. So I'm asking if there are any specific hooks or features in pytest that may improve this.

From my external test runner I can add pytests using the following:

    custom_generate_tests(module.with.pytests, this)

Current solution is based on parsing pytest --collect-only output:

def find_tests(mod, fixture_mod=None):
    ''' Uses pytest to generate a list of tests to run
    The following environment variables are used by pytest and can be used to control which tests are collected:
        PYTEST_ADDOPTS
    '''
    # example return from pytest
    result = '''============================= test session starts ==============================
    platform linux -- Python 3.6.8, pytest-5.4.3, py-1.8.2, pluggy-0.13.1
    plugins: cov-2.10.0
    collected 46 items
    <Module ####.py>
      <Function test_###testname###[###fixture_name###]>
      <Function test_###testname###[###fixture_name###]>
      <Function test_###testname###[###fixture_name###]>
    ============================ no tests ran in 0.01s ============================='''
    if fixture_mod:
        result = subprocess.run(['pytest', os.path.abspath(mod.__file__), fixture_mod.__file__, '--collect-only'], stdout=subprocess.PIPE).stdout.decode('utf-8')
    else:
        result = subprocess.run(['pytest', os.path.abspath(mod.__file__), '--collect-only'], stdout=subprocess.PIPE).stdout.decode('utf-8')
    testlist = re.findall(r' *<Function (.*)\[(.*)\]>', result)
    return testlist

The testlist is used to generate test functions that can be added to the cocotb regression manager:

def generate_tests(dut, test_mod, fixture_mod=None, test_namespace=None):
    ''' Will create cocotb tests for all unittests in test_mod
        if fixture_mod: tests will be generated for fixtures (and their parameters)
        if test_namespace: tests will be created under a different module name'''
    if not test_namespace:
        test_namespace = test_mod

    for name, param in find_tests(test_mod):
        test_name = f"{name}[{param}]"
        test_fn = getattr(test_mod, name)
        documentation = inspect.getdoc(test_fn)

        fn_spec= inspect.getfullargspec(test_fn)
        assert len(fn_spec.args) == 1, 'only one fixture supported'
        fixture_name = fn_spec.args[0]
        fixture_args = param
        fixture = make_fixture(dut, fixture_name, fixture_args, fixture_mod)

        create_test(test_name, documentation, test_namespace, test_fn, fixture)

As I said, this is fragile. The make_fixture() method creates the required fixture from the DUT. The fixtures I support all are known up front and I'm not reusing pytest fixtures - but i'd like to.
eg:

  if fixture_name == "dut":
    return dut
  if fixture_name == "subsystem_name":
    return dut.path.to.subsystem
  if fixture_name == "platform":
    return "simulation"

As you can see, it would be much more powerful to have pytest handle this.
Something like:

SIMHANDLE = None
@pytest.fixture()
def cocotb_simulation():
  return SIMHANDLE 

@pytest.fixture()
def system(cocotb_simulation):
  # override default system fixture with one that has a handle to the simulation
  return cocotb_simulation.dut

@pytest.fixture()
def platform():
  # override default platform fixture
  return "simulation"

# all other fixtures that require access to hardware being simulated by cocotb are derived from system fixture.

@cocotb.coroutine
def runner(dut, test_fn):
    # first argument of runner must be sim handle.
    global SIMHANDLE = dut
    yield with_timeout(cocotb.external(test_fn)(), 50, 'ms')

pytests = cocotb.regression.TestFactory(runner)
pytests.add_option('argv', pytest.get_test_function_list())
pytests.generate_tests()

Could this be done using hooks?

• pytest_runtestloop - use this to add a list of functions to cocotb and exit pytest without running anything - probably doesn't work well for fixtures.
• pytest_runtest_call - call cocotb here, cocotb would see one testcase and report wouldn't be very specific
• pytest_pyfunc_call - call cocotb here, cocotb would see one testcase, pytest report could be used
  Or maybe something using MetaFunc._calls? Or based on show_fixtures_per_test()?

I think I've got some alternatives to try out but would love to hear from the community.

Thanks,
-Adam

[RonnyPfannschmidt]

pytest currently is not designed to be used as a library, so collecting and running tests at best can be a hack, if you don't integrate in the framework

as your surrounding use-case is unclear to me (im not familiar with cocotb) i cant give any further input yet

[movadaml]

Hi @RonnyPfannschmidt,
I agree using pytest as a library does seem unsupported.

Some of the cocotb features that encourage this approach are:

• cocotb is an async coroutine frame work, thus tests written for cocotb are skipped by the pytest. I've not tried pytest-asyncio plugin, nor a similar plugin by beartype yet because this is part of the issue
• i need to call cocotb at the top level because it also builds the hardware that is under test and the simulation process . The python process is launched from within the simulation process and there is a handle to an async c-api here.
• cocotb includes some overrides to python pdb and exceptions to work within this above environment (eg no console). These can conflict with pytests own overrides (i've started investigating pytest options to disable this).

[nicoddemus]

Hi @movadaml,

DIdn't look into the details, but if I were in the same situation I would approach this by calling pytest.main(..., plugins=[cocotb_plugin]), where cocotb_plugin is an instance of class which implements some pytest hooks, which are in turn used to communicate back to the cocotb frame work. This should provide some flexibility plus is the supported way to interact with pytest. Creating the structures and using them directly is not supported, and prone to break between pytest releases.

[movadaml]

Hi @nicoddemus,
This approached turned out to be quite simple to implement. It doesn't support tests or fixtures that are coroutines/async but that is something I can add in the future or workaround using pytest_pyfunc_call and equivalent hook for fixtures.

(note that cocotb async coroutines are not compatible with std python ones)

Conceptually, rather than merging the two frameworks I'm just layering them and treating their results independently (two junit outputs).

cocotbext_pytest.py

import os
import pytest
import cocotb


_SIM_HANDLE = None


@cocotb.test()
def run_pytests(dut):
    """Run all pytests discovered using PYTEST_ADDOPTS"""
    global _SIM_HANDLE
    _SIM_HANDLE = dut
    yield cocotb.external(pytest.main)([
        "-p", "cocotbext_pytest",
        "--pdb" if os.environ.get("COCOTB_PDB_ON_EXCEPTION") else "",
        "-rA",
        "--junit-xml=pytest/" + os.environ.get("COCOTB_RESULTS_FILE", "results.xml"),
        "--capture=tee-sys",
        ])


@pytest.fixture(scope="session")
def sim_handle():
    return _SIM_HANDLE