Plugin performing automatic setup - best practices?

[LewisGaul]

I have put together a MWE gist at https://gist.github.com/LewisGaul/126049eec2911a5d557285cff0e3799e to illustrate the behaviour in question.

To summarise:

• Pytest plugin that provides a --setup-vm arg, which will boot a VM before any other fixtures are run
• Tests that are written to work with or without this plugin (or a different drop-in plugin), i.e. not declaring a plugin fixture as a dependency
• The tests check for a host_address config option when determining which host to use, defaulting to localhost
• The plugin therefore sets host_address to correspond to the VM that it's booting, and the tests' conftest doesn't need to care about this happening

Problems/concerns/questions:

• How/when should the plugin boot the VM?
  • The mechanism in the example is inserting a session-scoped fixture into position 0 of each item's fixturenames in the pytest_collection_modifyitems() hook. However, this is not a documented mechanism... This approach was required since a simple autouse fixture would run after autouse fixtures declared in the tests' conftest.py, which is too late.
  • Could do it in pytest_configure() in the plugin, but booting a VM is setup, not configuration, and this would make running with --collect-only slow/fragile. On the other hand, this would allow checking host features earlier in the flow, which would make skipping tests cleaner.
• Is skipping tests with an autouse fixture (_skip_marked_tests_as_applicable in the example) a reasonable approach?
  • The reason for doing it this way is we want to skip based on properties of the host, i.e. depending on the host fixture.
  • If the host was set up earlier and accessible some other way (e.g. on the pytestconfig object) then we could skip in the pytest_collection_modifyitems() hook instead, which would probably be more normal(?).

Does anyone have any thoughts on the approach taken here, or suggestions for how to improve things? Any best practices I've missed?

[LewisGaul]

Uncovered flaw

A problem with this approach has been uncovered where the first testcase cannot be skipped using item.add_marker(pytest.mark.skip(...)) from a fixture. This makes sense after thinking it through - fixtures are setup functions associated with testcases, so at the point the first fixture runs (even if autouse) the first testcase has already begun (the fixture being run is in some way part of its setup), and it's too late to skip using the 'skip' marker.

Possible workarounds

Dummy testcase

One possible approach would be to always create a dummy testcase, inserting it at the beginning of items in the pytest_collection_modifyitems() hook, such that session-scoped autouse fixtures are associated with this testcase, meaning the _skip_marked_tests_as_applicable() fixture can run before any real testcases, skipping via the 'skip' mark as required.

The downside of this is that there would always be a dummy testcase showing up in the results... Perhaps it can be filtered out from results using more pytest hooks, but this all starts to feel quite hacky.

Decorate testcases

Instead of marking testcases with their required features (e.g. @pytest.mark.require_features("foo") in the gist) a decorator could be used to perform the check and call pytest.skip(...) if required, perhaps @require_features_else_skip("foo"). The key difference is that this decorator would allow running code "inside the test function body", allowing skipping a testcase after it's already been started (using pytest.skip() rather than pytest.mark.skip). This decorator can also apply the original pytest mark if desired.

One apparent problem with this approach is that the decorator needs access to the resources required to perform the feature check, encapsulated in the host fixture in the example gist. I assumed this would mean that every testcase would need to explicitly declare it as a dependency, however, it turns out it's the wrapper function defined in the decorator who's arguments matter to pytest (I assume pytest uses inspect.getfullargspec(), so this makes sense).

This means that the decorator can define the testcase wrapper function to depend on the host fixture and this is respected by pytest! ... except passing through *args and **kwargs (as is common practice in decorators) loses the information about which fixtures the testcase does explicitly depend on :( As far as I can tell, this means the decorator would have to dynamically construct a function with the same argspec as the original testcase function (possibly also adding in the host dependency) so that the information about which fixtures are requested is not lost. Unfortunately, dynamically constructing functions seems quite complicated (e.g. see https://stackoverflow.com/questions/11291242/python-dynamically-create-function-at-runtime), and this also starts to feel very hacky.

Conclusion

I'm increasingly feeling that this plugin setup should just be done in the pytest_configure() hook. This would allow skipping/removing tests in the pytest_collection_modifyitems() hook. This also removes the need for the use of the undocumented fixturenames to ensure the fixture runs first.

My previous concern about doing this in pytest_configure() rather than a fixture was:

booting a VM is setup, not configuration, and this would make running with --collect-only slow/fragile

I still feel a bit uneasy about this... However, the fact that all tests will run in the context of the host (e.g. the VM) it seems less of an obvious fit with fixtures. We could make it so that if --collect-only is given then the VM booting is skipped, or perhaps it's a good thing that we'd be able to report what would be skipped based on the provided host's features.

I'm still hopeful of some thoughts from someone with more familiarity with pytest than me!

[LewisGaul]

Oh, the other reason the fixture approach was preferable is there's cleanup required, which is neatly supported using yield in fixtures... Since there don't seem to be any explicit global shutdown hooks (which is fair, since pytest_configure() isn't really a setup hook!) perhaps a better approach would be to add the setup and teardown in the pytest_runtestloop() hook... ?

[LewisGaul]

The problem with implementing setup and teardown in pytest_runtestloop(), which is a "stop at first non-None result" hook, is that the default implementation must be copied (17 lines, so non-trivial) since otherwise there's no way to insert the teardown. What's more, this would then mean the plugin is incompatible with another plugin that reimplements this hook (i.e. returns non-None).

Please let me know if I'm misunderstanding something here!

[LewisGaul]

I've realised @pytest.hookimpl(hookwrapper=True) should solve the problems described above with using the pytest_runloop() hook. I've also discovered the pytest_sessionstart() and pytest_sessionfinish() hooks (that run before and after the runloop) which would be an alternative place to do this. Unsure which is preferable - I quite like the setup and cleanup being in one function, separated by yield.

[nicoddemus]

I was about to suggest that you look into pytest_sessionstart() and pytest_sessionfinish().

Using a hookwrapper around pytest_runtestloop() is also a possibility, don't see any drawbacks at first glance.

About skipping: you might consider using an autouse fixture which calls pytest.skip() if a certain test needs to be skipped; that fixture can lookup at some information set to pytest.Config during pytest_sessionstart (or the setup part of the hookwrapper around pytest_runtestloop()).

[LewisGaul]

pytest_sessionstart() seems to run before collection, which would allow adding the skip marker in pytest_collection_modifyitems(), which I assume would be more efficient than have a fixture run for every testcase to check whether it needs to be skipped? Thanks for reminding me of this option though.

[LewisGaul]

@nicoddemus From your comment above, can I take it implicitly from absence of disagreement that you agree with the following conclusions I'm arriving at?

• It's correct/recommended for plugin setup to be performed via pytest_sessionstart() (or pytest_runtestloop()) rather than an autouse fixture in the case where it must run before all other fixtures.
• Skipping tests from an autouse session fixture doesn't really work because the first test cannot be skipped - alternatives include using pytest_collection_modifyitems() and a per-test autouse fixture.

[nicoddemus]

It's correct/recommended for plugin setup to be performed via pytest_sessionstart() (or pytest_runtestloop()) rather than an autouse fixture in the case where it must run before all other fixtures.

Yes, I would go with that approach.

Skipping tests from an autouse session fixture doesn't really work because the first test cannot be skipped

Hmmm it certainly can:

import pytest

@pytest.fixture(autouse=True)
def fix():
    pytest.skip("skipped for some reason")

def test_a(): pass
def test_b(): pass
def test_c(): pass

 λ pytest .tmp\test-auto.py --no-header -ra
======================== test session starts ========================
collected 3 items

.tmp\test-auto.py sss                                          [100%]

====================== short test summary info ======================
SKIPPED [3] .tmp\test-auto.py:5: skipped for some reason
======================== 3 skipped in 0.00s =========================

So I think using such an autouse fixture (if you don't mind possibly executing other autouse fixtures too) is the way to go.

[nicoddemus]

Ahh I see that you meant "skipping does not work" if you use a marker, and indeed that's true, but pytest.skip() does not have that limitation. 👍

[LewisGaul]

Yes - the problem with a session-level autouse fixture doing pytest.skip() is that it would skip all tests right? The marker approach was used for targeting specific tests (from session.items).

[nicoddemus]

Use a function-scoped autouse fixture: it can then use the request fixture to decide if a particular test can be skipped or not.

[LewisGaul]

Yep I have this working, thanks - this option just slipped my mind! It is a bit less efficient, but not in a way that matters in my case...