[Question] exceptions in pytest_unconfigure(...) exit 1

[symonk]

Hi quick question, is it by design that raising an exception in a pytest_unconfigure(...) hook exits 1 ? whereas the pytest_configure(...) exits 3 INTERNAL_ERROR ? (makes sense as they happen at very different time and serve different use cases)

if so is it preferred to raise a pytest.UsageError | pytest.exit(...) after catching such exception? If i wanted the run to be a failure that is not considered a 'test failure' i.e ExitCode.TESTS_FAILED.

It's a complicated use case and heavily using end2end tests so we do some setup/teardown stuff through configure/unconfigure which is maybe i'll advised in favour of inside the session instead, i.e we need to do stuff on a runtime environment that when setup may work, but when torn down later may be problematic and or not possible

Also the reason we use 1 is because we run multiple invocations of pytest in a single CI run due to some xdist limitations / isolated tests sequential vs parallel ones and exiting 1 means something very different to us than >1 as far as subsequent CI logic is concerned.

Whats the preferred approach? we can shift our plugin implementations after session start vs the pytest configure/unconfigure and leave that largely for handling some logic on 'if' a certain plugin should be registered, I suspect using configure/unconfigure like we do is maybe not it's intended use case.

[nicoddemus]

Hi quick question, is it by design that raising an exception in a pytest_unconfigure(...) hook exits 1 ? whereas the pytest_configure(...) exits 3 INTERNAL_ERROR ?

Not sure. I would expect both to return INTERNAL_ERROR actually. The discrepancy might be accidental.

i'll advised in favour of inside the session instead, i.e we need to do stuff on a runtime environment that when setup may work, but when torn down later may be problematic and or not possible

Hmm I would suggest using configure/unconfigure for pytest specific stuff, like managing some data structure used by your plugin, while leaving session start/end for other external services (not pytest related). From your use case seems like session start/finish might make more sense then.

[symonk]

Interesting @nicoddemus; with a simple example:

#conftest.py
def pytest_configure():
    raise Exception("I exit 3 if no unconfigure raises")

# test_foo.py
def test_foo():
    assert True

results in:

pytest -s .
echo $?
3

when adding:

# additional hook in conftest.py
def pytest_unconfigure():
    raise Exception("now I exit 1")

pytest -s .
echo $?
1

I had a look over the code and _ensure_unconfigure(...) isn't as well guarded; so I assumed it was by design; is there a chance you think that is maybe an oversight? Happy to convert to an issue if you think it has any merit; i'm still not quite sure.

[nicoddemus]

is there a chance you think that is maybe an oversight?

That's a part of the code which I wasn't around when introduced so I can't say for sure, but I think it is definitely possible. @RonnyPfannschmidt do you have any insights here?

[RonnyPfannschmidt]

I suspect its a missed case for the error handling surrounding the introduction of extra return codes

I cannot investigate the details soon, but i recommend following the call chain in config /unconfig of the framework

[symonk]

Thanks, i'll convert to an issue here and do some digging when I have some time (likely this weekend) on vacation before then

[symonk]

opened: #9808 for more indepth discussions and investigation.