Best way to ensure that there are no unexpected unraisable exceptions (e.g. from __del__)

[pelson]

I would like to ensure that every test in a codebase checks that there were no unexpected unraisable exceptions. My current approach is to turn the PyTest warning that is propagated into an exception - any unraisables will result in a failing case:

class Case1:
    def __del__(self):
        raise ValueError("Raise in __del__")


@pytest.mark.filterwarnings('error::pytest.PytestUnraisableExceptionWarning')
def test_unraisable_filter_working():
    Case1()

This results in:

$ pytest conftest.py  -ssvv
======================================================================================================= test session starts ========================================================================================================
platform linux -- Python 3.9.7, pytest-7.2.0, pluggy-1.0.0 
cachedir: .pytest_cache
rootdir: .
plugins: asyncio-0.20.3
asyncio: mode=strict
collected 1 item                                                                                                                                                                                                                   

test.py::test_unraisable_filter_working FAILED

============================================================================================================= FAILURES =============================================================================================================
__________________________________________________________________________________________________ test_unraisable_filter_working __________________________________________________________________________________________________

cls = <class '_pytest.runner.CallInfo'>, func = <function call_runtest_hook.<locals>.<lambda> at 0x7f51b90854c0>, when = 'call', reraise = (<class '_pytest.outcomes.Exit'>, <class 'KeyboardInterrupt'>)

    @classmethod
    def from_call(
        cls,
        func: "Callable[[], TResult]",
        when: "Literal['collect', 'setup', 'call', 'teardown']",
        reraise: Optional[
            Union[Type[BaseException], Tuple[Type[BaseException], ...]]
        ] = None,
    ) -> "CallInfo[TResult]":
        """Call func, wrapping the result in a CallInfo.
    
        :param func:
            The function to call. Called without arguments.
        :param when:
            The phase in which the function is called.
        :param reraise:
            Exception or exceptions that shall propagate if raised by the
            function, instead of being wrapped in the CallInfo.
        """
        excinfo = None
        start = timing.time()
        precise_start = timing.perf_counter()
        try:
>           result: Optional[TResult] = func()

venv-39/lib/python3.9/site-packages/_pytest/runner.py:339: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
venv-39/lib/python3.9/site-packages/_pytest/runner.py:260: in <lambda>
    lambda: ihook(item=item, **kwds), when=when, reraise=reraise
venv-39/lib/python3.9/site-packages/pluggy/_hooks.py:265: in __call__
    return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
venv-39/lib/python3.9/site-packages/pluggy/_manager.py:80: in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
venv-39/lib/python3.9/site-packages/_pytest/unraisableexception.py:88: in pytest_runtest_call
    yield from unraisable_exception_runtest_hook()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

    def unraisable_exception_runtest_hook() -> Generator[None, None, None]:
        with catch_unraisable_exception() as cm:
            yield
            if cm.unraisable:
                if cm.unraisable.err_msg is not None:
                    err_msg = cm.unraisable.err_msg
                else:
                    err_msg = "Exception ignored in"
                msg = f"{err_msg}: {cm.unraisable.object!r}\n\n"
                msg += "".join(
                    traceback.format_exception(
                        cm.unraisable.exc_type,
                        cm.unraisable.exc_value,
                        cm.unraisable.exc_traceback,
                    )
                )
>               warnings.warn(pytest.PytestUnraisableExceptionWarning(msg))
E               pytest.PytestUnraisableExceptionWarning: Exception ignored in: <function Case1.__del__ at 0x7f51b90ea4c0>
E               
E               Traceback (most recent call last):
E                 File "test.py", line 68, in __del__
E                   raise ValueError("Raise in __del__")
E               ValueError: Raise in __del__

venv-39/lib/python3.9/site-packages/_pytest/unraisableexception.py:78: PytestUnraisableExceptionWarning
===================================================================================================== short test summary info ======================================================================================================
FAILED ...::test_unraisable_filter_working - pytest.PytestUnraisableExceptionWarning: Exception ignored in: <function Case1.__del__ at 0x7f51b90ea4c0>
======================================================================================================== 1 failed in 0.13s =========================================================================================================
(venv-39) $ 

Not surprisingly, this approach fails if there is a reference to the object somewhere. Unfortunately, it looks like PyTest holds on to such a reference within an exception:

class Case2:
    def __init__(self):
        raise TypeError("Raise in __init__")

    def __del__(self):
        raise ValueError("Raise in __del__")


@pytest.mark.filterwarnings('error::pytest.PytestUnraisableExceptionWarning')
def test_unraisable_filter_working():

    with pytest.raises(TypeError) as err:
        Case2()

$ pytest ./pyda/tests/conftest.py  -ssvv
======================================================================================================= test session starts ========================================================================================================
platform linux -- Python 3.9.7, pytest-7.2.0, pluggy-1.0.0 -- cachedir: .pytest_cache
rootdir: .
plugins: asyncio-0.20.3
asyncio: mode=strict
collected 1 item                                                                                                                                                                                                                   

test.py::test_unraisable_filter_working PASSED

======================================================================================================== 1 passed in 0.01s =========================================================================================================
Exception ignored in: <function Case2.__del__ at 0x7fb11bdf1550>
Traceback (most recent call last):
  File "test.py", line 75, in __del__
    raise ValueError("Raise in __del__")
ValueError: Raise in __del__

This can be worked around by explicitly del err at the end of the test (as discussed in #8153 (comment)).

My questions are:

• How to avoid the need to manually del the error object (and avoid any other gotchas like that) in order to be sure that __del__ implementations aren't accidentally raising
• Is there a pattern to test specific behaviours in __del__? We could use mock, or a pattern such as the following, but are there better approaches?

c = Case2.__new__(Case2)
with pytest.raises(ValueError) as err:
    c.__del__()

The problem with the latter approach is that it will result in __del__ being called twice, so some further patching is needed to get it to behave nicely in error modes.

[bluetech]

How to avoid the need to manually del the error object (and avoid any other gotchas like that) in order to be sure that del implementations aren't accidentally raising

First, this isn't specific to pytest.raises; the same happens in

try:
    Case2()
except TypeError as e:
    exc = e

I think what happens (didn't verify), is that err/exc hold a reference to the exception, which holds a reference to the exception's traceback object (exc.__traceback__), which holds a reference to the stack frame which raised (Case2.__init__), including the locals, which includes the Case2 instance. Further, exc.__traceback__ causes a reference cycle (see here and here). So the Case2() isn't being cleaned immediately by the reference count going to zero, instead it is cleaned by the GC cycle collector which runs too late for the warning to be raised.

But more generally, the Python language itself does not guarantee "__del__ is called when instance goes out of scope", it only guarantees "__del__ at some point in the future", so it is best not to rely on such behavior.

My recommendation would be:

1. If you can, avoid __del__ entirely in favor of an explicit context manager which does needed cleanups in __exit__ (you can keep __del__ as a fail-over if someone doesn't use the context manager).
2. If you can't do 1, do del explicitly in the test.