From 8be3cdcd7ab7b163e190505b5027f109d1112797 Mon Sep 17 00:00:00 2001 From: Zac Hatfield-Dodds Date: Wed, 9 Apr 2025 13:42:38 -0700 Subject: [PATCH 1/4] PEP-785: initial draft --- .github/CODEOWNERS | 1 + peps/pep-0785.rst | 387 +++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 388 insertions(+) create mode 100644 peps/pep-0785.rst diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS index ef47f5b9357..982d8b7c608 100644 --- a/.github/CODEOWNERS +++ b/.github/CODEOWNERS @@ -663,6 +663,7 @@ peps/pep-0781.rst @methane peps/pep-0782.rst @vstinner peps/pep-0783.rst @hoodmane @ambv peps/pep-0784.rst @gpshead +peps/pep-0785.rst @gpshead # ... peps/pep-0789.rst @njsmith # ... diff --git a/peps/pep-0785.rst b/peps/pep-0785.rst new file mode 100644 index 00000000000..c96fa1676c1 --- /dev/null +++ b/peps/pep-0785.rst @@ -0,0 +1,387 @@ +PEP: 785 +Title: New methods for easier handling of :class:`ExceptionGroup`\ s +Author: Zac Hatfield-Dodds +Sponsor: Gregory P. Smith +Status: Draft +Type: Standards Track +Created: 08-Apr-2025 +Python-Version: 3.14 + +Abstract +======== + +As :pep:`654` :class:`ExceptionGroup` has come into widespread use across the +Python community, some common but awkward patterns have emerged. We therefore +propose adding two new methods to exception objects: + +- :meth:`!BaseExceptionGroup.flat_exceptions`, returning the 'leaf' exceptions as + a list, with each traceback composited from any intermediate groups. + +- :meth:`!BaseException.preserve_context`, a context manager which + saves and restores the :attr:`!self.__context__` attribute of ``self``, + so that raising the exception within an ``except:`` block. + +We expect this to enable more concise expression of error handling logic in +many medium-complexity cases. Without them, exception-group handlers will +continue to discard intermediate tracebacks and mis-handle ``__context__`` +exceptions, to the detriment of anyone debugging async code. + + +Motivation +========== + +As exception groups come into widespread use, library authors and end users +often write code to process or respond to individual leaf exceptions, for +example when implementing middleware, error logging, or response handlers in +a web framework. + +`Searching GitHub`__ found four implementations of :meth:`!flat_exceptions` by +various names in the first sixty hits, of which none handle +tracebacks.\ [#numbers]_ The same search found thirteen cases where +:meth:`!.flat_exceptions` could be used. We therefore believe that providing +a method on the :class:`BaseException` type with proper traceback preservation +will improve error-handling and debugging experiences across the ecosystem. + +__ https://github.com/search?q=%2Ffor+%5Cw%2B+in+%5Beg%5D%5Cw*%5C.exceptions%3A%2F+language%3APython&type=code + +The rise of exception groups has also made re-raising exceptions caught by an +earlier handler much more common: for example, web-server middleware might +unwrap ``HTTPException`` if that is the sole leaf of a group: + +.. code-block:: python + + except* HTTPException as group: + first, *rest = group.flat_exceptions() # get the whole traceback :-) + if not rest: + raise first + raise + +However, this innocent-seeming code has a problem: ``raise first`` will do +``first.__context__ = group`` as a side effect. This discards the original +context of the error, which may contain crucial information to understand why +the exception was raised. In many production apps it also causes tracebacks +to balloon from hundreds of lines, to tens or even `hundreds of thousands of +lines`__ - a volume which makes understanding errors far more difficult than +it should be. + +__ https://github.com/python-trio/trio/issues/2001#issuecomment-931928509 + + +A new :meth:`!BaseException.preserve_context` method would be a discoverable, +readable, and easy-to-use solution for these cases. + + +Specification +============= + +A new method ``flat_exceptions()`` will be added to ``BaseExceptionGroup``, with the +following signature: + +.. code-block:: python + + def flat_exceptions(self, *, fix_tracebacks=True) -> list[BaseException]: + """ + Return a flat list of all 'leaf' exceptions in the group. + + If fix_tracebacks is True, each leaf will have the traceback replaced + with a composite so that frames attached to intermediate groups are + still visible when debugging. Pass fix_tracebacks=False to disable + this modification, e.g. if you expect to raise the group unchanged. + """ + +A new method ``preserve_context()`` will be added to ``BaseException``, with the +following signature: + +.. code-block:: python + + def preserve_context(self) -> contextlib.AbstractContextManager[Self]: + """ + Context manager that preserves the exception's __context__ attribute. + + When entering the context, the current values of __context__ is saved. + When exiting, the saved value is restored, which allows raising an + exception inside an except block without changing its context chain. + """ + +Usage example: + +.. code-block:: python + + # We're an async web framework, where user code can raise an HTTPException + # to return a particular HTTP error code to the client. However, it may + # (or may not) be raised inside a TaskGroup, so we need to use `except*`; + # and if there are *multiple* such exceptions we'll treat that as a bug. + try: + user_code_here() + except* HTTPException as group: + first, *rest = group.flat_exceptions() + if rest: + raise # handled by internal-server-error middleware + ... # logging, cache updates, etc. + with first.preserve_context(): + raise first + +Without ``.preserve_context()``, this could would have to either: + +* arrange for the exception to be raised *after* the ``except*`` block, + making code difficult to follow in nontrivial cases, or +* discard the existing ``__context__`` of the ``first`` exception, replacing + it with an ``ExceptionGroup`` which is simply an implementation detail, or +* use ``try/except`` instead of ``except*``, handling the possibility that + the group doesn't contain an ``HTTPException`` at all,[#catch-raw-group]_ or +* implement the semantics of ``.preserve_context()`` inline:: + + prev_ctx = first.__context__ + try: + raise first # or `raise first from None`, etc. + finally: + first.__context__ = prev_ctx + del prev_ctx # break gc cycle + + which is not *literally unheard-of*, but remains very very rare. + + +Backwards Compatibility +======================= + +Adding new methods to built-in classes, especially those as widely used as +``BaseException``, can have substantial impacts. However, GitHub search shows +no collisions for these method names (`zero hits `_ and +`three unrelated hits `_ respectively). If user-defined +methods with these names exist in private code they will shadow those proposed +in the PEP, without changing runtime behavior. + +.. _flat-exceptions: https://github.com/search?q=%2F%5C.flat_exceptions%5C%28%2F+language%3APython&type=code +.. _preserve-context: https://github.com/search?q=%2F%5C.preserve_context%5C%28%2F+language%3APython&type=code + + +How to Teach This +================= + +Working with exception groups is an intermediate-to-advanced topic, unlikely +to arise for beginning programmers. We therefore suggest teaching this topic +via documentation, and via just-in-time feedback from static analysis tools. +In intermediate classes, we recommend teaching ``.flat_exceptions()`` together +with the ``.split()`` and ``.subgroup()`` methods, and mentioning +``.preserve_context()`` as an advanced option to address specific pain points. + +Both the API reference and the existing `ExceptionGroup tutorial +`_ should +be updated to demonstrate and explain the new methods. The tutorial should +include examples of common patterns where ``.flat_exceptions()`` and +``.preserve_context()`` help simplify error handling logic. Downstream +libraries which often use exception groups could include similar docs. + +We have also designed lint rules for inclusion in ``flake8-async`` which will +suggest using ``.flat_exceptions()`` when iterating over ``group.exceptions`` +or re-raising a leaf exception, and suggest using ``.preserve_context()`` when +re-raising a leaf exception inside an ``except*`` block would override any +existing context. + + +Reference Implementation +======================== + +While the methods on built-in exceptions will be implemented in C if this PEP +is accepted, we hope that the following Python implementation will be useful +on older versions of Python, and can demonstrate the intended semantics. + +We have found these helper functions quite useful when working with +:class:`ExceptionGroup`\ s in a large production codebase. + +A ``flat_exceptions()`` helper function +--------------------------------------- + +.. code-block:: python + + import copy + import types + from types import TracebackType + + + def flat_exceptions( + self: BaseExceptionGroup, *, fix_traceback: bool = True + ) -> list[BaseException]: + """ + Return a flat list of all 'leaf' exceptions. + + If fix_tracebacks is True, each leaf will have the traceback replaced + with a composite so that frames attached to intermediate groups are + still visible when debugging. Pass fix_tracebacks=False to disable + this modification, e.g. if you expect to raise the group unchanged. + """ + + def _flatten(group: BaseExceptionGroup, parent_tb: TracebackType | None = None): + group_tb = group.__traceback__ + combined_tb = _combine_tracebacks(parent_tb, group_tb) + result = [] + for exc in group.exceptions: + if isinstance(exc, BaseExceptionGroup): + result.extend(_flatten(exc, combined_tb)) + elif fix_tracebacks: + tb = _combine_tracebacks(combined_tb, exc.__traceback__) + result.append(exc.with_traceback(tb)) + else: + result.append(exc) + return result + + return _flatten(self) + + + def _combine_tracebacks( + tb1: TracebackType | None, + tb2: TracebackType | None, + ) -> TracebackType | None: + """ + Combine two tracebacks, putting tb1 frames before tb2 frames. + + If either is None, return the other. + """ + if tb1 is None: + return tb2 + if tb2 is None: + return tb1 + + # Convert tb1 to a list of frames + frames = [] + current = tb1 + while current is not None: + frames.append((current.tb_frame, current.tb_lasti, current.tb_lineno)) + current = current.tb_next + + # Create a new traceback starting with tb2 + new_tb = tb2 + + # Add frames from tb1 to the beginning (in reverse order) + for frame, lasti, lineno in reversed(frames): + new_tb = types.TracebackType( + tb_next=new_tb, tb_frame=frame, tb_lasti=lasti, tb_lineno=lineno + ) + + return new_tb + + +A ``preserve_context()`` context manager +---------------------------------------- + +.. code-block:: python + + class preserve_context: + def __init__(self, exc: BaseException): + self.__exc = exc + self.__context = exc.__context__ + + def __enter__(self): + return self.__exc + + def __exit__(self, exc_type, exc_value, traceback): + assert exc_value is self.__exc, f"did not raise the expected exception {self.__exc!r}" + exc_value.__context__ = self.__context + del self.__context # break gc cycle + + +Rejected Ideas +============== + +Add utility functions instead of methods +---------------------------------------- + +Rather than adding methods to exceptions, we could provide utility functions +like the reference implementations above. + +There are however several reasons to prefer methods: there's no obvious place +where helper functions should live, they take exactly one argument which must +be an instance of ``BaseException``, and methods are both more convenient and +more discoverable. + + +Add ``BaseException.as_group()`` (or group methods) +--------------------------------------------------- + +Our survey of ``ExceptionGroup``-related error handling code also observed +many cases of duplicated logic to handle both a bare exception, and the same +kind of exception inside a group (often incorrectly, motivating +``.flat_exceptions()``). + +We briefly `proposed `__ +adding ``.split(...)`` and ``.subgroup(...)`` methods too all exceptions, +before considering ``.flat_exceptions()`` made us feel this was too clumsy. +As a cleaner alternative, we sketched out an ``.as_group()`` method: + +.. code-block:: python + + def as_group(self): + if not isinstance(self, BaseExceptionGroup): + return BaseExceptionGroup("", [self]) + return self + +However, applying this method to refactor existing code was a negligible +improvement over writing the trivial inline version. We also hope that many +current uses for such a method will be addressed by ``except*`` as older +Python versions reach end-of-life. + +We recommend documenting a "convert to group" recipe for de-duplicated error +handling, instead of adding group-related methods to ``BaseException``. + + +Add ``e.raise_with_preserved_context()`` instead of a context manager +--------------------------------------------------------------------- + +We prefer the context-manager form because it allows ``raise ... from ...`` +if the user wishes to (re)set the ``__cause__``, and is overall somewhat +less magical and tempting to use in cases where it would not be appropriate. +We could be argued around though, if others prefer this form. + + +Footnotes +========= + +.. [#numbers] + From the first sixty `GitHub search results + `__ + for ``for \w+ in [eg]\w*\.exceptions:``, we find: + + * Four functions implementing ``flat_exceptions()`` semantics, none of + which preserve tracebacks: + (`one `__, + `two `__, + `three `__, + `four `__) + + * Six handlers which raise the first exception in a group, discarding + any subsequent errors; these would benefit from both proposed methods. + (`one `__, + `two `__, + `three `__, + `four `__, + `five `__, + `six `__) + + * Seven cases which mishandle nested exception groups, and would thus + benefit from ``flat_exceptions()``. We were surprised to note that only + one of these cases could straightforwardly be replaced by use of an + ``except*`` clause or ``.subgroup()`` method. + (`one `__, + `two `__, + `three `__, + `four `__, + `five `__, + `six `__, + `seven `__) + + indicating that more than a quarter of _all_ hits for this fairly general + search would benefit from the methods proposed in this PEP. + +.. [#catch-raw-group] + This remains very rare, and most cases duplicate logic across + ``except FooError:`` and ``except ExceptionGroup: # containing FooError`` + clauses rather than using something like the as_group trick. + We expect that ``except*`` will be widely used in such cases, before + the methods proposed by this PEP are widely available. + + +Copyright +========= + +This document is placed in the public domain or under the CC0-1.0-Universal license, +whichever is more permissive. From ec91a08308864541d05d10900221e05004562dd6 Mon Sep 17 00:00:00 2001 From: Zac Hatfield-Dodds Date: Sun, 13 Apr 2025 09:05:02 -0700 Subject: [PATCH 2/4] PEP 785: update title formatting Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com> --- peps/pep-0785.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/peps/pep-0785.rst b/peps/pep-0785.rst index c96fa1676c1..0513cba80d5 100644 --- a/peps/pep-0785.rst +++ b/peps/pep-0785.rst @@ -1,5 +1,5 @@ PEP: 785 -Title: New methods for easier handling of :class:`ExceptionGroup`\ s +Title: New methods for easier handling of `ExceptionGroup`\ s Author: Zac Hatfield-Dodds Sponsor: Gregory P. Smith Status: Draft From 1a5100708ae7af1915fcf7538909c0a840708484 Mon Sep 17 00:00:00 2001 From: Jelle Zijlstra Date: Sun, 13 Apr 2025 10:05:44 -0700 Subject: [PATCH 3/4] Update peps/pep-0785.rst --- peps/pep-0785.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/peps/pep-0785.rst b/peps/pep-0785.rst index 0513cba80d5..9eb006af33e 100644 --- a/peps/pep-0785.rst +++ b/peps/pep-0785.rst @@ -1,5 +1,5 @@ PEP: 785 -Title: New methods for easier handling of `ExceptionGroup`\ s +Title: New methods for easier handling of ``ExceptionGroup``\ s Author: Zac Hatfield-Dodds Sponsor: Gregory P. Smith Status: Draft From 596520754e4efec6b5840c139e2fe75dd5afed6c Mon Sep 17 00:00:00 2001 From: Zac Hatfield-Dodds Date: Sun, 13 Apr 2025 12:27:58 -0700 Subject: [PATCH 4/4] PEP-785: fix links and formatting --- peps/pep-0785.rst | 51 ++++++++++++++++++++--------------------------- 1 file changed, 22 insertions(+), 29 deletions(-) diff --git a/peps/pep-0785.rst b/peps/pep-0785.rst index 9eb006af33e..241f3fdc1aa 100644 --- a/peps/pep-0785.rst +++ b/peps/pep-0785.rst @@ -19,7 +19,8 @@ propose adding two new methods to exception objects: - :meth:`!BaseException.preserve_context`, a context manager which saves and restores the :attr:`!self.__context__` attribute of ``self``, - so that raising the exception within an ``except:`` block. + so that re-raising the exception within another handler does not overwrite + the existing context. We expect this to enable more concise expression of error handling logic in many medium-complexity cases. Without them, exception-group handlers will @@ -121,24 +122,16 @@ Usage example: with first.preserve_context(): raise first -Without ``.preserve_context()``, this could would have to either: +Without ``.preserve_context()``, this code would have to either: * arrange for the exception to be raised *after* the ``except*`` block, making code difficult to follow in nontrivial cases, or * discard the existing ``__context__`` of the ``first`` exception, replacing it with an ``ExceptionGroup`` which is simply an implementation detail, or -* use ``try/except`` instead of ``except*``, handling the possibility that - the group doesn't contain an ``HTTPException`` at all,[#catch-raw-group]_ or -* implement the semantics of ``.preserve_context()`` inline:: - - prev_ctx = first.__context__ - try: - raise first # or `raise first from None`, etc. - finally: - first.__context__ = prev_ctx - del prev_ctx # break gc cycle - - which is not *literally unheard-of*, but remains very very rare. +* use ``try/except`` instead of ``except*``, handling the possibility that the + group doesn't contain an ``HTTPException`` at all,\ [#catch-raw-group]_ or +* implement the semantics of ``.preserve_context()`` inline; while this is not + *literally unheard-of*, it remains very rare. Backwards Compatibility @@ -146,13 +139,13 @@ Backwards Compatibility Adding new methods to built-in classes, especially those as widely used as ``BaseException``, can have substantial impacts. However, GitHub search shows -no collisions for these method names (`zero hits `_ and -`three unrelated hits `_ respectively). If user-defined -methods with these names exist in private code they will shadow those proposed -in the PEP, without changing runtime behavior. +no collisions for these method names (`zero hits`__ and +`three unrelated hits`__ respectively). If user-defined methods with these +names exist in private code they will shadow those proposed in the PEP, +without changing runtime behavior. -.. _flat-exceptions: https://github.com/search?q=%2F%5C.flat_exceptions%5C%28%2F+language%3APython&type=code -.. _preserve-context: https://github.com/search?q=%2F%5C.preserve_context%5C%28%2F+language%3APython&type=code +__ https://github.com/search?q=%2F%5C.flat_exceptions%5C%28%2F+language%3APython&type=code +__ https://github.com/search?q=%2F%5C.preserve_context%5C%28%2F+language%3APython&type=code How to Teach This @@ -165,13 +158,14 @@ In intermediate classes, we recommend teaching ``.flat_exceptions()`` together with the ``.split()`` and ``.subgroup()`` methods, and mentioning ``.preserve_context()`` as an advanced option to address specific pain points. -Both the API reference and the existing `ExceptionGroup tutorial -`_ should -be updated to demonstrate and explain the new methods. The tutorial should -include examples of common patterns where ``.flat_exceptions()`` and +Both the API reference and the existing `ExceptionGroup tutorial`__ +should be updated to demonstrate and explain the new methods. The tutorial +should include examples of common patterns where ``.flat_exceptions()`` and ``.preserve_context()`` help simplify error handling logic. Downstream libraries which often use exception groups could include similar docs. +__ https://docs.python.org/3/tutorial/errors.html#raising-and-handling-multiple-unrelated-exceptions + We have also designed lint rules for inclusion in ``flake8-async`` which will suggest using ``.flat_exceptions()`` when iterating over ``group.exceptions`` or re-raising a leaf exception, and suggest using ``.preserve_context()`` when @@ -288,7 +282,6 @@ Add utility functions instead of methods Rather than adding methods to exceptions, we could provide utility functions like the reference implementations above. - There are however several reasons to prefer methods: there's no obvious place where helper functions should live, they take exactly one argument which must be an instance of ``BaseException``, and methods are both more convenient and @@ -369,15 +362,15 @@ Footnotes `six `__, `seven `__) - indicating that more than a quarter of _all_ hits for this fairly general + indicating that more than a quarter of *all* hits for this fairly general search would benefit from the methods proposed in this PEP. .. [#catch-raw-group] This remains very rare, and most cases duplicate logic across ``except FooError:`` and ``except ExceptionGroup: # containing FooError`` - clauses rather than using something like the as_group trick. - We expect that ``except*`` will be widely used in such cases, before - the methods proposed by this PEP are widely available. + clauses rather than using something like the ``as_group()`` trick. + We expect that ``except*`` will be widely used in such cases by the time + that the methods proposed by this PEP are widely available. Copyright