Fix annotations support on 3.14 #852

JelleZijlstra · 2025-05-26T16:09:36Z

With this change, the tests run for me on a local build of Python 3.14.
There are a lot of failures related to sys.getrefcount() but that seems
to be an unrelated issue.

Closes #810. Fixes #651. Fixes #795.

With this change, the tests run for me on a local build of Python 3.14. There are a lot of failures related to sys.getrefcount() but that seems to be an unrelated issue. Closes jcrist#810. Fixes jcrist#651. Fixes jcrist#795.

JelleZijlstra · 2025-05-26T16:12:21Z

msgspec/_core.c

+    if (annotations == NULL) {
+        if (mod->get_annotate_from_class_namespace != NULL) {
+            PyObject *annotate = PyObject_CallOneArg(
+                mod->get_annotate_from_class_namespace, info->namespace


This is the portable approach, which should continue to work on future Python versions. If you value performance over portability, you can instead inline this function https://github.com/python/cpython/blob/9ddc7c548d45b73c84131e6d75b03c26a3e8b6e8/Lib/annotationlib.py#L824 ; it's just dict operations.

I made it a separate function in CPython so that we can be free to optimize the internal representation of annotate functions in the future. For example, perhaps in 3.15 the class will store just a code object instead of a function.

JelleZijlstra · 2025-05-26T16:13:34Z

msgspec/_utils.py

                value = _forward_ref(value)
            value = _eval_type(value, cls_locals, cls_globals)
            if mapping is not None:
                value = _apply_params(value, mapping)
+            if value is None:


I changed _eval_type so it no longer turns None into NoneType.

hroncok · 2025-05-26T16:50:54Z

I get a copule of SystemError: error return without exception set:

___________ TestTypedDict.test_total_partially_optional[json-False] ____________

self = <test_common.TestTypedDict object at 0x7f25e7f160a0>
proto = <module 'msgspec.json' from '/builddir/build/BUILD/python-msgspec-0.19.0-build/BUILDROOT/usr/lib64/python3.14/site-packages/msgspec/json.py'>
use_typing_extensions = False

    @pytest.mark.parametrize("use_typing_extensions", [False, True])
    def test_total_partially_optional(self, proto, use_typing_extensions):
        if use_typing_extensions:
            tex = pytest.importorskip("typing_extensions")
            cls = tex.TypedDict
        else:
            cls = TypedDict
    
        class Base(cls):
            a: int
            b: str
    
        class Ex(Base, total=False):
            c: str
    
        dec = proto.Decoder(Ex)
    
        x = {"a": 1, "b": "two", "c": "extra"}
>       assert dec.decode(proto.encode(x)) == x
E       SystemError: <built-in method decode of msgspec.json.Decoder object at 0x7f25e89b85e0> returned NULL without setting an exception

tests/test_common.py:2106: SystemError
__________ TestTypedDict.test_total_partially_optional[msgpack-False] __________

self = <test_common.TestTypedDict object at 0x7f25e7fcab70>
proto = <module 'msgspec.msgpack' from '/builddir/build/BUILD/python-msgspec-0.19.0-build/BUILDROOT/usr/lib64/python3.14/site-packages/msgspec/msgpack.py'>
use_typing_extensions = False

    @pytest.mark.parametrize("use_typing_extensions", [False, True])
    def test_total_partially_optional(self, proto, use_typing_extensions):
        if use_typing_extensions:
            tex = pytest.importorskip("typing_extensions")
            cls = tex.TypedDict
        else:
            cls = TypedDict
    
        class Base(cls):
            a: int
            b: str
    
        class Ex(Base, total=False):
            c: str
    
        dec = proto.Decoder(Ex)
    
        x = {"a": 1, "b": "two", "c": "extra"}
>       assert dec.decode(proto.encode(x)) == x
E       SystemError: error return without exception set

tests/test_common.py:2106: SystemError
___________ TestTypedDict.test_required_and_notrequired[json-False] ____________

self = <test_common.TestTypedDict object at 0x7f25e813e180>
proto = <module 'msgspec.json' from '/builddir/build/BUILD/python-msgspec-0.19.0-build/BUILDROOT/usr/lib64/python3.14/site-packages/msgspec/json.py'>
use_typing_extensions = False

    @pytest.mark.parametrize("use_typing_extensions", [False, True])
    def test_required_and_notrequired(self, proto, use_typing_extensions):
        if use_typing_extensions:
            module = "typing_extensions"
        else:
            module = "typing"
    
        ns = pytest.importorskip(module)
    
        if not hasattr(ns, "Required"):
            pytest.skip(f"{module}.Required is not available")
    
        source = f"""
        from __future__ import annotations
        from {module} import TypedDict, Required, NotRequired
    
        class Base(TypedDict):
            a: int
            b: NotRequired[str]
    
        class Ex(Base, total=False):
            c: str
            d: Required[bool]
        """
    
        with temp_module(source) as mod:
            dec = proto.Decoder(mod.Ex)
    
            x = {"a": 1, "b": "two", "c": "extra", "d": False}
>           assert dec.decode(proto.encode(x)) == x
E           SystemError: <built-in method decode of msgspec.json.Decoder object at 0x7f25e91a0590> returned NULL without setting an exception

tests/test_common.py:2144: SystemError
__________ TestTypedDict.test_required_and_notrequired[msgpack-False] __________

self = <test_common.TestTypedDict object at 0x7f25e8199550>
proto = <module 'msgspec.msgpack' from '/builddir/build/BUILD/python-msgspec-0.19.0-build/BUILDROOT/usr/lib64/python3.14/site-packages/msgspec/msgpack.py'>
use_typing_extensions = False

    @pytest.mark.parametrize("use_typing_extensions", [False, True])
    def test_required_and_notrequired(self, proto, use_typing_extensions):
        if use_typing_extensions:
            module = "typing_extensions"
        else:
            module = "typing"
    
        ns = pytest.importorskip(module)
    
        if not hasattr(ns, "Required"):
            pytest.skip(f"{module}.Required is not available")
    
        source = f"""
        from __future__ import annotations
        from {module} import TypedDict, Required, NotRequired
    
        class Base(TypedDict):
            a: int
            b: NotRequired[str]
    
        class Ex(Base, total=False):
            c: str
            d: Required[bool]
        """
    
        with temp_module(source) as mod:
            dec = proto.Decoder(mod.Ex)
    
            x = {"a": 1, "b": "two", "c": "extra", "d": False}
>           assert dec.decode(proto.encode(x)) == x
E           SystemError: error return without exception set

tests/test_common.py:2144: SystemError

Do you also get those?

JelleZijlstra · 2025-05-26T16:55:21Z

I didn't how, are you running the test suite exactly?

I get these failures which all seem related to getrefcount calls:

FAILED tests/test_common.py::TestGenericStruct::test_generic_struct_info_cached[json] - assert 3 == 4
FAILED tests/test_common.py::TestGenericStruct::test_generic_struct_info_cached[msgpack] - assert 3 == 4
FAILED tests/test_common.py::TestGenericDataclassOrAttrs::test_generic_info_cached[dataclass-json] - assert 3 == 4
FAILED tests/test_common.py::TestGenericDataclassOrAttrs::test_generic_info_cached[dataclass-msgpack] - assert 3 == 4
FAILED tests/test_convert.py::TestConvert::test_custom_input_type_works_with_any - assert 2 == 3
FAILED tests/test_convert.py::TestConvert::test_custom_input_type_works_with_custom - assert 2 == 3
FAILED tests/test_convert.py::TestConvert::test_custom_input_type_works_with_dec_hook - assert 1 == 2
FAILED tests/test_convert.py::TestInt::test_int_subclass - assert 2 == 3
FAILED tests/test_convert.py::TestBinary::test_bytes_subclass - AssertionError: assert 1 == 2
FAILED tests/test_convert.py::TestEnum::test_int_enum_int_subclass - assert 1 == 2
FAILED tests/test_json.py::TestDatetime::test_decode_timezone_cache - assert 2 == 3
FAILED tests/test_json.py::TestStruct::test_decode_struct - AssertionError: assert 2 == 3
FAILED tests/test_msgpack.py::TestTypedDecoder::test_decode_memoryview_zerocopy[bytes] - AssertionError: assert 2 == 3
FAILED tests/test_msgpack.py::TestTypedDecoder::test_decode_memoryview_zerocopy[memoryview] - AssertionError: assert 2 == 3
FAILED tests/test_msgpack.py::TestTypedDecoder::test_vartuple_lengths[1] - AssertionError: assert 2 == 3
FAILED tests/test_msgpack.py::TestTypedDecoder::test_vartuple_lengths[31] - AssertionError: assert 2 == 3
FAILED tests/test_msgpack.py::TestTypedDecoder::test_vartuple_lengths[32] - AssertionError: assert 2 == 3
FAILED tests/test_msgpack.py::TestTypedDecoder::test_vartuple_lengths[255] - AssertionError: assert 2 == 3
FAILED tests/test_msgpack.py::TestTypedDecoder::test_vartuple_lengths[256] - AssertionError: assert 2 == 3
FAILED tests/test_msgpack.py::TestTypedDecoder::test_vartuple_lengths[65535] - AssertionError: assert 2 == 3
FAILED tests/test_msgpack.py::TestTypedDecoder::test_vartuple_lengths[65536] - AssertionError: assert 2 == 3
FAILED tests/test_struct.py::test_struct_reference_counting - assert 2 == 3

To run it I do:

$ python -VV
Python 3.14.0b1+ (heads/3.14:2a089244f0d, May 26 2025, 08:38:42) [Clang 15.0.0 (clang-1500.3.9.4)]
$ python -m pytest -s -m "not mypy and not pyright"

That's the latest tip of the 3.14 branch.

JelleZijlstra · 2025-05-26T16:56:43Z

The failures you post feel like they wouldn't be related to retrieving annotations; the code I'm changing is just in gathering annotations at class creation time, and whatever is happening in those tests is after the class is already created.

JelleZijlstra · 2025-05-26T17:05:52Z

Oh actually this is because of a bug in b1 that I fixed (python/cpython#133701); the fix will be in b2 which is about to go out.

I can reproduce it with the following change in the test:

$ git diff
diff --git a/tests/test_common.py b/tests/test_common.py
index 38898be..1c4c969 100644
--- a/tests/test_common.py
+++ b/tests/test_common.py
@@ -2099,6 +2099,7 @@ class TestTypedDict:
 
         class Ex(Base, total=False):
             c: str
+        Ex.__annotations__ = {"c": "str"}
 
         dec = proto.Decoder(Ex)

I do think that indicates a bug in msgspec; presumably it shouldn't crash even if people mess with the __annotations__ manually. I'll see if I can submit a fix.

hroncok · 2025-05-26T17:20:21Z

The failures occurred on b1 indeed.

Fixes at least some of the failures reported in: jcrist#852 (comment) These were exposed by a bug in 3.14b1 where TypedDict reported incorrect `__annotations__` but correct `__required_keys__`. msgspec would crash in this case. The bug is reproducible on earlier Python versions by manually manipulating attributes on a TypedDict class. It's a pretty marginal bug but I would argue the extension should be robust to this sort of edge case.

JelleZijlstra · 2025-05-26T17:50:55Z

#853 for that one.

hroncok · 2025-05-26T18:37:02Z

There are a lot of failures related to sys.getrefcount() but that seems to be an unrelated issue.

I opened #854

Fix annotations support on 3.14

89ecb0b

With this change, the tests run for me on a local build of Python 3.14. There are a lot of failures related to sys.getrefcount() but that seems to be an unrelated issue. Closes jcrist#810. Fixes jcrist#651. Fixes jcrist#795.

JelleZijlstra mentioned this pull request May 26, 2025

Python 3.14: Call __annotate__ on type objects to get annotations #810

Closed

JelleZijlstra commented May 26, 2025

View reviewed changes

JelleZijlstra mentioned this pull request May 26, 2025

Fix crash when TypedDict contains incorrect metadata #853

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Fix annotations support on 3.14 #852

Fix annotations support on 3.14 #852

Uh oh!

JelleZijlstra commented May 26, 2025

Uh oh!

JelleZijlstra May 26, 2025 •

edited

Loading

Uh oh!

JelleZijlstra May 26, 2025

Uh oh!

hroncok commented May 26, 2025

Uh oh!

JelleZijlstra commented May 26, 2025

Uh oh!

JelleZijlstra commented May 26, 2025

Uh oh!

JelleZijlstra commented May 26, 2025

Uh oh!

hroncok commented May 26, 2025

Uh oh!

JelleZijlstra commented May 26, 2025

Uh oh!

hroncok commented May 26, 2025

Uh oh!

Uh oh!

Fix annotations support on 3.14 #852

Are you sure you want to change the base?

Fix annotations support on 3.14 #852

Uh oh!

Conversation

JelleZijlstra commented May 26, 2025

Uh oh!

JelleZijlstra May 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

JelleZijlstra May 26, 2025

Choose a reason for hiding this comment

Uh oh!

hroncok commented May 26, 2025

Uh oh!

JelleZijlstra commented May 26, 2025

Uh oh!

JelleZijlstra commented May 26, 2025

Uh oh!

JelleZijlstra commented May 26, 2025

Uh oh!

hroncok commented May 26, 2025

Uh oh!

JelleZijlstra commented May 26, 2025

Uh oh!

hroncok commented May 26, 2025

Uh oh!

Uh oh!

JelleZijlstra May 26, 2025 •

edited

Loading