tests/test_tags.py:43: AssertionError on Darwin

[doronbehar]

Links:

• https://hydra.nixos.org/build/298947878 (python312Packages.meson-python.aarch64-darwin)
• https://hydra.nixos.org/build/298947879 (python312Packages.meson-python.x86_64-darwin)
• https://hydra.nixos.org/build/298967891 (python313Packages.meson-python.aarch64-darwin)
• https://hydra.nixos.org/build/298967893 (python313Packages.meson-python.x86_64-darwin)

________________________________ test_wheel_tag ________________________________

    def test_wheel_tag():
>       assert str(mesonpy._tags.Tag()) == f'{INTERPRETER}-{ABI}-{PLATFORM}'
E       AssertionError: assert 'cp313-cp313-...sx_11_0_arm64' == 'cp313-cp313-...sx_15_0_arm64'
E         
E         - cp313-cp313-macosx_15_0_arm64
E         ?                     ^
E         + cp313-cp313-macosx_11_0_arm64
E         ?                     ^

tests/test_tags.py:43: AssertionError
____________________________ test_tag_platlib_wheel ____________________________

    def test_tag_platlib_wheel():
        builder = wheel_builder_test_factory({
            'platlib': [f'extension{SUFFIX}'],
        })
>       assert str(builder.tag) == f'{INTERPRETER}-{ABI}-{PLATFORM}'
E       AssertionError: assert 'cp313-cp313-...sx_11_0_arm64' == 'cp313-cp313-...sx_15_0_arm64'
E         
E         - cp313-cp313-macosx_15_0_arm64
E         ?                     ^
E         + cp313-cp313-macosx_11_0_arm64
E         ?                     ^

tests/test_tags.py:99: AssertionError
_____________________________ test_tag_stable_abi ______________________________

    def test_tag_stable_abi():
        builder = wheel_builder_test_factory({
            'platlib': [f'extension{ABI3SUFFIX}'],
        }, limited_api=True)
        # PyPy does not support the stable ABI.
        abi = 'abi3' if '__pypy__' not in sys.builtin_module_names else ABI
>       assert str(builder.tag) == f'{INTERPRETER}-{abi}-{PLATFORM}'
E       AssertionError: assert 'cp313-abi3-macosx_11_0_arm64' == 'cp313-abi3-macosx_15_0_arm64'
E         
E         - cp313-abi3-macosx_15_0_arm64
E         ?                    ^
E         + cp313-abi3-macosx_11_0_arm64
E         ?                    ^

tests/test_tags.py:108: AssertionError

[dnicolodi]

For some reason, on your platform, packaging and meson-python come to two different conclusions about what the current macOS version is. The algorithm to derive the platform tag is almost identical in the two implementations, thus it is very hard to imagine where the discrepancy arises, unless the tests are run with the $MACOSX_DEPLOYMENT_TARGET environment variable set. The code in meson-python and packaging responsible for deriving the platform tag has not changed recently, thus I don't think that this is a regression in recent releases of either projects.

The tests that are failing on your platform are run as part of the CI jobs for meson-python, thus I'm fairly sure that the issue is with your particular setup. However, the bug report lacks information that may be useful to track the issue down.

[mweinelt]

We do indeed set MACOSX_DEPLOYMENT_TARGET=11.3 for the lower end of supported darwin versions, but our darwin builders are of course on recent versions (15.5).

Now meson-python respecting that variable seems to be interested in the target version, while packaging detects the running OS version.

I think the question we're having is: Is it safe to disable the tests in that case?

[dnicolodi]

Is it safe to disable the tests in that case?

You can apply the same workaround you applied for previous releases of meson-python: the meson-python behavior and these tests are not something new. Indeed, I have the feeling that I have been discussing this before...

[rgommers]

There are some tests in test_tags.py that explicitly set MACOSX_DEPLOYMENT_TARGET via the monkeypatch fixture to test its handling. I think we have two ways to avoid failures for the tests that don't do that:

1. Skipping them on macOS if that env var is already set
2. Use [monkeypatch.delenv](https://docs.pytest.org/en/7.1.x/reference/reference.html#pytest.MonkeyPatch.delenv) with raising=False` to unset the env var

This isn't really a bug, but it is a little annoying that the tests implicitly assume that MACOSX_DEPLOYMENT_TARGET is unset. For distro packaging setups where the build and test phases aren't completely separated, having it set by default in the build env is expected and that may then leak into the test env.

[dnicolodi]

I agree that deleting the environment variable in the test is the right thing to do.

I wonder how this was not a problem for Nix on macOS before: the behavior of this tests hasn't changed in a long time. Is setting the environment variable when testing the package a new thing or is running the tests as part of the package build a new thing?

I would also like to note that the Nix packaging seems to disable many other tests for the wrong reason: the package definition states that they require to be execute from a git checkout, but AFAIK this is not the case. Some tests require to run git, but that is a different thing, and I have the impression that the disabled tests are many more than the ones requiring git, which should be only the ones defined in test_sdist.py and maybe a few more.

[mweinelt]

I just recently adopted the package and enabled the tests, so we've only been running them for the last two weeks and only on 0.18.0.

And yeah, I did indeed disable too many tests, because I parsed the error message badly. After passing git to the tests they now all work. Will submit that change shortly.

And as a workaround I proposed a change to unset MACOSX_DEPLOYMENT_TARGET for the tests and that made them work on darwin.