Prototype support for PEP 739 #13945

FFY00 · 2024-11-22T23:21:19Z

The overall goal is to support targeting Python installations without need for introspection, given all necessary information is given externally.

This is built upon:

PEP 739, for all information necessary for the build process
A future PEP, for all information necessary for the installation paths
Convert _sysconfigdata to a JSON file python/cpython#127178, for py_installation.get_variable/py_installation.has_variable support

This PR is just a prototype, as I work to get all necessary pieces finalized.

FFY00 · 2025-01-20T21:27:13Z

Rebased on master.

eli-schwartz

This is starting to look quite nice! \o/ How close are we to getting the PEP approved and deployed (e.g. in a cpython alpha)?

test cases/unit/125 python extension/build-details.json

mesonbuild/modules/python.py

FFY00 · 2025-01-21T00:54:57Z

This is starting to look quite nice! \o/ How close are we to getting the PEP approved and deployed (e.g. in a cpython alpha)?

Thanks!

I have requested a review for the PEP, so hopefully we should hear something soon. If it gets accepted, I'll add the file to CPython as soon as possible, so it should be available on the following release. I'll also publish a package to generate the build config file to PyPI, so that we can use it on older Python versions, or Python implementations that do not ship it, as long as you are able to run code on the target.

Afterwards, in a followup to PEP 739, I also want to standardize a file for the installation environment, which will provide the installation paths. We'll ship a file for the default environment, and unless we run into some issue, I'll also get venv to generate a file for the virtual environment it creates. With the build config + environment config, we should be able to provide all Python-related information statically to Meson.

FFY00 · 2025-02-05T13:49:41Z

The PEP has been accepted.

https://peps.python.org/pep-0739/

I'll add support for it in Python 3.14, and then come back to this PR.

eli-schwartz

Fantastic news! Looking forward to local testing as soon as the backportable version exists. Some more quick comments.

mesonbuild/dependencies/python.py

eli-schwartz · 2025-02-05T14:25:54Z

mesonbuild/dependencies/pkgconfig.py

-        extra_paths: T.List[str] = self.env.coredata.optstore.get_value(key)[:]
+        extra_paths: T.List[str] = self.env.coredata.optstore.get_value(key)[:] + self.extra_paths


Hmm, I see what you're doing here. You allow a PkgConfigDependency to take an additional PKG_CONFIG_PATH which is joined to the one in the cross file, which also means we don't need to push/pop os.environ.

On the flip side, the way we previously used it was via PKG_CONFIG_LIBDIR which was more narrowly scoped. I suspect this change means we can accidentally fall back to a different python than the one we were searching for (but still the same major.minor version at least)...

Ah hold on, we anyways fall back to checking for pkg-config without the internal LIBPC which means this is actually doing exactly what we want with one less check. Heh. :)

Yeah, and without modifying os.environ 😊

mesonbuild/dependencies/pkgconfig.py

mesonbuild/dependencies/python.py

unittests/allplatformstests.py

Signed-off-by: Filipe Laíns <lains@riseup.net>

…figDependency Signed-off-by: Filipe Laíns <lains@riseup.net>

Signed-off-by: Filipe Laíns <lains@riseup.net>

virtuald · 2025-03-28T15:12:09Z

@FFY00

I have requested a review for the PEP, so hopefully we should hear something soon. If it gets accepted, I'll add the file to CPython as soon as possible, so it should be available on the following release. I'll also publish a package to generate the build config file to PyPI, so that we can use it on older Python versions, or Python implementations that do not ship it, as long as you are able to run code on the target.

Have you done any work for generating the build configuration for older python interpreters? If not, I can probably spend some time on that.

FFY00 · 2025-03-28T16:54:08Z

Yes. It might not handle every quirk and doesn't have tests, but it's here:

https://github.com/FFY00/python-instrospection/blob/main/python_introspection/scripts/generate-build-details.py

mgorny

Overall, great work. Thanks!

That said, from the initial eyeball review it seems that some of the code assumes that non-required values will be present, and they look especially incorrect for PyPy.

I'm going to actually run the code tomorrow and try to experiment a bit with it.

mgorny · 2025-05-21T18:05:14Z

mesonbuild/dependencies/python.py

+        value = self._data
+        for part in key.split('.'):
+            value = value[part]
+        return value


There's a reduce() pattern you could use here:

Suggested change

value = self._data

for part in key.split('.'):

value = value[part]

return value

return functools.reduce(lambda d, k: d[k], key.split('.'), self._data)

Sure. If we are simplifying it that way, we can also use operator.getitem to replace the lambda.

Suggested change

value = self._data

for part in key.split('.'):

value = value[part]

return value

return functools.reduce(operator.getitem, key.split('.'), self._data)

Oh, yes, I swear I have looked for one and couldn't find it. But now I see it, I must have made some silly mistake.

Included in my fork now.

mgorny · 2025-05-21T18:43:50Z

mesonbuild/dependencies/python.py

+            if not self.build_config['libpython']:
+                mlog.debug('This Python installation does not provide a libpython')
+                return False


I'm sorry if I'm missing something but I think this assertion is incorrect. Per PEP 739, libpython is optional, and at least on Gentoo PyPy is installed without a shared library. Given that C extensions can be built without linking to it explicitly, and Meson has no problem building such extensions for PyPy, I don't think we should enforce libpython being present here.

Yep, The build-detail.json file semantics changed a bit during the PEP's lifetime, this is a leftover of that.

Taking your feedback into account, I think this error should be tied to libpython.link_extensions, but unfortunately, the spec doesn't permit libpython.link_extensions when libpython.dynamic is missing.

I have opened pypa/packaging.python.org#1862 to track changes we'd like to make for a new spec version, and added this as one of them.

Thanks for noticing this shortcoming!

Hmm, that's an interesting point. I'll reply to the PyPA bug with my thoughts.

mgorny · 2025-05-21T18:46:12Z

mesonbuild/dependencies/python.py

+    @property
+    def version(self) -> str:
+        if self.build_config:
+            value = self.build_config['language']['version']


Is there a reason that we aren't using the dotted key support that seems to be provided by BuildConfig?

mgorny · 2025-05-21T18:47:13Z

mesonbuild/dependencies/python.py

+            self.version = self.build_config['language']['version']
+            self.platform = self.build_config['platform']
+            self.is_freethreaded = 't' in self.build_config['abi']['flags']
+            self.link_libpython = self.build_config['libpython']['link_extensions']


Per my note above, I suppose this should be:

Suggested change

self.link_libpython = self.build_config['libpython']['link_extensions']

self.link_libpython = self.build_config.get('libpython.link_extensions', False)

mgorny · 2025-05-22T15:50:52Z

@FFY00, would you mind me taking this over? @rgommers mentioned you don't have time to work on it, and this certainly falls within my interests.

mgorny · 2025-05-24T14:31:59Z

Hmm, another thought: should we disable/skip pkg-config detection if c_api.pkgconfig_path is missing?

mgorny · 2025-05-26T07:09:04Z

mesonbuild/dependencies/python.py

+            if not path:
+                raise DependencyException('Python does not provide a dynamic libpython library')
+            libdirs = [os.path.dirname(path)]
+            libname = os.path.basename(path)


This seems to be incorrect. FWICS find_library() can't handle the full filename (e.g. libpython3.14.so, but needs the bare python3.14.

FFY00 · 2025-05-28T07:25:33Z

@FFY00, would you mind me taking this over? @rgommers mentioned you don't have time to work on it, and this certainly falls within my interests.

Sorry for the delay getting back to you!

No worries at all from me! In fact, that would make me very happy 😊. I have been suffering from burnout and haven't been able to make productive progress on my own in this project, so having someone pick it back up would make me feel a lot better.

While I don't think I am in a position where I can productively drive the development, I am more than happy to collaborate and help navigate the complexity around this feature.

mgorny · 2025-05-28T18:55:23Z

Thanks! I've opened #14657 to continue the work — unless you have a better idea how to proceed.

mgorny · 2025-05-29T15:53:01Z

mesonbuild/dependencies/python.py

+            pkg_libdir = installation.info['variables'].get('LIBPC')
+            pkg_libdir_origin = 'LIBPC' if pkg_libdir else 'the default paths'
+        mlog.debug(f'Searching for {pkg_libdir!r} via pkgconfig lookup in {pkg_libdir_origin}')
+        pkgconfig_paths = [pkg_libdir] if pkg_libdir else []


Okay, the assumption that we check default pkg-config dirs when LIBPC is not set introduced a regression — now we end up taking python-3.11.pc on PyPy, so effectively PyPy extensions start being built against the headers from CPython.

The way I see it, we have two options here: either we restore the previous logic that pkg-config is not used when LIBPC is not set, or we unconditionally skip pkg-config when not targeting CPython (in the end, the pkg_name assumption is CPython-only).

I'd lean towards restoring the previous logic with a descriptive code comment, so it's easier to understand for whoever looks at this next. I'm thinking either change is reasonable; restoring the logic may be better just in case something like crossenv, GraalPython or Pyodide relies on this logic.

It would be good to make sure all of this works with crossenv. Even if meson can get all of the python information via PEP 739 support, when using pkgconf-pypi there isn't yet a way for a package that depends on pkg-config information published by another wheel to retrieve it without using crossenv.

I should probably add a crossenv testcase to pkgconf-pypi.

Reference: pypackaging-native/pkgconf-pypi#74

I'm sorry but I'm thoroughly confused by what you're saying. Meson here looks up the .pc files for Python itself. I don't think there is a case where this logic would be used for any other package or any other purpose.

Also, the discussion is about PyPy, not PyPI.

My apologies for the confusion. It wasn't directly related to your comment, but rgommers mention of crossenv -- the TL;DR of my comment is "It would be good to make sure all of this works with crossenv.". The rest of the comment explains a situation where one might still need crossenv after this PR is merged.

See mesonbuild#13945 (comment) Signed-off-by: Michał Górny <mgorny@quansight.com>

rgommers mentioned this pull request Dec 10, 2024

Prototype support for --python.target-config #12193

Closed

FFY00 force-pushed the pep-739-prototype branch from ea9aed7 to 697e20c Compare January 20, 2025 21:27

FFY00 force-pushed the pep-739-prototype branch 2 times, most recently from 64f9781 to e9deda3 Compare January 20, 2025 23:42

eli-schwartz reviewed Jan 21, 2025

View reviewed changes

test cases/unit/125 python extension/build-details.json Outdated Show resolved Hide resolved

mesonbuild/modules/python.py Outdated Show resolved Hide resolved

FFY00 force-pushed the pep-739-prototype branch 5 times, most recently from 43fcae5 to f30406d Compare January 26, 2025 05:56

eli-schwartz reviewed Feb 5, 2025

View reviewed changes

dcbaker reviewed Feb 6, 2025

View reviewed changes

mesonbuild/dependencies/pkgconfig.py Outdated Show resolved Hide resolved

dcbaker reviewed Feb 6, 2025

View reviewed changes

mesonbuild/dependencies/python.py Outdated Show resolved Hide resolved

dcbaker reviewed Feb 6, 2025

View reviewed changes

unittests/allplatformstests.py Show resolved Hide resolved

FFY00 added 2 commits March 3, 2025 13:30

python: add a python.build_config option (PEP 739)

a211c3f

Signed-off-by: Filipe Laíns <lains@riseup.net>

tests: add test for the Python build config

7c0ca08

Signed-off-by: Filipe Laíns <lains@riseup.net>

FFY00 force-pushed the pep-739-prototype branch 4 times, most recently from bc6ef41 to b5e8baa Compare March 3, 2025 13:47

FFY00 added 2 commits March 3, 2025 13:51

python: move pkg-config discovery introspection logic to PythonPkgCon…

730c157

…figDependency Signed-off-by: Filipe Laíns <lains@riseup.net>

python: validate python.build_config version and expand relative paths

13b89ae

Signed-off-by: Filipe Laíns <lains@riseup.net>

FFY00 force-pushed the pep-739-prototype branch from b5e8baa to 13b89ae Compare March 3, 2025 13:51

eli-schwartz mentioned this pull request Apr 8, 2025

meson discards the pre-defined environment variable PKG_CONFIG_LIBDIR when finding Python dependency #14461

Open

mgorny reviewed May 21, 2025

View reviewed changes

FFY00 mentioned this pull request May 22, 2025

build-details.json v1.1 pypa/packaging.python.org#1862

Open

1 task

mgorny reviewed May 26, 2025

View reviewed changes

mgorny mentioned this pull request May 28, 2025

Prototype support for PEP 739 (continued) #14657

Open

mgorny reviewed May 29, 2025

View reviewed changes

mgorny added a commit to mgorny/meson that referenced this pull request May 29, 2025

python: Use a reduce pattern in PythonBuildConfig.__getitem__()

a5e27f1

See mesonbuild#13945 (comment) Signed-off-by: Michał Górny <mgorny@quansight.com>

virtuald mentioned this pull request Jun 2, 2025

Add cross-compile testcase pypackaging-native/pkgconf-pypi#75

Open

mgorny added a commit to mgorny/meson that referenced this pull request Jun 10, 2025

python: Use a reduce pattern in PythonBuildConfig.__getitem__()

e2a34f7

See mesonbuild#13945 (comment) Signed-off-by: Michał Górny <mgorny@quansight.com>

mgorny added a commit to mgorny/meson that referenced this pull request Jun 24, 2025

python: Use a reduce pattern in PythonBuildConfig.__getitem__()

2d01e48

See mesonbuild#13945 (comment) Signed-off-by: Michał Górny <mgorny@quansight.com>

mgorny added a commit to mgorny/meson that referenced this pull request Jul 8, 2025

python: Use a reduce pattern in PythonBuildConfig.__getitem__()

72c175b

See mesonbuild#13945 (comment) Signed-off-by: Michał Górny <mgorny@quansight.com>

		extra_paths: T.List[str] = self.env.coredata.optstore.get_value(key)[:]
		extra_paths: T.List[str] = self.env.coredata.optstore.get_value(key)[:] + self.extra_paths

	self.link_libpython = self.build_config['libpython']['link_extensions']
	self.link_libpython = self.build_config.get('libpython.link_extensions', False)

Uh oh!

Prototype support for PEP 739 #13945

Are you sure you want to change the base?

Prototype support for PEP 739 #13945

Uh oh!

Conversation

FFY00 commented Nov 22, 2024

Uh oh!

FFY00 commented Jan 20, 2025

Uh oh!

eli-schwartz left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

FFY00 commented Jan 21, 2025

Uh oh!

FFY00 commented Feb 5, 2025

Uh oh!

eli-schwartz left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

virtuald commented Mar 28, 2025

Uh oh!

FFY00 commented Mar 28, 2025

Uh oh!

mgorny left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mgorny commented May 22, 2025

Uh oh!

mgorny commented May 24, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

FFY00 commented May 28, 2025

Uh oh!

mgorny commented May 28, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mgorny Jun 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mgorny Jun 2, 2025 •

edited

Loading