meson discards the pre-defined environment variable PKG_CONFIG_LIBDIR when finding Python dependency

[lbmeng]

When cross-compiling NumPy against aarch64 from an x86_64 build machine, I invoked meson setup like this:

PKG_CONFIG_LIBDIR=/path/to/host/sysroot/usr/lib/pkgconfig \
    PKG_CONFIG_SYSROOT_DIR=/path/to/host/sysroot \
    PKG_CONFIG_SYSTEM_INCLUDE_PATH=/path/to/host/sysroot/usr/include \
    PKG_CONFIG_SYSTEM_LIBRARY_PATH=/path/to/host/sysroot/usr/lib \
    meson setup build_dir --cross-file=/path/to/my/cross-file.txt

with a cross-file.txt that does NOT have pkg_config_libdir in the [properties] section. I got the following output in the message in the meson-log.txt file:

Found pkg-config: YES (/path/to/build/x86_64-linux/usr/bin/pkg-config) 0.29.2
Determining dependency 'python-3.10' with pkg-config executable '/path/to/build/x86_64-linux/usr/bin/pkg-config'
env[PKG_CONFIG_SYSROOT_DIR]: /path/to/host/sysroot/sysroot
env[PKG_CONFIG_SYSTEM_LIBRARY_PATH]: /path/to/host/sysroot/usr/lib
env[PKG_CONFIG_SYSTEM_INCLUDE_PATH]: /path/to/host/sysroot/usr/include
env[PKG_CONFIG_LIBDIR]: /usr/lib/pkgconfig
env[PKG_CONFIG_PATH]:
env[PKG_CONFIG]: /path/to/build/x86_64-linux/usr/bin/pkg-config
-----------
Called: `/path/to/build/x86_64-linux/usr/bin/pkg-config --modversion python-3.10` -> 1
stderr:
Package python-3.10 was not found in the pkg-config search path.
Perhaps you should add the directory containing `python-3.10.pc'
to the PKG_CONFIG_PATH environment variable
No package 'python-3.10' found
-----------
"python-3.10" could not be found in LIBPC, this is likely due to a relocated python installation
Searching for 'python-3.10' via fallback pkgconfig lookup in default paths

-----------

This will add "-I/usr/include/python3.10 -I/path/to/build/x86_64-linux/usr/include/python3.10" when compiling NumPy sources, which is apparently wrong.

After I added pkg_config_libdir in the [properties] section like below:

[properties]
pkg_config_libdir = '/path/to/host/sysroot/usr/lib/pkgconfig'

Then I got the following in the meson log:

env[PKG_CONFIG_LIBDIR]: /path/to/host/sysroot/usr/lib/pkgconfig
env[PKG_CONFIG]: /path/to/build/x86_64-linux/usr/bin/pkg-config
-----------
Called: `/path/to/build/x86_64-linux/usr/bin/pkg-config --modversion python-3.10` -> 0
stdout:
3.10
-----------

Is this behavior by design?

[lbmeng]

The meson documentation only mentions such for pkg_config_libdir:

Since 0.54.0 The pkg_config_libdir property may point to a list of path used internally by Meson to set the 
PKG_CONFIG_LIBDIR environment variable for pkg-config. This prevents pkg-config from searching cross
dependencies in system directories.

My understanding when I first looked at this, was that if cross-file.txt does not have pkg_config_libdir property, the the PKG_CONFIG_LIBDIR environment variable will be used as it is. But the real fact is that pkg_config_libdir property has to be set in all situations, otherwise if unset, the pre-existing PKG_CONFIG_LIBDIR environment variable's value will be erased.

[lbmeng]

Testing exact the same command:

PKG_CONFIG_LIBDIR=/path/to/host/sysroot/usr/lib/pkgconfig \
    PKG_CONFIG_SYSROOT_DIR=/path/to/host/sysroot \
    PKG_CONFIG_SYSTEM_INCLUDE_PATH=/path/to/host/sysroot/usr/include \
    PKG_CONFIG_SYSTEM_LIBRARY_PATH=/path/to/host/sysroot/usr/lib \
    meson setup build_dir --cross-file=/path/to/my/cross-file.txt

to find libpcre library via pkg-config, without pkg_config_libdir in the [properties] section of cross-file.txt, the libpcre can be found by pkg-config. Here is the log:

Found pkg-config: YES (/path/to/build/x86_64-linux/usr/bin/pkg-config) 0.29.2
Determining dependency 'libpcre2-8' with pkg-config executable '/path/to/build/x86_64-linux/usr/bin/pkg-config'
env[PKG_CONFIG_SYSTEM_LIBRARY_PATH]: /path/to/host/sysroot/usr/lib
env[PKG_CONFIG_SYSROOT_DIR]: /path/to/host/sysroot
env[PKG_CONFIG_LIBDIR]: /path/to/host/sysroot/usr/lib/pkgconfig
env[PKG_CONFIG_SYSTEM_INCLUDE_PATH]: /path/to/host/sysroot/usr/include
env[PKG_CONFIG_PATH]:
env[PKG_CONFIG]: /path/to/build/x86_64-linux/usr/bin/pkg-config
-----------
Called: `/path/to/build/x86_64-linux/usr/bin/pkg-config --modversion libpcre2-8` -> 0
stdout:
10.44
-----------

So there is a inconsistency. Python finding package logic seems to be the only one who ignores PKG_CONFIG_LIBDIR environment variable when pkg_config_libdir is not specified in the [properties].

[eli-schwartz]

When cross compiling python modules there are a number of settings you need to successfully cross.

In addition to handling the correct compile/link flags you also need to know what the extension module file extension is, as well as details such as whether you're using PyPy, if the python installation has debian-specific quirks, whether your build is free-threaded, and more. There are python PEPs to make that introspectable without literally executing the python interpreter and printing importlib properties, but it's not finalized.

Meson allows you to select an arbitrary python to build against, independent of "what version of Python happened to be used to install meson itself". But it still needs to run that python well enough to execute a script (python_info.py) which dumps that information.

...

Meson also tries to guarantee that the python dependency found by import('python').find_installation().dependency() refers to the same thing as find_installation() itself. Which means that it e.g. checks and adds the pkgconfig directory from cpython, to the pkgconfig search path.

When the introspection PEP is complete and deployed by cpython / distros that install cpython and supported in meson, you'll be able to pass the build-details.json file from your cross python. Until then you need a runnable python -- even if only for discovering the extension module filename suffix. One thing you can do is define python as qemu-user /path/to/cross-python.

[eli-schwartz]

See e.g. #13945

[lbmeng]

While I appreciate that PEP 739 proposes a generic solution to Python cross-compilation challenges, I'm currently encountering a specific issue when cross-compiling NumPy, and it exposes an issue of the meson-build itself.

The NumPy project has well-constructed meson.build files that successfully handle all typical cross-compilation requirements. However, this process has revealed what appears to be an inconsistency in meson-build itself - one that's independent of Python cross-compilation complexities.

My previous example demonstrates this clearly: Meson properly respects the PKG_CONFIG_LIBDIR environment variable when searching for packages (like libpcre2-8) via pkg-config, but makes an exception for Python dependencies. This behavioral discrepancy suggests a framework-level issue rather than a Python-specific one.

After investigation, I've identified the likely problematic code section:
https://github.com/mesonbuild/meson/blob/1.7.2/mesonbuild/dependencies/python.py#L408

Is this differential treatment of Python packages in pkg-config resolution intentional design behavior? The current implementation appears to handle Python dependencies differently than other packages when processing PKG_CONFIG_LIBDIR, which seems inconsistent with Meson's standard package resolution logic.

Based on my understanding of the documentation, it seems to convey the following points:

• When the pkg_config_libdir property is explicitly defined in the cross-file.txt, the environment variable PKG_CONFIG_LIBDIR should be overridden with the value specified in this property.
• If the pkg_config_libdir property is omitted, the existing configuration should remain unchanged.

Testing result shows that the above points are true when finding packages via pkg-config. But not for Python! The actual finding python behavior appears to deviate from this expectation. If this discrepancy is indeed intentional, it would be advisable to update the documentation to clarify.

[eli-schwartz]

Is this differential treatment of Python packages in pkg-config resolution intentional design behavior?

I believe I already answered that question.

Again, the current implementation was written by someone who chose to intentionally design it in this way, under the belief that

one that's independent of Python cross-compilation complexities

is incorrect and it does matter for cross compilation (and for targeting python versions installed in /opt or $HOME).

Recall that this does not use dependency('python3.13-embed') at all. It uses python_installation.dependency(), an entirely different function and one which derives its "what to find" from the object it is attached to (namely, a python interpreter binary).

While it's certainly possible to discuss whether this behavior is optimal, there isn't much to say about whether it's intentional. The latter is a question of historical accuracy. It is historically accurate to say that the current behavior was intentional.

[eli-schwartz]

/cc @rgommers

[rgommers]

This does seem like something that should be fixed/changed. I'd be more confident of doing that after we have PEP 739 support though, since it's hard (at least for me) to reason through what the effects of this kind of change will be. Once gh-13945 is in, we should be able to have cross-compile CI jobs for NumPy et al.