Can Pex be used in place of pip download when targeting a foreign interpreter?

[jsirois]

This discussion is forked from pypa/pip#11664 (comment)

[jsirois]

Pex [...] uses Pip under the covers, but patches it to implement this feature.

Cool! I tried exploring this but didn't get very far. It seems pex is not really a pip stand-in, from the perspective of a first-time user. Is there an example analogue to pip download -r requirements.txt --platform= ... that gives the same result as pip except setting and using the environment markers as expected?

Pex is definitely not a drop-in for Pip - its primary use case revolves around shipping whole environments in a single zipapp (PEX) file; so depending on what you need to accomplish it may or may not work for your needs. If you exactly need to ship the distributions (sdists and wheels) needed by a foreign platform, Pex won't work. If you just need to ~pre-build a venv for a foreign target, that can work.

For example, running on Linux x86-64 and targeting Linux aarch64:

# Characterize the foreign platform target - a 1 time operation:
:; docker run --rm -it --platform linux/aarch64 python:3.13 bash -c 'pip -q install --root-user-action ignore pex && pex3 interpreter inspect --markers --tags' > linux-aarch64-complete-platform.json

# Now use the foreign target platform description to resolve a hermetic environment for it:
:; cat <<EOF > requirements.txt
cowsay
psutil
EOF
:; pex -r requirements.txt --complete-platform linux-aarch64-complete-platform.json --venv -o example.pex
:; zipinfo example.pex | grep METADATA
?rw-r--r--  2.0 unx     5581 b- defN 80-Jan-01 00:00 .deps/cowsay-6.1-py3-none-any.whl/cowsay-6.1.dist-info/METADATA
?rw-r--r--  2.0 unx    22315 b- defN 80-Jan-01 00:00 .deps/psutil-7.0.0-cp36-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl/psutil-7.0.0.dist-info/METADATA

:; python -c 'import os; print(os.uname())' && docker run --rm -it --platform linux/aarch64 -v $PWD/example.pex:/example.pex python:3.13 /example.pex -c 'import cowsay, psutil, os; cowsay.tux(f"{psutil.Process()}\n{os.uname()}")'
posix.uname_result(sysname='Linux', nodename='gill-xps', release='6.14.0-22-generic', version='#22-Ubuntu SMP PREEMPT_DYNAMIC Wed May 21 15:01:51 UTC 2025', machine='x86_64')
  _________________________________________________
 /                                                 \
| psutil.Process(pid=1, name='python', status='runn |
| ing', started='23:00:07')                         |
| posix.uname_result(sysname='Linux', nodename='4fa |
| a4504fcbb', release='6.14.0-22-generic', version= |
| '#22-Ubuntu SMP PREEMPT_DYNAMIC Wed May 21 15:01: |
| 51 UTC 2025', machine='aarch64')                  |
 \                                                 /
  =================================================
                                                      \
                                                       \
                                                        \
                                                         .--.
                                                        |o_o |
                                                        |:_/ |
                                                       //   \ \
                                                      (|     | )
                                                     /'\_   _/`\
                                                     \___)=(___/

[hackowitz-af]

Cool, thanks! I think what I was missing is that I can unzip the *.pex file to get the *.whls

[jsirois]

Cool, thanks! I think what I was missing is that I can unzip the *.pex file to get the *.whls

So, I was hoping you'd explain your use case a bit more. Do you actually need wheels? Many use cases just need a venv to run something in, and the intermediate products of wheels or sdists to build the venv with are incidental.

That said - if you do really need wheels:

:; docker run --rm -it --platform linux/aarch64 -v $PWD/example.pex:/example.pex python:3.13 bash -c 'PEX_TOOLS=1 /example.pex repository extract --dest-dir ./wheels && ls -l ./wheels/'
cowsay 6.1: Repacking wheel as /wheels/cowsay-6.1-py3-none-any.whl...OK
psutil 7.0.0: Repacking wheel as /wheels/psutil-7.0.0-cp36-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.whl...OK
total 304
-rw-r--r-- 1 root root  26362 Jun 25 16:15 cowsay-6.1-py3-none-any.whl
-rw-r--r-- 1 root root 279709 Jun 25 16:15 psutil-7.0.0-cp36-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.whl

Or, if you want a traditional venv instead of a PEX file:

:; docker run --rm -it --platform linux/aarch64 -v $PWD/example.pex:/example.pex python:3.13 bash -c 'PEX_TOOLS=1 /example.pex venv --pip ./.venv && ./.venv/bin/pip list'
Package Version
------- -------
cowsay  6.1
pip     25.1.1
psutil  7.0.0

N.B.: That neither of the prior exercises uses the network. All code and artifacts are self-contained in the PEX zip file to do all this.

[hackowitz-af]

So, I was hoping you'd explain your use case a bit more

My organization, and the defense community at large, use several air-gapped networks to isolate and compartmentalize sensitive data. We have set up self-hosted package indexes on some of these intranets, where we want to mirror select approved packages from PyPI, as well as distribute some internally-developed packages. Each air-gapped network has it's own environment, often Anaconda, often Windows, and anywhere up to 5 years out of date - although other platforms are getting more popular through some of our self-hosted web IDEs, VMs, and containerized apps.

The only way to bring data onto an air-gapped network is a one-way file transfer on disposable media. Usually burning a CD from an internet-connected computer, getting it approved by security, transferred by an administrator, then shredded. There is virtually no approved way to get anything off an air-gapped network and back to anything internet-connected. All for good reason - security is the game.

The way we currently collect packages to transfer is by keeping a requirements.txt of the packages we want and using pip download to collect them and all their dependencies. We get these approved and transferred, then use twine to put them on a private package index. To make sure we handle a good breadth of environments, we download with several options:

for platform in win_amd64 manylinux2014_x86_64; do
    for version in 3.8 3.9 3.10 3.11 3.12; do
        pip download \
            -r requirements.txt \
            --dest=dist \
            --only-binary=:all: \
            --platform=$platform \
            --python-version=$version
    done
done

But this runs into problems with sdist packages (or any package that has sdist dependencies), and any packages that use environment markers. It also assumes that users are within this subset of environments, which is generally a safe assumption but not necessarily true.

[hackowitz-af]

The short version is I'm not trying to create a virtual environment per se, I'm distributing packages which users can install into their own environments

[jsirois]

Gotcha. Thanks for the details - that helps alot. The "often Windows" likely still rules out Pex, but I'll just run you down what otherwise would work with Pex today.

In both cases detailed below, when using --platform or --complete-platform to target a foreign platform, Pex will "yolo" when a required distribution is only available in sdist form: It will attempt to build the sdist into a wheel on the local machine, and if that succeeds, it will check if the wheel is compatible with the --platform or --complete-platform and fail if not.

1. Create a lock file and curl the downloads from it - then create an archive to sneakernet on your own.

   # Create the lock for the foreign platform:
   :; pex3 lock create -r requirements.txt --complete-platform linux-aarch64-complete-platform.json --indent 2 -o linux-aarch64.lock.json
   # Extract the hashes and urls of the preferred artifacts (.artifact[0] is always the most specific wheel if available; otherwise degrading to the sdist if that's all there is):
   :; jq '.locked_resolves[0].locked_requirements[] | .artifacts[0]' linux-aarch64.lock.json 
   {
     "algorithm": "sha256",
     "hash": "274b1e6fc1b966d53976333eb90ac94cb07a450a700b455af9fbdf882244b30a",
     "url": "https://files.pythonhosted.org/packages/f1/13/63c0a02c44024ee16f664e0b36eefeb22d54e93531630bd99e237986f534/cowsay-6.1-py3-none-any.whl"
   }
   {
     "algorithm": "sha256",
     "hash": "a5f098451abc2828f7dc6b58d44b532b22f2088f4999a937557b603ce72b1993",
     "url": "https://files.pythonhosted.org/packages/eb/a2/709e0fe2f093556c17fbafda93ac032257242cabcc7ff3369e2cb76a97aa/psutil-7.0.0-cp36-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl"
   }
   # Script up some `curl` / `sha256sum` to download these.
   ...

2. Create a repository PEX - sneakernet it - and have it serve up a find-links repo on the target airgapped host.

   # Create the repository PEX with `--include-tools` to get access to PEX_TOOLS=1 commands:
   :; pex -r requirements.txt --complete-platform linux-aarch64-complete-platform.json -o linux-aarch64.repository.pex --include-tools
   
   # Use the `repository extract` tool to extract the wheels from the PEX and serve them up as a find-links repo:
   :; docker run --rm --detach -t --platform linux/aarch64 -v $PWD/linux-aarch64.repository.pex:/linux-aarch64.repository.pex -p 8888:8888 python:3.13 bash -c 'PEX_TOOLS=1 /linux-aarch64.repository.pex repository extract --dest-dir find-links-repo --serve --port 8888'
   b18cb05fdbd553cec34fca8ab041b47958ee36e23b5a0a8a9bb15a364bc8f550
   
   :; docker logs -f b18cb05fdbd553cec34fca8ab041b47958ee36e23b5a0a8a9bb15a364bc8f550
   cowsay 6.1: Repacking wheel as /find-links-repo/cowsay-6.1-py3-none-any.whl...OK
   psutil 7.0.0: Repacking wheel as /find-links-repo/psutil-7.0.0-cp36-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.whl...OK
   Serving find-links repo of /linux-aarch64.repository.pex via /find-links-repo at http://localhost:8888
   
   :; curl http://localhost:8888
   <!DOCTYPE HTML>
   <html lang="en">
   <head>
   <meta charset="utf-8">
   <title>Directory listing for /</title>
   </head>
   <body>
   <h1>Directory listing for /</h1>
   <hr>
   <ul>
   <li><a href="cowsay-6.1-py3-none-any.whl">cowsay-6.1-py3-none-any.whl</a></li>
   <li><a href="psutil-7.0.0-cp36-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.whl">psutil-7.0.0-cp36-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.whl</a></li>
   </ul>
   <hr>
   </body>
   </html>

Of these options, 1 is most in line with what you said you want to do, but it relies on knowing details of the Pex lock file format. I think it would make sense to add a new pex3 download [ --lock X | --requirements Y ] --dest Z to support downloading artifacts directly which would eliminate the jq / curl / sha256sum scripting, but that would still leave you without official Windows support. I've filed #2790 for this idea.

[jsirois]

The pex3 download support is here: #2791
I should have a release out later today in case this is of use to you. I added --pylock support for interop. For example, here's me using uv to export the Pex uv.lock to pylock.toml, then using this new feature to download all distributions needed for Pex development, but on a foreign platform (I'm on Linux x86-64, not Linux aarch64 or Windows x86-64 (whose example below relies on #2794)):

# Get a PEP-751 pylock.toml for the Pex project:
:; uv export -q --no-emit-project -o pylock.toml

# Now download all the dists needed by Pex for a different Linux platform (uv locks are "universal" and contain distributions for all platforms):
:; python -mpex.cli download --pip-version latest --pylock pylock.toml --complete-platform ../pexcz/linux-aarch64-complete-platform.json -d foreign-dists-via-uv-exported-lock/linux-aarch64

# This Windows download uses `--platform` which relies on #2794:
:; python -mpex.cli download --pip-version latest --pylock pylock.toml --platform win_amd64-cp-3.13-cp3.13 -d foreign-dists-via-uv-exported-lock/win-amd64

:; comm <(ls -1 foreign-dists-via-uv-exported-lock/linux-aarch64/) <(ls -1 foreign-dists-via-uv-exported-lock/win-amd64/)
		aioconsole-0.8.1-py3-none-any.whl
	alabaster-0.7.16-py3-none-any.whl
alabaster-1.0.0-py3-none-any.whl
		ansicolors-1.1.8-py2.py3-none-any.whl
		anyio-4.9.0-py3-none-any.whl
		appdirs-1.4.4-py2.py3-none-any.whl
		argon2_cffi-23.1.0-py3-none-any.whl
argon2_cffi_bindings-21.2.0-cp36-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
	argon2_cffi_bindings-21.2.0-cp36-abi3-win_amd64.whl
		attrs-21.5.0.dev0.zip
		autoflake-1.7.8-py3-none-any.whl
		babel-2.17.0-py3-none-any.whl
		beautifulsoup4-4.13.3-py3-none-any.whl
		black-21.12b0-py3-none-any.whl
Brotli-1.1.0-cp313-cp313-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
	Brotli-1.1.0.tar.gz
		build-1.2.2.post1-py3-none-any.whl
		certifi-2025.1.31-py3-none-any.whl
cffi-1.17.1-cp313-cp313-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
	cffi-1.17.1.tar.gz
		chardet-5.2.0-py3-none-any.whl
charset_normalizer-3.4.1-cp313-cp313-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
	charset_normalizer-3.4.1-py3-none-any.whl
		click-8.0.1-py3-none-any.whl
	colorama-0.4.6-py2.py3-none-any.whl
		coloredlogs-15.0.1-py2.py3-none-any.whl
cryptography-42.0.8-cp39-abi3-manylinux_2_28_aarch64.whl
	cryptography-42.0.8-cp39-abi3-win_amd64.whl
		cssselect2-0.8.0-py3-none-any.whl
		DataProperty-1.1.0-py3-none-any.whl
		defusedxml-0.7.1-py2.py3-none-any.whl
		Deprecated-1.2.18-py2.py3-none-any.whl
		dev_cmd-0.31.2-py3-none-any.whl
	devpi_common-3.7.2-py2.py3-none-any.whl
devpi_common-4.0.4-py3-none-any.whl
		devpi_server-6.12.0-py3-none-any.whl
	docutils-0.19-py3-none-any.whl
docutils-0.21.2-py3-none-any.whl
		execnet-2.1.1-py3-none-any.whl
		filelock-3.18.0-py3-none-any.whl
fonttools-4.57.0-cp313-cp313-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
	fonttools-4.57.0-py3-none-any.whl
	furo-2023.3.27-py3-none-any.whl
furo-2024.8.6-py3-none-any.whl
		h11-0.14.0-py3-none-any.whl
		html5lib-1.1-py2.py3-none-any.whl
		httpcore-1.0.7-py3-none-any.whl
		httpx-0.28.1-py3-none-any.whl
		humanfriendly-10.0-py2.py3-none-any.whl
		hupper-1.12.1-py3-none-any.whl
		idna-3.10-py3-none-any.whl
		imagesize-1.4.1-py2.py3-none-any.whl
		iniconfig-2.1.0-py3-none-any.whl
		isort-5.10.1-py3-none-any.whl
		itsdangerous-2.2.0-py3-none-any.whl
		jinja2-3.1.6-py3-none-any.whl
		lazy-1.6-py2.py3-none-any.whl
legacy_cgi-2.6.3-py3-none-any.whl
libcst-1.7.0-cp313-cp313-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
	libcst-1.7.0.tar.gz
	libsass-0.23.0-cp38-abi3-win_amd64.whl
libsass-0.23.0.tar.gz
	linkify_it_py-1.0.3-py3-none-any.whl
linkify_it_py-2.0.3-py3-none-any.whl
	markdown_it_py-2.2.0-py3-none-any.whl
markdown_it_py-3.0.0-py3-none-any.whl
		MarkupSafe-2.1.5.tar.gz
		mbstrdecoder-1.1.4-py3-none-any.whl
	mdit_py_plugins-0.3.5-py3-none-any.whl
mdit_py_plugins-0.4.2-py3-none-any.whl
		mdurl-0.1.2-py3-none-any.whl
	mypy-0.971-py3-none-any.whl
		mypy_extensions-1.0.0-py3-none-any.whl
	myst_parser-1.0.0-py3-none-any.whl
myst_parser-4.0.1-py3-none-any.whl
	packaging-20.9-py2.py3-none-any.whl
packaging-24.2-py3-none-any.whl
packaging_legacy-23.0.post0-py3-none-any.whl
		passlib-1.7.4-py2.py3-none-any.whl
		PasteDeploy-3.1.0-py3-none-any.whl
		pathspec-0.12.1-py3-none-any.whl
		pathvalidate-3.2.3-py3-none-any.whl
		pexpect-4.9.0-py2.py3-none-any.whl
pillow-11.1.0-cp313-cp313-manylinux_2_28_aarch64.whl
	pillow-11.1.0.tar.gz
		pip-24.0-py3-none-any.whl
		plaster-1.1.2-py2.py3-none-any.whl
		plaster_pastedeploy-1.0.1-py2.py3-none-any.whl
		platformdirs-4.3.7-py3-none-any.whl
		pluggy-1.5.0-py3-none-any.whl
psutil-7.0.0-cp36-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
	psutil-7.0.0-cp37-abi3-win_amd64.whl
		ptyprocess-0.7.0-py2.py3-none-any.whl
		py-1.11.0-py2.py3-none-any.whl
		pycparser-2.22-py3-none-any.whl
		pydyf-0.10.0-py3-none-any.whl
		pyflakes-2.4.0-py2.py3-none-any.whl
		PyGithub-2.6.1-py3-none-any.whl
		pygments-2.19.1-py3-none-any.whl
		PyJWT-2.10.1-py3-none-any.whl
PyNaCl-1.5.0-cp36-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.manylinux_2_24_aarch64.whl
	PyNaCl-1.5.0-cp36-abi3-win_amd64.whl
	pyparsing-3.2.3-py3-none-any.whl
		pyphen-0.17.2-py3-none-any.whl
		pyproject_hooks-1.2.0-py3-none-any.whl
		pyramid-2.0.2-py3-none-any.whl
	pyreadline3-3.5.4-py3-none-any.whl
		pytablewriter-1.2.1-py3-none-any.whl
		pytest-8.3.5-py3-none-any.whl
		pytest_md_report-0.6.3-py3-none-any.whl
		pytest_retry-1.7.0-py3-none-any.whl
		pytest_xdist-3.6.1-py3-none-any.whl
		python_dateutil-2.9.0.post0-py2.py3-none-any.whl
		pytz-2025.2-py2.py3-none-any.whl
PyYAML-6.0.2-cp313-cp313-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
	pyyaml-6.0.2.tar.gz
		repoze.lru-0.7-py3-none-any.whl
		requests-2.32.3-py3-none-any.whl
roman_numerals_py-3.1.0-py3-none-any.whl
		ruamel.yaml-0.18.10-py3-none-any.whl
	ruamel.yaml.clib-0.2.12.tar.gz
		setuptools-50.3.2-py3-none-any.whl
		six-1.17.0-py2.py3-none-any.whl
		sniffio-1.3.1-py3-none-any.whl
		snowballstemmer-2.2.0-py2.py3-none-any.whl
		soupsieve-2.6-py3-none-any.whl
	Sphinx-5.1.1-py3-none-any.whl
sphinx-8.2.3-py3-none-any.whl
		sphinx_basic_ng-1.0.0b2-py3-none-any.whl
		sphinxcontrib_applehelp-2.0.0-py3-none-any.whl
		sphinxcontrib_devhelp-2.0.0-py3-none-any.whl
		sphinxcontrib_htmlhelp-2.1.0-py3-none-any.whl
		sphinxcontrib_jsmath-1.0.1-py2.py3-none-any.whl
		sphinxcontrib_qthelp-2.0.0-py3-none-any.whl
		sphinxcontrib_serializinghtml-2.0.0-py3-none-any.whl
		sphinx_simplepdf-1.6.0-py3-none-any.whl
		strictyaml-1.7.3-py3-none-any.whl
		subprocess32-3.5.4.tar.gz
		tabledata-1.3.4-py3-none-any.whl
		tcolorpy-0.1.7-py3-none-any.whl
		tinycss2-1.4.0-py3-none-any.whl
		toml-0.10.2-py2.py3-none-any.whl
tomli-2.2.1-cp313-cp313-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
	tomli-2.2.1-py3-none-any.whl
		translationstring-1.4-py2.py3-none-any.whl
	typed_ast-1.5.5.tar.gz
		typepy-1.3.4-py3-none-any.whl
		types_appdirs-1.4.3.5-py3-none-any.whl
		types_docutils-0.20.0.20240310-py3-none-any.whl
		types_mock-5.1.0.20240106-py3-none-any.whl
		types_pexpect-4.9.0.20240207-py3-none-any.whl
		types_PyYAML-6.0.12.12-py3-none-any.whl
		types_setuptools-69.1.0.20240302-py3-none-any.whl
		types_toml-0.10.5-py3-none-any.whl
		typing_extensions-4.13.1-py3-none-any.whl
		uc_micro_py-1.0.3-py3-none-any.whl
		urllib3-2.3.0-py3-none-any.whl
		venusian-3.1.1-py3-none-any.whl
		waitress-3.0.2-py3-none-any.whl
		weasyprint-62.3-py3-none-any.whl
		webencodings-0.5.1-py2.py3-none-any.whl
		WebOb-1.8.9-py2.py3-none-any.whl
		wheel-0.35.1-py2.py3-none-any.whl
wrapt-1.17.2-cp313-cp313-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
	wrapt-1.17.2-py3-none-any.whl
		zope.deprecation-5.1-py3-none-any.whl
zope.interface-7.2-cp313-cp313-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
	zope.interface-7.2.tar.gz
zopfli-0.2.3.post1-cp313-cp313-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
	zopfli-0.2.3.post1.tar.gz

[jsirois]

With the release of Pex 2.41.0 I'll mark this as answered. That release supports pex3 download directly allowing 1st class support for downloading distributions needed by foreign platforms via --platform or --complete-platform.