add support to Python easyblock for conditionally adding patches when EasyBuild is configured to filter `$LD_LIBRARY_PATH` #3860

casparvl · 2025-07-30T09:34:18Z

This change introduces extra easyblock-specific configuration items. They allow us to define patches that should be applied conditionally if EasyBuild is configured to filter LD_LIBRARY_PATH. Since these patches (see Should be used together with: easybuilders/easybuild-easyconfigs#23499) are quite fundamental (change ctypes functions) we prefer them to only be applied in the scenario where they are really needed.

Replaces: #3798
Should be used together with: easybuilders/easybuild-easyconfigs#23499

edit: relevant EESSI support issue: https://gitlab.com/eessi/support/-/issues/154

…iltering LD_LIBRARY_PATH

…e the patch isn't fetched

…-easyblocks#3860

easybuild/easyblocks/p/python.py

Co-authored-by: ocaisa <alan.ocais@cecam.org>

boegel · 2025-08-12T18:03:24Z

Code style check still fails:

./easybuild/easyblocks/p/python.py:386:23: F541 f-string is missing placeholders
./easybuild/easyblocks/p/python.py:387:24: F541 f-string is missing placeholders
./easybuild/easyblocks/p/python.py:393:5: E303 too many blank lines (2)

easybuild/easyblocks/p/python.py

…ion option

casparvl · 2025-08-27T13:23:22Z

@boegelbot please test @ jsc-zen3
EB_ARGS="Python-3.11.5-GCCcore-13.2.0.eb Python-3.12.3-GCCcore-13.3.0.eb Python-3.13.1-GCCcore-14.2.0.eb"

boegelbot · 2025-08-27T13:33:10Z

@casparvl: Request for testing this PR well received on jsczen3l1.int.jsc-zen3.fz-juelich.de

PR test command 'if [[ develop != 'develop' ]]; then EB_BRANCH=develop ./easybuild_develop.sh 2> /dev/null 1>&2; EB_PREFIX=/home/boegelbot/easybuild/develop source init_env_easybuild_develop.sh; fi; EB_PR=3860 EB_ARGS="Python-3.11.5-GCCcore-13.2.0.eb Python-3.12.3-GCCcore-13.3.0.eb Python-3.13.1-GCCcore-14.2.0.eb" EB_CONTAINER= EB_REPO=easybuild-easyblocks EB_BRANCH=develop /opt/software/slurm/bin/sbatch --job-name test_PR_3860 --ntasks=8 ~/boegelbot/eb_from_pr_upload_jsc-zen3.sh' executed!

exit code: 0
output:

Submitted batch job 7768

Test results coming soon (I hope)...

- notification for comment with ID 3228197327 processed

Message to humans: this is just bookkeeping information for me,
it is of no use to you (unless you think I have a bug, which I don't).

casparvl · 2025-08-27T13:50:03Z

Test report by @casparvl

Overview of tested easyconfigs (in order)

SUCCESS Python-3.11.5-GCCcore-13.2.0.eb
SUCCESS Python-3.12.3-GCCcore-13.3.0.eb
SUCCESS Python-3.13.1-GCCcore-14.2.0.eb

Build succeeded for 3 out of 3 (3 easyconfigs in total)
tcn358.local.snellius.surf.nl - Linux RHEL 9.4, x86_64, AMD EPYC 7H12 64-Core Processor, Python 3.9.18
See https://gist.github.com/casparvl/f13c0fa1d22f6a8c060edf8f1ec54f4a for a full test report.

Edit: note that these were all without the changes from easybuilders/easybuild-easyconfigs#23499 i.e. just to prove that this PR doesn't change any existing behavior and still works when EB is NOT configured to filter LD_LIBRARY_PATH.

boegelbot · 2025-08-27T14:21:48Z

Test report by @boegelbot

Overview of tested easyconfigs (in order)

SUCCESS Python-3.11.5-GCCcore-13.2.0.eb
SUCCESS Python-3.12.3-GCCcore-13.3.0.eb
SUCCESS Python-3.13.1-GCCcore-14.2.0.eb

Build succeeded for 3 out of 3 (3 easyconfigs in total)
jsczen3c1.int.jsc-zen3.fz-juelich.de - Linux Rocky Linux 9.6, x86_64, AMD EPYC-Milan Processor (zen3), Python 3.9.21
See https://gist.github.com/boegelbot/ed316fef093d4d6b7b54b7e01c59a848 for a full test report.

easybuild/easyblocks/p/python.py

…s_ld_library_path. use self.cfg.get. Also check for not filtering LIBRARY_PATH. Make sure to update the rest of the code for these name changes (and the fact that patch_ctypes_ld_library_path is now the name of a single patch fle, not a list.

casparvl

TODO: remark by @boegel is that we should have a sanity check that checks if the patch was applied, or more specifically, that the behavior is now correct. Since system libraries are system-specific, the only thing we can do is check if a library on LIBRARY_PATH is found correctly. E.g. we could try to load libpython3.so through a CDLL and a LoadLibrary call, and even call a find_library on it for good measure.

… to verify that the patch has been applied correctly when EasyBuild is configured to filter LD_LIBRARY_PATH

…sanity step...

…types

casparvl · 2025-10-15T08:38:22Z

easybuild/easyblocks/p/python.py

+        if (
+            'LD_LIBRARY_PATH' in filtered_env_vars and
+            'LIBRARY_PATH' not in filtered_env_vars and
+            patch_ctypes_ld_library_path


Should we remove the condition that a patch is applied? In other words: should we allow a user to build with filtered LD_LIBRARY_PATH but without the patch? (currently, that is allowed, the user will just receive a warning from the elif on line 397

casparvl · 2025-10-15T09:11:49Z

Ok, sanity check seems to work as intended. Using the patch from easybuilders/easybuild-easyconfigs#23499 I get a successful sanity check, with this in the debugging output:

== 2025-10-15 10:53:41,597 run.py:457 INFO run_shell_cmd: command environment of "python -c 'import ctypes; print(ctypes.CDLL("libpython3.so"))'" will be saved to /scratch-local/casparl.15333717/eb-g6qr2e5w/run-shell-cmd-output/python-utohkjqy
== 2025-10-15 10:53:41,597 run.py:460 INFO run_shell_cmd: Output of "python -c 'import ctypes; print(ctypes.CDLL("libpython3.so"))'" will be logged to /scratch-local/casparl.15333717/eb-g6qr2e5w/run-shell-cmd-output/python-utohkjqy/out.txt
== 2025-10-15 10:53:41,617 run.py:494 INFO Path to bash that will be used to run shell commands: /cvmfs/software.eessi.io/versions/2023.06/compat/linux/x86_64/bin/bash
== 2025-10-15 10:53:41,617 run.py:508 INFO Running shell command 'python -c 'import ctypes; print(ctypes.CDLL("libpython3.so"))'' in /tmp/casparl/easybuild/build/Python/3.11.5/GCCcore-13.2.0/pip/pip-23.2.1
== 2025-10-15 10:53:42,135 run.py:622 INFO Shell command completed successfully: python -c 'import ctypes; print(ctypes.CDLL("libpython3.so"))'
== 2025-10-15 10:53:42,135 run.py:624 INFO Output of 'python ...' shell command (stdout + stderr):
<CDLL '/home/casparl/eessi/versions/2023.06/software/linux/x86_64/amd/zen2/software/Python/3.11.5-GCCcore-13.2.0/lib/../lib64/libpython3.so', handle 11ed940 at 0x1456f50afb10>

== 2025-10-15 10:53:42,138 python.py:748 DEBUG Matching regular expression pattern ^<CDLL '/home/casparl/eessi/versions/2023\.06/software/linux/x86_64/amd/zen2/software/Python/3\.11\.5\-GCCcore\-13\.2\.0.*', handle [a-f0-9]+ at 0x[a-f0-9]+>$ to string <CDLL '/home/casparl/eessi/versions/2023.06/software/linux/x86_64/amd/zen2/software/Python/3.11.5-GCCcore-13.2.0/lib/../lib64/libpython3.so', handle 11ed940 at 0x1456f50afb10>
== 2025-10-15 10:53:42,138 python.py:752 INFO Call to ctypes.CDLL('libpython3.so') succesfully opened libpython3.so, indicating that the patch that fixes ctypes when EasyBuild is configured to filter LD_LIBRARY_PATH was applied succesfully.

If I use a dummy patch that just inserts some comments, I get:

== FAILED: Installation ended unsuccessfully: Opening of libpython3.so using ctypes.CDLL('libpython3.so') failed. Ctypes requires a patch when
EasyBuild is configured to filter LD_LIBRARY_PATH. Please check if you specified a patch through patch_ctypes_ld_library_path and check the
logs to see if it applied correctly. (took 12 mins 41 secs)
== Results of the build can be found in the log file(s)
/scratch-local/casparl.15371430/eb-h2rz1r18/easybuild-Python-3.11.5-20251015.104150.PXTEX.log
== Running post-easyblock hook...
== Summary:
   * [FAILED]  Python/3.11.5-GCCcore-13.2.0

ERROR: Installation of Python-3.11.5-GCCcore-13.2.0.eb failed: "Opening of libpython3.so using ctypes.CDLL('libpython3.so') failed. Ctypes requires a patch when EasyBuild is configured to filter LD_LIBRARY_PATH. Please check if you specified a patch through patch_ctypes_ld_library_path and check the logs to see if it applied correctly."

casparvl · 2025-10-15T19:37:00Z

@boegelbot please test @ jsc-zen3
EB_ARGS="Python-3.11.5-GCCcore-13.2.0.eb Python-3.12.3-GCCcore-13.3.0.eb Python-3.13.1-GCCcore-14.2.0.eb"

boegelbot · 2025-10-15T19:53:10Z

@casparvl: Request for testing this PR well received on jsczen3l1.int.jsc-zen3.fz-juelich.de

PR test command 'if [[ develop != 'develop' ]]; then EB_BRANCH=develop ./easybuild_develop.sh 2> /dev/null 1>&2; EB_PREFIX=/home/boegelbot/easybuild/develop source init_env_easybuild_develop.sh; fi; EB_PR=3860 EB_ARGS="Python-3.11.5-GCCcore-13.2.0.eb Python-3.12.3-GCCcore-13.3.0.eb Python-3.13.1-GCCcore-14.2.0.eb" EB_CONTAINER= EB_REPO=easybuild-easyblocks EB_BRANCH=develop /opt/software/slurm/bin/sbatch --job-name test_PR_3860 --ntasks=8 ~/boegelbot/eb_from_pr_upload_jsc-zen3.sh' executed!

exit code: 0
output:

Submitted batch job 8415

Test results coming soon (I hope)...

- notification for comment with ID 3407980100 processed

Message to humans: this is just bookkeeping information for me,
it is of no use to you (unless you think I have a bug, which I don't).

boegelbot · 2025-10-15T20:27:45Z

Test report by @boegelbot

Overview of tested easyconfigs (in order)

FAIL (build issue) Python-3.11.5-GCCcore-13.2.0.eb (partial log available at https://gist.github.com/boegelbot/5a9a4a6bc891002793da112a908e4ccd)
SUCCESS Python-3.12.3-GCCcore-13.3.0.eb
SUCCESS Python-3.13.1-GCCcore-14.2.0.eb

Build succeeded for 2 out of 3 (3 easyconfigs in total)
jsczen3c3.int.jsc-zen3.fz-juelich.de - Linux Rocky Linux 9.6, x86_64, AMD EPYC-Milan Processor (zen3), Python 3.9.21
See https://gist.github.com/boegelbot/321093a94c35bb285ea06efca5fe4069 for a full test report.

casparvl · 2025-10-15T21:57:25Z

3.11.5 failed because I was rebuilding that in easybuilders/easybuild-easyconfigs#23499 as well, so lock file existed. Trying again for good measure...

@boegelbot please test @ jsc-zen3
EB_ARGS="Python-3.11.5-GCCcore-13.2.0.eb"

boegelbot · 2025-10-15T22:13:10Z

@casparvl: Request for testing this PR well received on jsczen3l1.int.jsc-zen3.fz-juelich.de

PR test command 'if [[ develop != 'develop' ]]; then EB_BRANCH=develop ./easybuild_develop.sh 2> /dev/null 1>&2; EB_PREFIX=/home/boegelbot/easybuild/develop source init_env_easybuild_develop.sh; fi; EB_PR=3860 EB_ARGS="Python-3.11.5-GCCcore-13.2.0.eb" EB_CONTAINER= EB_REPO=easybuild-easyblocks EB_BRANCH=develop /opt/software/slurm/bin/sbatch --job-name test_PR_3860 --ntasks=8 ~/boegelbot/eb_from_pr_upload_jsc-zen3.sh' executed!

exit code: 0
output:

Submitted batch job 8416

Test results coming soon (I hope)...

- notification for comment with ID 3408456358 processed

Message to humans: this is just bookkeeping information for me,
it is of no use to you (unless you think I have a bug, which I don't).

boegelbot · 2025-10-15T22:28:55Z

Test report by @boegelbot

Overview of tested easyconfigs (in order)

SUCCESS Python-3.11.5-GCCcore-13.2.0.eb

Build succeeded for 1 out of 1 (1 easyconfigs in total)
jsczen3c2.int.jsc-zen3.fz-juelich.de - Linux Rocky Linux 9.6, x86_64, AMD EPYC-Milan Processor (zen3), Python 3.9.21
See https://gist.github.com/boegelbot/6cb03b081d28cc424efa14fd78aff697 for a full test report.

dagonzalezfo and others added 9 commits July 25, 2025 14:15

Disable soname substitution

e1425a4

Update python.py

1e03933

Update python.py

197fb18

remove not used import

55298c1

Conditionally apply patches based on configuration of easybuild for f…

d8af2d2

…iltering LD_LIBRARY_PATH

Add forgotten commas

125c23f

Use the correct way to append to the patches and checksums lists

b1f3fcd

Do this in the fetch step: it needs to be done early enough, otherwis…

c2bf826

…e the patch isn't fetched

Clarify descriptions, remove commented section

8b17a9a

casparvl changed the title ~~Use patch when filter ld library path~~ Ptyhon: conditionally add patches when EasyBuild is configured to filter LD_LIBRARY_PATH Jul 30, 2025

casparvl changed the title ~~Ptyhon: conditionally add patches when EasyBuild is configured to filter LD_LIBRARY_PATH~~ Python: conditionally add patches when EasyBuild is configured to filter LD_LIBRARY_PATH Jul 30, 2025

casparvl pushed a commit to casparvl/easybuild-easyconfigs that referenced this pull request Jul 30, 2025

Changed EasyConfig to use custom keywords from easybuilders/easybuild…

115f7b0

…-easyblocks#3860

casparvl mentioned this pull request Jul 30, 2025

Small change to use the new EasyBlock config items dagonzalezfo/easybuild-easyconfigs#1

Merged

Make hound happy

d32dc4c

casparvl mentioned this pull request Jul 30, 2025

Python 3.11.5 librarypath prefix support easybuilders/easybuild-easyconfigs#23499

Open

dagonzalezfo mentioned this pull request Jul 30, 2025

WIP: update Python easyblock to use custom ctypes for shared libraries #3798

Closed

ocaisa requested changes Jul 30, 2025

View reviewed changes

easybuild/easyblocks/p/python.py Outdated Show resolved Hide resolved

casparvl and others added 2 commits July 30, 2025 14:03

Update easybuild/easyblocks/p/python.py

993f991

Co-authored-by: ocaisa <alan.ocais@cecam.org>

Fix too long lines

9fd0b9e

boegel changed the title ~~Python: conditionally add patches when EasyBuild is configured to filter LD_LIBRARY_PATH~~ conditionally add patches fo Python when EasyBuild is configured to filter LD_LIBRARY_PATH Jul 30, 2025

boegel changed the title ~~conditionally add patches fo Python when EasyBuild is configured to filter LD_LIBRARY_PATH~~ add support to Python easyblock for conditionally adding patches when EasyBuild is configured to filter LD_LIBRARY_PATH Jul 30, 2025

boegel added the enhancement label Jul 30, 2025

boegel added this to the 5.x milestone Jul 30, 2025

boegel added the EESSI Related to EESSI project label Aug 18, 2025

boegel mentioned this pull request Aug 18, 2025

ld is unable to find libraries in the host_injections EESSI/compatibility-layer#225

Open

boegel requested changes Aug 27, 2025

View reviewed changes

easybuild/easyblocks/p/python.py Outdated Show resolved Hide resolved

Caspar van Leeuwen added 3 commits August 27, 2025 11:53

Changes based on @boegels review. Reducing to 1 new custom configurat…

0c6d7ab

…ion option

Make explicit remark where the checksum for the custom patch should go

4ec46bb

Fix config variable name

817c51c

Caspar van Leeuwen and others added 2 commits August 27, 2025 13:07

Make sure to handle the default value behaviour correctly

67b35ec

Remove white line

4510e29

boegel changed the title ~~add support to Python easyblock for conditionally adding patches when EasyBuild is configured to filter LD_LIBRARY_PATH~~ add support to Python easyblock for conditionally adding patches when EasyBuild is configured to filter $LD_LIBRARY_PATH Aug 27, 2025

boegel requested changes Oct 8, 2025

View reviewed changes

easybuild/easyblocks/p/python.py Outdated Show resolved Hide resolved

easybuild/easyblocks/p/python.py Outdated Show resolved Hide resolved

easybuild/easyblocks/p/python.py Outdated Show resolved Hide resolved

Caspar van Leeuwen added 2 commits October 13, 2025 17:35

Fix hound issues

c622e77

casparvl commented Oct 14, 2025

View reviewed changes

Caspar van Leeuwen added 4 commits October 14, 2025 21:07

Add sanity check to see if a ctypes.CDLL call on libpython3.so works,…

247c394

… to verify that the patch has been applied correctly when EasyBuild is configured to filter LD_LIBRARY_PATH

Call the actual function for the specific ctypes sanity check in the …

cbab116

…sanity step...

Rephrase for clarity

13471fd

Add warning for users that filter LD_LIBRARY_PATH, but do not patch c…

c9f47a3

…types

casparvl commented Oct 15, 2025

View reviewed changes

casparvl mentioned this pull request Oct 15, 2025

{astro}[foss/2023b] LoSoTo v2.5.0, LSMTool v1.7.0, LINC v5.0, ... easybuilders/easybuild-easyconfigs#23155

Draft

1 task

add support to Python easyblock for conditionally adding patches when EasyBuild is configured to filter $LD_LIBRARY_PATH #3860

Are you sure you want to change the base?

add support to Python easyblock for conditionally adding patches when EasyBuild is configured to filter $LD_LIBRARY_PATH #3860

Conversation

casparvl commented Jul 30, 2025 • edited by boegel Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

boegel commented Aug 12, 2025

Uh oh!

Uh oh!

casparvl commented Aug 27, 2025

Uh oh!

boegelbot commented Aug 27, 2025

Uh oh!

casparvl commented Aug 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Overview of tested easyconfigs (in order)

Uh oh!

boegelbot commented Aug 27, 2025

Overview of tested easyconfigs (in order)

Uh oh!

Uh oh!

Uh oh!

Uh oh!

casparvl left a comment

Choose a reason for hiding this comment

Uh oh!

casparvl Oct 15, 2025

Choose a reason for hiding this comment

Uh oh!

casparvl commented Oct 15, 2025

Uh oh!

casparvl commented Oct 15, 2025

Uh oh!

boegelbot commented Oct 15, 2025

Uh oh!

boegelbot commented Oct 15, 2025

Overview of tested easyconfigs (in order)

Uh oh!

casparvl commented Oct 15, 2025

Uh oh!

boegelbot commented Oct 15, 2025

Uh oh!

boegelbot commented Oct 15, 2025

Overview of tested easyconfigs (in order)

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

add support to Python easyblock for conditionally adding patches when EasyBuild is configured to filter `$LD_LIBRARY_PATH` #3860

add support to Python easyblock for conditionally adding patches when EasyBuild is configured to filter `$LD_LIBRARY_PATH` #3860

casparvl commented Jul 30, 2025 •

edited by boegel

Loading

casparvl commented Aug 27, 2025 •

edited

Loading