Skip to content

Also check the EasyBuild hooks when checking missing installations, restrict the CUDA license hook to only trigger under specific conditions #1075

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 5 commits into from
May 20, 2025
Merged

Also check the EasyBuild hooks when checking missing installations, restrict the CUDA license hook to only trigger under specific conditions #1075

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
3f1e0a5
Also check the EasyBuild hooks when checking missing installations
ocaisa May 6, 2025
42008fd
Add quoting
ocaisa May 6, 2025
5c1c31b
Make sure to get full repo history
ocaisa May 15, 2025
5e900de
Merge remote-tracking branch 'origin/2023.06-software.eessi.io' into …
ocaisa May 20, 2025
1177c2b
Only trigger EESSI CUDA hook for EESSI installations
ocaisa May 20, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 26 additions & 0 deletions .github/workflows/test-software.eessi.io.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,8 @@ jobs:
steps:
- name: Check out software-layer repository
uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # v4.1.1
with:
fetch-depth: 0 # Fetch all history for all branches and tags

- name: Mount EESSI CernVM-FS pilot repository
uses: cvmfs-contrib/github-action-cvmfs@55899ca74cf78ab874bdf47f5a804e47c198743c # v4.0
Expand Down Expand Up @@ -132,3 +134,27 @@ jobs:
echo "captured missing package; test PASSED"
exit 0
fi

- name: Check that EasyBuild hook is up to date
run: |
FILE="eb_hooks.py"
TEMP_FILE="$(mktemp)"

# Fetch base branch
git fetch origin ${{ github.base_ref }}

# Check if the hooks has changed in the PR
if git diff --name-only origin/${{ github.base_ref }}...HEAD | grep -q "^$FILE$"; then
echo "Hooks changed in PR. Using PR version."
cp "$FILE" "$TEMP_FILE"
else
echo "File not changed in PR. Using default branch version."
git show origin/${{ github.base_ref }}:$FILE > "$TEMP_FILE"
fi

# Compare the hooks to what is shipped in the repository
# (it is overkill, but harmless, to do this for every architecture)
export EESSI_SOFTWARE_SUBDIR_OVERRIDE=${{matrix.EESSI_SOFTWARE_SUBDIR_OVERRIDE}}
source /cvmfs/software.eessi.io/versions/${EESSI_VERSION}/init/bash
module load EESSI-extend
diff "$TEMP_FILE" "$EASYBUILD_HOOKS"
92 changes: 48 additions & 44 deletions eb_hooks.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -965,52 +965,56 @@ def post_postproc_cuda(self, *args, **kwargs):
Remove files from CUDA installation that we are not allowed to ship,
and replace them with a symlink to a corresponding installation under host_injections.
"""
if self.name == 'CUDA':
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The change here is actually quite minimal. The previous condition was

if self.name == 'CUDA' and eessi_installation:

which meant that when someone tried to install CUDA but wasn't doing an EESSI installation you would (incorrectly) get the EasyBuildError

Copy link
Member Author

@ocaisa ocaisa May 20, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This change works as expected (tested using EESSI-extend):

ocaisa@~/EESSI/software-layer(hooks_check)$ module list

Currently Loaded Modules:
  1) EESSI/2023.06   2) EasyBuild/5.0.0   3) EESSI-extend/2023.06-easybuild

ocaisa@~/EESSI/software-layer(hooks_check)$ eb CUDA-12.1.1.eb --rebuild --accept-eula-for=CUDA --hooks=eb_hooks.py
...
== Running post-postproc hook...
== EESSI hook to respect CUDA license not triggered for installation path /home/ocaisa/eessi/versions/2023.06/software/linux/x86_64/intel/skylake_avx512/software/CUDA/12.1.1

# This hook only acts on an installation under repositories that _we_ ship (*.eessi.io/versions)
eessi_installation = bool(re.search(EESSI_INSTALLATION_REGEX, self.installdir))

if eessi_installation:
print_msg("Replacing files in CUDA installation that we can not ship with symlinks to host_injections...")

# read CUDA EULA, construct allowlist based on section 2.6 that specifies list of files that can be shipped
eula_path = os.path.join(self.installdir, 'EULA.txt')
relevant_eula_lines = []
with open(eula_path) as infile:
copy = False
for line in infile:
if line.strip() == "2.6. Attachment A":
copy = True
continue
elif line.strip() == "2.7. Attachment B":
copy = False
continue
elif copy:
relevant_eula_lines.append(line)

# create list without file extensions, they're not really needed and they only complicate things
allowlist = ['EULA', 'README']
file_extensions = ['.so', '.a', '.h', '.bc']
for line in relevant_eula_lines:
for word in line.split():
if any(ext in word for ext in file_extensions):
allowlist.append(os.path.splitext(word)[0])
# The EULA of CUDA 12.4 introduced a typo (confirmed by NVIDIA):
# libnvrtx-builtins_static.so should be libnvrtc-builtins_static.so
if 'libnvrtx-builtins_static' in allowlist:
allowlist.remove('libnvrtx-builtins_static')
allowlist.append('libnvrtc-builtins_static')
allowlist = sorted(set(allowlist))
self.log.info(
"Allowlist for files in CUDA installation that can be redistributed: " + ', '.join(allowlist)
)

# We need to check if we are doing an EESSI-distributed installation
eessi_installation = bool(re.search(EESSI_INSTALLATION_REGEX, self.installdir))

if self.name == 'CUDA' and eessi_installation:
print_msg("Replacing files in CUDA installation that we can not ship with symlinks to host_injections...")

# read CUDA EULA, construct allowlist based on section 2.6 that specifies list of files that can be shipped
eula_path = os.path.join(self.installdir, 'EULA.txt')
relevant_eula_lines = []
with open(eula_path) as infile:
copy = False
for line in infile:
if line.strip() == "2.6. Attachment A":
copy = True
continue
elif line.strip() == "2.7. Attachment B":
copy = False
continue
elif copy:
relevant_eula_lines.append(line)

# create list without file extensions, they're not really needed and they only complicate things
allowlist = ['EULA', 'README']
file_extensions = ['.so', '.a', '.h', '.bc']
for line in relevant_eula_lines:
for word in line.split():
if any(ext in word for ext in file_extensions):
allowlist.append(os.path.splitext(word)[0])
# The EULA of CUDA 12.4 introduced a typo (confirmed by NVIDIA):
# libnvrtx-builtins_static.so should be libnvrtc-builtins_static.so
if 'libnvrtx-builtins_static' in allowlist:
allowlist.remove('libnvrtx-builtins_static')
allowlist.append('libnvrtc-builtins_static')
allowlist = sorted(set(allowlist))
self.log.info("Allowlist for files in CUDA installation that can be redistributed: " + ', '.join(allowlist))

# Do some quick sanity checks for things we should or shouldn't have in the list
if 'nvcc' in allowlist:
raise EasyBuildError("Found 'nvcc' in allowlist: %s" % allowlist)
if 'libcudart' not in allowlist:
raise EasyBuildError("Did not find 'libcudart' in allowlist: %s" % allowlist)
# Do some quick sanity checks for things we should or shouldn't have in the list
if 'nvcc' in allowlist:
raise EasyBuildError("Found 'nvcc' in allowlist: %s" % allowlist)
if 'libcudart' not in allowlist:
raise EasyBuildError("Did not find 'libcudart' in allowlist: %s" % allowlist)

# replace files that are not distributable with symlinks into
# host_injections
replace_non_distributable_files_with_symlinks(self.log, self.installdir, self.name, allowlist)
# replace files that are not distributable with symlinks into
# host_injections
replace_non_distributable_files_with_symlinks(self.log, self.installdir, self.name, allowlist)
else:
print_msg(f"EESSI hook to respect CUDA license not triggered for installation path {self.installdir}")
else:
raise EasyBuildError("CUDA-specific hook triggered for non-CUDA easyconfig?!")

Expand Down