Add dev.eessi.io support including project subdirectories #885

Neves-P · 2025-01-24T09:19:14Z

This PR replaces open WIP PR #804 (to merge from a more up to date and tested branch) and adds support for building into the dev.eessi.io repository.

Documentation for how to install into dev.eessi.io is available in https://www.eessi.io/docs/adding_software/adding_development_software/, with maintainer documentation to come in EESSI/dev.eessi.io-example#12

With this PR, software builds are installed into /cvmfs/dev.eessi.io/2023.06/$EESSI_DEV_PROJECT/. $EESSI_DEV_PROJECT is determined automatically during the build process and comes from the name of the GitHub repository that the bot is listening according to the format: dev.eessi.io-EESSI_DEV_PROJECT. See: https://github.com/EESSI/dev.eessi.io-example

Existing builds to dev.eessi.io were tested with the code that is about to be merged as verified in EESSI/dev.eessi.io-example#14

…tware-layer into feature/dev.eessi.io

…minimal_eessi_env

…ILD_INSTALLPATH override

Only remove EESSI_DEV_PROJECT

…software-layer into feature/dev.eessi.io-merge2

Neves-P · 2025-04-24T14:10:35Z

I've finished reworking and testing the dev.eessi.io build process after the review and taking into account changes that happened upstream in the meantime. For these changes to work, we also had to make small changes to the ingestion procedure, which were added with (merged) PR EESSI/filesystem-layer#212.

I think this is ready for a new review. I tested the changes in this PR using my personal fork of the repository via https://github.com/EESSI/dev.eessi.io-example/pull/. The installations land in the right path: /cvmfs/dev.eessi.io/example/... and are available as expected.

@ocaisa , since you reviewed this before could you take another look and see what you think, please?

ocaisa

Few trivial suggestions

EESSI-install-software.sh

bot/build.sh

create_tarball.sh

eessi_container.sh

Co-authored-by: ocaisa <alan.ocais@cecam.org>

These create ${EESSI_PROJECT_INSTALL} and export ${EESSI_DEV_PROJECT}. The first is handled inside the container, the latter must be exported already if this block is running

…-P/software-layer into feature/dev.eessi.io-merge

ocaisa

LGTM

eessi-bot · 2025-04-25T09:06:50Z

PR merged! Moved [] to /project/def-users/SHARED/trash_bin/EESSI/software-layer/2025.04.25

eessi-bot · 2025-04-25T09:06:51Z

PR merged! Moved [] to /project/def-users/SHARED/trash_bin/EESSI/software-layer/2025.04.25

gpu-bot-ugent · 2025-04-25T09:06:51Z

PR merged! Moved [] to /scratch/gent/vo/002/gvo00211/SHARED/trash_bin/EESSI/software-layer/2025.04.25

Neves-P and others added 30 commits November 1, 2024 11:26

Add EESSI_DEV_PROJET to prefix

8a6cd13

Merge branch 'EESSI:2023.06-software.eessi.io' into feature/dev.eessi.io

ee6aea9

Print EESSI_CVMFS_REPO to debug

0e9402b

Merge branch 'feature/dev.eessi.io' of https://github.com/Neves-P/sof…

8e628be

…tware-layer into feature/dev.eessi.io

Check if running in subshell is the cause of some variables being reset

888b874

Revert to run minimal_eessi_env in subshell and add log line

1af43f1

Fix condition to build with subdir in prefix in dev.eessi.io

25e8a5f

Add some logs to see where EESSI_CVMFS_REPO is being ovewritten

3e5e2af

Add more log output for EESSI_DEV_PROJECT variable

722ca22

export EESSI_DEV_PROJECT to compat layer env

1c3f20a

Expose cvmfs and version override in eessi_defaults

7c9dee8

Also expose EESSI_CVMFS_REPO in defaults

d5d1522

Try exporting EESSI_CVMFS_INSTALL

fb45d7e

Overwrite EASYBUILD_INSTALLPATH and use EESSI_CVMFS_REPO_OVERRIDE in …

a1f11fe

…minimal_eessi_env

Load $EASYBUILD_INSTALLPATH modules in dev.eessi.io

dd9fccd

Use EESSI_DEV_PROJECT for check instead of EESSI_CVMFS_REPO in EASYBU…

821455b

…ILD_INSTALLPATH override

Fix running install_scripts.sh only if not in dev.eessi.io

a19c79c

Remove debug printing to log

afaa94f

Improved logging when $EESSI_DEV_PROJECT is set

2c3e74e

Fix tarball creations in dev.eessi.io

3d9424d

Actually set install_prefix_dir

028f935

Log EESSI_DEV_PROJECT and export it

4f06864

Fix test checking for ${install_prefix_dir}

9eca5e4

Look for installed software in the right directory

0e68f18

Fix error reporting of input in create_tarball.sh

82d36c7

Fix installpath to include cpu target

3248438

Merge branch 'EESSI:2023.06-software.eessi.io' into feature/dev.eessi.io

d162ef3

Track EASYBUILD_EXPERIMENTAL set

4db1083

Fix typo in path

720d6e9

Use $TOPDIR to load_eessi_extend_module.sh

e126f78

Neves-P added 15 commits March 20, 2025 16:57

Leave rest of cvmfs_repo_name untouched

ece8127

Only remove EESSI_DEV_PROJECT

EESSI_DEV_PROJECT comes after overlays

9d980f9

Fix overlay paths in container

e98ce2b

Print debug output

16c77b5

Don't create child directory

38f129c

Create child dir, more debug output

82d3828

Print created overlay dir in container

013d942

Create dir, but don´t add subdir to envvar

3758ecd

No special handling of tarball path in check-build

38b6b07

Cleanup

d0b5df6

Handle .lmod files in dev.eessi.io builds

74ea9ec

Handle LMOD_PACKAGE_PATH that was missing

8e3e010

Format tarball name

4bb1559

Merge from main

b314d96

Merge branch '2023.06-software.eessi.io' of https://github.com/eessi/…

5be4ab2

…software-layer into feature/dev.eessi.io-merge2

Neves-P requested a review from ocaisa April 24, 2025 14:10

Neves-P marked this pull request as ready for review April 24, 2025 14:10

ocaisa requested changes Apr 24, 2025

View reviewed changes

Neves-P and others added 5 commits April 24, 2025 17:34

Add comment on cvmfs_repo_name and remove blank line at EOF

bd78807

Co-authored-by: ocaisa <alan.ocais@cecam.org>

Comments on how $EESSI_DEV_PROJECT is handled

00d2e6b

Improve comment on how SitePackage.lua and .lmodrc are handled

5c3154b

Co-authored-by: ocaisa <alan.ocais@cecam.org>

Remove leftover lines

6af5e85

These create ${EESSI_PROJECT_INSTALL} and export ${EESSI_DEV_PROJECT}. The first is handled inside the container, the latter must be exported already if this block is running

Merge branch 'feature/dev.eessi.io-merge' of https://github.com/Neves…

7d02c15

…-P/software-layer into feature/dev.eessi.io-merge

ocaisa approved these changes Apr 25, 2025

View reviewed changes

ocaisa merged commit c9e38e8 into EESSI:2023.06-software.eessi.io Apr 25, 2025
66 of 68 checks passed

Neves-P mentioned this pull request Apr 25, 2025

Install gh 2.68.1 EESSI/dev.eessi.io-example#25

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add dev.eessi.io support including project subdirectories #885

Add dev.eessi.io support including project subdirectories #885

Uh oh!

Neves-P commented Jan 24, 2025

Uh oh!

Neves-P commented Apr 24, 2025

Uh oh!

ocaisa left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ocaisa left a comment

Uh oh!

Uh oh!

eessi-bot bot commented Apr 25, 2025

Uh oh!

eessi-bot bot commented Apr 25, 2025

Uh oh!

gpu-bot-ugent bot commented Apr 25, 2025

Uh oh!

Uh oh!

Add dev.eessi.io support including project subdirectories #885

Add dev.eessi.io support including project subdirectories #885

Uh oh!

Conversation

Neves-P commented Jan 24, 2025

Uh oh!

Neves-P commented Apr 24, 2025

Uh oh!

ocaisa left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ocaisa left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

eessi-bot bot commented Apr 25, 2025

Uh oh!

eessi-bot bot commented Apr 25, 2025

Uh oh!

gpu-bot-ugent bot commented Apr 25, 2025

Uh oh!

Uh oh!