Skip to content

ESPResSo 5.0-dev #1

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 34 commits into from
Jun 25, 2025
Merged

ESPResSo 5.0-dev #1

merged 34 commits into from
Jun 25, 2025

Conversation

jngrad
Copy link
Member

@jngrad jngrad commented Jun 5, 2025

Build the development version of ESPResSo against foss/2023b.

@jngrad jngrad mentioned this pull request Jun 12, 2025
Merged
@Neves-P
Copy link
Member

Neves-P commented Jun 12, 2025

bot: show_config

5 similar comments
@Neves-P
Copy link
Member

Neves-P commented Jun 12, 2025

bot: show_config

@Neves-P
Copy link
Member

Neves-P commented Jun 12, 2025

bot: show_config

@Neves-P
Copy link
Member

Neves-P commented Jun 12, 2025

bot: show_config

@Neves-P
Copy link
Member

Neves-P commented Jun 12, 2025

bot: show_config

@Neves-P
Copy link
Member

Neves-P commented Jun 12, 2025

bot: show_config

@multixscale-dev-eessi-io-bot
Copy link

Instance dev.eessi.io-bot-espresso-aws is configured to build for:

  • architectures: x86_64/amd/zen2, x86_64/amd/zen4, aarch64/neoverse_n1, aarch64/neoverse_v1
  • repositories: dev.eessi.io

@Neves-P
Copy link
Member

Neves-P commented Jun 12, 2025

bot: build arch:x86_64/amd/zen2 repo:dev.eessi.io

@multixscale-dev-eessi-io-bot
Copy link

multixscale-dev-eessi-io-bot bot commented Jun 12, 2025 •
edited
Loading

New job on instance dev.eessi.io-bot-espresso-aws for CPU micro-architecture x86_64-amd-zen2 for repository dev.eessi.io in job dir /project/def-espresso/SHARED/jobs/2025.06/pr_1/21

date job status comment
Jun 12 14:58:24 UTC 2025 submitted job id 21 awaits release by job manager
Jun 12 14:58:34 UTC 2025 released job awaits launch by Slurm scheduler
Jun 12 15:03:38 UTC 2025 running job 21 is running
Jun 12 15:14:54 UTC 2025 finished
😢 FAILURE (click triangle for details)
Details
✅ job output file slurm-21.out
✅ no message matching FATAL:
❌ found message matching ERROR:
❌ found message matching FAILED:
❌ found message matching required modules missing:
❌ no message matching No missing installations
✅ found message matching .tar.gz created!
Artefacts
eessi-2023.06-software-linux-x86_64-amd-zen2-espresso-1749741267.tar.gzsize: 0 MiB (45 bytes)
entries: 0
modules under 2023.06/software/linux/x86_64/amd/zen2/modules/all
no module files in tarball
software under 2023.06/software/linux/x86_64/amd/zen2/software
no software packages in tarball
other under 2023.06/software/linux/x86_64/amd/zen2
no other files in tarball
Jun 12 15:14:54 UTC 2025 test result
🤷 UNKNOWN (click triangle for detailed information)
  • Job test file _bot_job21.test does not exist in job directory or reading it failed.

@Neves-P
Copy link
Member

Neves-P commented Jun 12, 2025

bot: build arch:x86_64/amd/zen2 repo:dev.eessi.io

@multixscale-dev-eessi-io-bot
Copy link

multixscale-dev-eessi-io-bot bot commented Jun 12, 2025 •
edited
Loading

New job on instance dev.eessi.io-bot-espresso-aws for CPU micro-architecture x86_64-amd-zen2 for repository dev.eessi.io in job dir /project/def-espresso/SHARED/jobs/2025.06/pr_1/22

date job status comment
Jun 12 15:49:06 UTC 2025 submitted job id 22 awaits release by job manager
Jun 12 15:50:00 UTC 2025 released job awaits launch by Slurm scheduler
Jun 12 15:51:03 UTC 2025 running job 22 is running
Jun 12 15:52:05 UTC 2025 finished
😢 FAILURE (click triangle for details)
Details
✅ job output file slurm-22.out
✅ no message matching FATAL:
❌ found message matching ERROR:
✅ no message matching FAILED:
✅ no message matching required modules missing:
❌ no message matching No missing installations
✅ found message matching .tar.gz created!
Artefacts
eessi-2023.06-software-linux-x86_64-amd-zen2-espresso-1749743497.tar.gzsize: 0 MiB (45 bytes)
entries: 0
modules under 2023.06/software/linux/x86_64/amd/zen2/modules/all
no module files in tarball
software under 2023.06/software/linux/x86_64/amd/zen2/software
no software packages in tarball
other under 2023.06/software/linux/x86_64/amd/zen2
no other files in tarball
Jun 12 15:52:05 UTC 2025 test result
🤷 UNKNOWN (click triangle for detailed information)
  • Job test file _bot_job22.test does not exist in job directory or reading it failed.

@Neves-P
Copy link
Member

Neves-P commented Jun 12, 2025

bot: build arch:x86_64/amd/zen2 repo:dev.eessi.io

@multixscale-dev-eessi-io-bot
Copy link

multixscale-dev-eessi-io-bot bot commented Jun 12, 2025 •
edited
Loading

New job on instance dev.eessi.io-bot-espresso-aws for CPU micro-architecture x86_64-amd-zen2 for repository dev.eessi.io in job dir /project/def-espresso/SHARED/jobs/2025.06/pr_1/23

date job status comment
Jun 12 15:59:42 UTC 2025 submitted job id 23 awaits release by job manager
Jun 12 16:00:09 UTC 2025 released job awaits launch by Slurm scheduler
Jun 12 16:01:12 UTC 2025 running job 23 is running
Jun 12 16:03:15 UTC 2025 finished
😢 FAILURE (click triangle for details)
Details
✅ job output file slurm-23.out
✅ no message matching FATAL:
❌ found message matching ERROR:
❌ found message matching FAILED:
❌ found message matching required modules missing:
❌ no message matching No missing installations
✅ found message matching .tar.gz created!
Artefacts
eessi-2023.06-software-linux-x86_64-amd-zen2-espresso-1749744120.tar.gzsize: 0 MiB (45 bytes)
entries: 0
modules under 2023.06/software/linux/x86_64/amd/zen2/modules/all
no module files in tarball
software under 2023.06/software/linux/x86_64/amd/zen2/software
no software packages in tarball
other under 2023.06/software/linux/x86_64/amd/zen2
no other files in tarball
Jun 12 16:03:15 UTC 2025 test result
🤷 UNKNOWN (click triangle for detailed information)
  • Job test file _bot_job23.test does not exist in job directory or reading it failed.

@Neves-P
Copy link
Member

Neves-P commented Jun 12, 2025 

== processing EasyBuild easyconfig /eessi_bot_job/easyconfigs/ESPResSo-foss-2023b-software-commit.eb
== building and installing ESPResSo/-4b9ccc45713137eb16e5f19e5a313b5342080c75-foss-2023b...
  >> installation prefix: /cvmfs/dev.eessi.io/espresso/versions/2023.06/software/linux/x86_64/amd/zen2/software/ESPResSo/-4b9ccc45713137eb16e5f19e5a313b5342080c75-foss-2023b
== fetching files...
== Running pre-fetch hook...
  >> download succeeded: https://github.com/jngrad/espresso/archive/4b9ccc45713137eb16e5f19e5a313b5342080c75.tar.gz
  >> download failed: https://i10git.cs.fau.de/api/v4/projects/walberla%2Fwalberla/repository/archive?sha=59c9b8b1/walberla-59c9b8b1.tar.gz
  >> download failed: https://sources.easybuild.io/e/ESPResSo/walberla-59c9b8b1.tar.gz
== ... (took 1 secs)
== FAILED: Installation ended unsuccessfully (build directory: /cvmfs/dev.eessi.io/espresso/versions/2023.06/software/linux/x86_64/amd/zen2/software/ESPResSo/-4b9ccc45713137eb16e5f19e5a313b5342080c75-foss-2023b): build failed (first 300 chars): Couldn't find file walberla-59c9b8b1.tar.gz anywhere, and downloading it didn't work either... Paths attempted (in order): /eessi_bot_job/easyconfigs/e/ESPResSo/walberla-59c9b8b1.tar.gz, /eessi_bot_job/easyconfigs/ESPResSo/walberla-59c9b8b1.tar.gz, /eessi_bot_job/easyconfigs/walberla-59c9b8b1.tar.gz (took 1 secs)
== Results of the build can be found in the log file(s) /tmp/eb-mozetc_7/eb-po1shrc7/easybuild-ESPResSo--4b9ccc45713137eb16e5f19e5a313b5342080c75-20250612.160131.neTMC.log
Fetching files:  43% (3/7) ━━━━━━━╸           0:00:01 (walberla-59c9b8b1.tar.gz)ERROR: Build of /eessi_bot_job/easyconfigs/ESPResSo-foss-2023b-software-commit.eb failed (err: "build failed (first 300 chars): Couldn't find file walberla-59c9b8b1.tar.gz anywhere, and downloading it didn't work either... Paths attempted (in order): /eessi_bot_job/easyconfigs/e/ESPResSo/walberla-59c9b8b1.tar.gz, /eessi_bot_job/easyconfigs/ESPResSo/walberla-59c9b8b1.tar.gz, /eessi_bot_job/easyconfigs/walberla-59c9b8b1.tar.gz")

Fails with above error.

@jngrad, I edited the easyconfig and easystack to use the %(software_commit)s template, previous builds failed because they couldn't download the ESPReSso source from your fork (e.g.: #1 (comment)) and I hadn't noticed the template wasn't used. I did mess up the version name in the process, I'll fix that.

The correct ESPReSso sources are now grabbed, but as seen above the download of the other tarballs fails. It's probably just a matter of slightly adjusting the sources section of the easyconfig. If that's okay with you, I can do that and push directly here (as soon as I get home about to 🚲 there now).

@jngrad
Copy link
Member Author

jngrad commented Jun 12, 2025

bot: build arch:x86_64/amd/zen2 repo:dev.eessi.io

@multixscale-dev-eessi-io-bot
Copy link

multixscale-dev-eessi-io-bot bot commented Jun 12, 2025 •
edited
Loading

New job on instance dev.eessi.io-bot-espresso-aws for CPU micro-architecture x86_64-amd-zen2 for repository dev.eessi.io in job dir /project/def-espresso/SHARED/jobs/2025.06/pr_1/24

date job status comment
Jun 12 16:30:07 UTC 2025 submitted job id 24 awaits release by job manager
Jun 12 16:30:21 UTC 2025 released job awaits launch by Slurm scheduler
Jun 12 16:31:23 UTC 2025 running job 24 is running
Jun 12 16:32:25 UTC 2025 finished
😢 FAILURE (click triangle for details)
Details
✅ job output file slurm-24.out
✅ no message matching FATAL:
❌ found message matching ERROR:
❌ found message matching FAILED:
❌ found message matching required modules missing:
❌ no message matching No missing installations
✅ found message matching .tar.gz created!
Artefacts
eessi-2023.06-software-linux-x86_64-amd-zen2-espresso-1749745924.tar.gzsize: 0 MiB (45 bytes)
entries: 0
modules under 2023.06/software/linux/x86_64/amd/zen2/modules/all
no module files in tarball
software under 2023.06/software/linux/x86_64/amd/zen2/software
no software packages in tarball
other under 2023.06/software/linux/x86_64/amd/zen2
no other files in tarball
Jun 12 16:32:25 UTC 2025 test result
🤷 UNKNOWN (click triangle for detailed information)
  • Job test file _bot_job24.test does not exist in job directory or reading it failed.

@jngrad
Copy link
Member Author

jngrad commented Jun 12, 2025

@Neves-P feel free to adjust the sources as you see fit. I have changed the URL from my fork to upstream, since the commit in question is now available upstream. I also adjusted the waLBerla URL with a pound symbol to discard the automatically-added tarball filename.

@Neves-P
Copy link
Member

Neves-P commented Jun 12, 2025

bot: build arch:x86_64/amd/zen2 repo:dev.eessi.io

@multixscale-dev-eessi-io-bot
Copy link

multixscale-dev-eessi-io-bot bot commented Jun 12, 2025 •
edited
Loading

New job on instance dev.eessi.io-bot-espresso-aws for CPU micro-architecture x86_64-amd-zen2 for repository dev.eessi.io in job dir /project/def-espresso/SHARED/jobs/2025.06/pr_1/25

date job status comment
Jun 12 20:40:23 UTC 2025 submitted job id 25 awaits release by job manager
Jun 12 20:40:45 UTC 2025 released job awaits launch by Slurm scheduler
Jun 12 20:45:48 UTC 2025 running job 25 is running
Jun 12 20:47:52 UTC 2025 finished
😢 FAILURE (click triangle for details)
Details
✅ job output file slurm-25.out
✅ no message matching FATAL:
❌ found message matching ERROR:
✅ no message matching FAILED:
✅ no message matching required modules missing:
❌ no message matching No missing installations
✅ found message matching .tar.gz created!
Artefacts
eessi-2023.06-software-linux-x86_64-amd-zen2-espresso-1749761194.tar.gzsize: 0 MiB (45 bytes)
entries: 0
modules under 2023.06/software/linux/x86_64/amd/zen2/modules/all
no module files in tarball
software under 2023.06/software/linux/x86_64/amd/zen2/software
no software packages in tarball
other under 2023.06/software/linux/x86_64/amd/zen2
no other files in tarball
Jun 12 20:47:52 UTC 2025 test result
🤷 UNKNOWN (click triangle for detailed information)
  • Job test file _bot_job25.test does not exist in job directory or reading it failed.

@Neves-P
Copy link
Member

Neves-P commented Jun 12, 2025

I missed a comma... 😕

bot: build arch:x86_64/amd/zen2 repo:dev.eessi.io

@multixscale-dev-eessi-io-bot
Copy link

multixscale-dev-eessi-io-bot bot commented Jun 12, 2025 •
edited
Loading

New job on instance dev.eessi.io-bot-espresso-aws for CPU micro-architecture x86_64-amd-zen2 for repository dev.eessi.io in job dir /project/def-espresso/SHARED/jobs/2025.06/pr_1/26

date job status comment
Jun 12 20:50:02 UTC 2025 submitted job id 26 awaits release by job manager
Jun 12 20:50:55 UTC 2025 released job awaits launch by Slurm scheduler
Jun 12 20:51:58 UTC 2025 running job 26 is running
Jun 12 20:54:01 UTC 2025 finished
😢 FAILURE (click triangle for details)
Details
✅ job output file slurm-26.out
✅ no message matching FATAL:
❌ found message matching ERROR:
❌ found message matching FAILED:
❌ found message matching required modules missing:
❌ no message matching No missing installations
✅ found message matching .tar.gz created!
Artefacts
eessi-2023.06-software-linux-x86_64-amd-zen2-espresso-1749761562.tar.gzsize: 0 MiB (45 bytes)
entries: 0
modules under 2023.06/software/linux/x86_64/amd/zen2/modules/all
no module files in tarball
software under 2023.06/software/linux/x86_64/amd/zen2/software
no software packages in tarball
other under 2023.06/software/linux/x86_64/amd/zen2
no other files in tarball
Jun 12 20:54:01 UTC 2025 test result
🤷 UNKNOWN (click triangle for detailed information)
  • Job test file _bot_job26.test does not exist in job directory or reading it failed.

@Neves-P
Copy link
Member

Neves-P commented Jun 12, 2025

bot: build arch:x86_64/amd/zen2 repo:dev.eessi.io

@jngrad
Copy link
Member Author

jngrad commented Jun 24, 2025

Let me remove the os.cpu_count() and self.cfg.max_parallel logic completely. Even halving that number doesn't stop timeouts. We'll go back to running the 350 tests one by one, since there is no reliable way of finding a sensible core count to run them in parallel.

@jngrad
Copy link
Member Author

jngrad commented Jun 24, 2025

bot: build arch:x86_64/amd/zen2 repo:dev.eessi.io

@multixscale-dev-eessi-io-bot
Copy link

multixscale-dev-eessi-io-bot bot commented Jun 24, 2025 •
edited
Loading

New job on instance dev.eessi.io-bot-espresso-aws for CPU micro-architecture x86_64-amd-zen2 for repository dev.eessi.io in job dir /project/def-espresso/SHARED/jobs/2025.06/pr_1/57

date job status comment
Jun 24 15:53:02 UTC 2025 submitted job id 57 awaits release by job manager
Jun 24 15:53:27 UTC 2025 released job awaits launch by Slurm scheduler
Jun 24 15:54:31 UTC 2025 running job 57 is running
Jun 24 16:24:44 UTC 2025 finished
😁 SUCCESS (click triangle for details)
Details
✅ job output file slurm-57.out
✅ no message matching FATAL:
✅ no message matching ERROR:
✅ no message matching FAILED:
✅ no message matching required modules missing:
✅ found message(s) matching No missing installations
✅ found message matching .tar.gz created!
Artefacts
eessi-2023.06-software-linux-x86_64-amd-zen2-espresso-1750782221.tar.gzsize: 11 MiB (11907640 bytes)
entries: 144
modules under 2023.06/software/linux/x86_64/amd/zen2/modules/all
ESPResSo/8aa60cecd56cdd10ab62042c567552f347374f36-foss-2023b.lua
software under 2023.06/software/linux/x86_64/amd/zen2/software
ESPResSo/8aa60cecd56cdd10ab62042c567552f347374f36-foss-2023b
other under 2023.06/software/linux/x86_64/amd/zen2
no other files in tarball
Jun 24 16:24:44 UTC 2025 test result
🤷 UNKNOWN (click triangle for detailed information)
  • Job test file _bot_job57.test does not exist in job directory or reading it failed.
Jun 24 18:19:11 UTC 2025 not uploaded transfer of eessi-2023.06-software-linux-x86_64-amd-zen2-espresso-1750782221.tar.gz to S3 bucket failed
Jun 24 18:46:10 UTC 2025 uploaded transfer of eessi-2023.06-software-linux-x86_64-amd-zen2-espresso-1750782221.tar.gz to S3 bucket succeeded
Jun 24 19:03:12 UTC 2025 uploaded transfer of eessi-2023.06-software-linux-x86_64-amd-zen2-espresso-1750782221.tar.gz to S3 bucket succeeded
Jun 24 19:15:59 UTC 2025 uploaded transfer of eessi-2023.06-software-linux-x86_64-amd-zen2-espresso-1750782221.tar.gz to S3 bucket succeeded
Jun 24 19:24:12 UTC 2025 uploaded transfer of eessi-2023.06-software-linux-x86_64-amd-zen2-espresso-1750782221.tar.gz to S3 bucket succeeded
Jun 24 19:30:15 UTC 2025 uploaded transfer of eessi-2023.06-software-linux-x86_64-amd-zen2-espresso-1750782221.tar.gz to S3 bucket succeeded
Jun 24 19:31:34 UTC 2025 uploaded transfer of eessi-2023.06-software-linux-x86_64-amd-zen2-espresso-1750782221.tar.gz to S3 bucket succeeded
Jun 25 07:49:42 UTC 2025 uploaded transfer of eessi-2023.06-software-linux-x86_64-amd-zen2-espresso-1750782221.tar.gz to S3 bucket succeeded
Jun 25 08:01:10 UTC 2025 uploaded transfer of eessi-2023.06-software-linux-x86_64-amd-zen2-espresso-1750782221.tar.gz to S3 bucket succeeded
Jun 25 08:15:07 UTC 2025 uploaded transfer of eessi-2023.06-software-linux-x86_64-amd-zen2-espresso-1750782221.tar.gz to S3 bucket succeeded

@ocaisa
Copy link

ocaisa commented Jun 24, 2025

@Neves-P Can we deploy this then? If so I can update the deliverable with a working example

@Neves-P
Copy link
Member

Neves-P commented Jun 24, 2025

I think so, we can always build the other architectures in a follow up PR

@Neves-P Neves-P added the bot:deploy Ask the bot to deploy missing software informations to dev.eessi.io label Jun 24, 2025
@Neves-P
Copy link
Member

Neves-P commented Jun 24, 2025

The tarball signing failed, there must be a field that needs to be added to the config. I'll try to solve it

@Neves-P Neves-P added bot:deploy Ask the bot to deploy missing software informations to dev.eessi.io and removed bot:deploy Ask the bot to deploy missing software informations to dev.eessi.io labels Jun 24, 2025
@Neves-P
Copy link
Member

Neves-P commented Jun 24, 2025

There is something wrong in the signing config that I cannot figure out... I will check and ask for help tomorrow morning.

@Neves-P Neves-P added bot:deploy Ask the bot to deploy missing software informations to dev.eessi.io and removed bot:deploy Ask the bot to deploy missing software informations to dev.eessi.io labels Jun 25, 2025
@Neves-P
Copy link
Member

Neves-P commented Jun 25, 2025 

 ~ $ ls /cvmfs/dev.eessi.io/espresso/versions/2023.06/software/linux/x86_64/amd/zen2/software/ESPResSo/8aa60cecd56cdd10ab62042c567552f347374f36-foss-2023b/ -l
total 14
dr-xr-xr-x. 2 cvmfs cvmfs-reserved 4096 Jun 24 16:22 bin
dr-xr-sr-x. 3 cvmfs cvmfs-reserved 4096 Jun 24 16:22 easybuild
dr-xr-xr-x. 3 cvmfs cvmfs-reserved 4096 Jun 24 16:22 lib
lrwxrwxrwx. 1 cvmfs cvmfs-reserved    3 Jun 24 16:22 lib64 -> lib

The zen2 build is now available! You'll need to initialize EESSI and do module use /cvmfs/dev.eessi.io/espresso/versions/2023.06/software/linux/x86_64/amd/zen2/modules/all/ to load the module, but besides that, it should be available wherever dev.eessi.io is mounted at 🎉

I'll merge this PR. A note to ourselves in subsequent PRs:
heFFTe should be available very soon in software.eessi.io, so soon we can reapply the change in 69a4b59 and also build for the remaining three microarchitectures.

@Neves-P Neves-P merged commit 192a69d into multixscale:main Jun 25, 2025
@jngrad jngrad deleted the easyblock branch June 25, 2025 10:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ocaisa ocaisa ocaisa left review comments

Assignees
No one assigned
Labels
bot:deploy Ask the bot to deploy missing software informations to dev.eessi.io
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@jngrad @Neves-P @ocaisa