Conference notes for 2021 08 03

Next Meeting Sept. 7th at 9:00 Central UTC-5

Agenda for 2021-08-03

• Topics from last meeting: is-avail for TCL modulefiles
• Tmod4 (4.8+) now supports module try-load <module-name>
• Module usage tracking ideas
• "module overview" discussion
• Other topics
• Interactive module spider
  • module --interactive spider boost

=> lists the various versions of boost and asks the user which one they want to have more details about

module --interactive spider boost/1.72.0

=> list the various options to load it... and asks the user which one they want to load

Notes

(by Kenneth Hoste)

Participants

• Robert McLay (TACC)
• Kenneth Hoste (HPC-UGent)
• Kurt Lust (Univ. of Antwerp, Belgium; LUMI User Support Team)
• Bennet Fauber (UMich)
• Shelley Johnson (UMich)
• Fotis Georgatos (SDSC, Switzerland)
• Matt Thompson
• Maxime Boissonneault (Compute Canada)
• Fang Lui (GATECH)

Use of is-avail expression in module files

• cfr. last meeting (notes here)
• is-avail from Tcl modules wasn't possible (worked fine in Lua modules)
  • requested by Jeremy (Intel)
  • now implemented in Lmod by running is-avail command + parsing output

module try-load in Tmod 4.8+

• implemented by Xavier (Tmod maintainer)
• was only possible in Lmod, now also in Tmod

module overview discussion

• is implemented in latest Lmod 8.5.10

• module overview shows module names (without version) + count

• module avail now also produces additional output with suggestions to filter results of avail

• Bennet: should there be a "quiet" mode to supress these suggestions?

• Bennet: could be interesting to implement module new-features to help with discovery of new Lmod features

Module usage tracking ideas

• Bennet: how are (admins) tracking module usage?
• at TACC:
  • all systems (except the Cray, RIP)
  • all syslog messages go to a "master" node, which pushes them to database
  • procedure in place for staff to query that database
  • mostly used for checking which compilers are used
  • important because software at TACC is installed on local disk
  • see https://github.com/TACC/Lmod/tree/master/contrib/tracking_module_usage
  • other sites ingest syslog through ELK
• Bennet:
  • different setup at UMich: nodes push directly to ELK setup
  • more difficult to get useful data out of ELK compared to just syslog
  • changing indices, fast moving ELK, ...
  • Robert: good question for the Lmod mailing list!
• Fotis:
  • see contrib folder: https://github.com/TACC/Lmod/blob/master/contrib/Bright/SitePackage.lua
  • Lmod can be packaged such that it works out of the box
• Kenneth:
  • similar hook to log module loads: https://github.com/hpcugent/Lmod-UGent/blob/master/SitePackage.lua
  • "module load" dashboard in HPC-UGent ELK setup
• Maxime
  • extensive Grafana dashboard for Compute Canada
  • easy to see evolution over time and filter stuff
  • see https://github.com/ComputeCanada/software-stack-config/blob/main/lmod/SitePackage_logging.lua
• Robert:
  • "nag" message to inform users about deprecated modules is very useful
  • see "admin list" feature

Interactive module spider

• idea by Maxime inspired by Mii tool to link (missing) commands to modules: https://github.com/codeandkey/mii
• module --interactive spider boost
• user-friendly interactive way of browsing through modules
  • give list of numbered options on how to proceed
• Robert: not a big fan of stuff that requires interaction in Lmod itself
  • module commands should be usable from a script
  • but, open to PRs, and up for discussion...
• Bennet: seems more like a job for Mii than for Lmod?
  • separate tool for interactively browsing module tree could help with keeping Lmod "simple" to use/learn

Other topics

• Fang: integration of Lmod and XALT (https://github.com/xalt/xalt)
  • correlate data from Lmod (module loads) and XALT (runtime env) based on job IDs
  • Lmod reverse map can be used to match binaries to modules in XALT
  • Kenneth: two different data streams, essentially needs "join" statements between two tables

Ideas to discuss next time

• Robert: short demo/tutorial sessions with follow-up discussion and Q&A?
  • potential topics
    • getting Lmod up and running
    • designing module trees (incl. hierarchies)
    • hooks
    • contributing to Lmod
    • single-digit compiler hierarchy
    • separate module hierarchies (cfr. biocontainers in Lmod)
  • Kenneth: could be interesting to record these sessions and post them on YouTube