Enable Slurm Accounting Database

[iamh2o]

Add mysql setup to the post-install-script (headnode only), connect to this and use it!

Will enable things like limiting inflight jobs to a specific max combined threads/vcpu count.

[iamh2o]

add to post install script

#!/bin/bash
set -euxo pipefail

sudo apt remove -y needrestart

# Variables (customize these if desired)
DB_NAME="slurm_acct_db"
DB_USER="slurm"
SECRET_NAME="parallelcluster/slurm_mysql_password"
REGION="us-west-2"


export DEBIAN_FRONTEND=noninteractive

# Completely remove existing MySQL installation
sudo systemctl stop mysql || true
sudo apt purge mysql-server mysql-client mysql-common mysql-server-core-* mysql-client-core-* -y
sudo rm -rf /var/lib/mysql
sudo rm -rf /etc/mysql
sudo apt autoremove -y
sudo apt clean

# Reinstall MySQL non-interactively
sudo apt update -y
sudo apt install mysql-server -y
sudo systemctl enable --now mysql

# Wait briefly to ensure MySQL is fully started
sleep 10

# Secure MySQL root user and generate secure random passwords
MYSQL_ROOT_PASS=$(openssl rand -base64 24)
DB_PASS=$(openssl rand -base64 24)


# Secure MySQL and set root user password
sudo mysql <<EOF
ALTER USER 'root'@'localhost' IDENTIFIED WITH mysql_native_password BY '${MYSQL_ROOT_PASS}';
DELETE FROM mysql.user WHERE User='';
DROP DATABASE IF EXISTS test;
DELETE FROM mysql.db WHERE Db='test' OR Db='test\_%';
FLUSH PRIVILEGES;
EOF

# Create Slurm database and user with correct DB_PASS
sudo mysql -u root -p"${MYSQL_ROOT_PASS}" <<EOF
CREATE DATABASE IF NOT EXISTS ${DB_NAME};
CREATE USER '${DB_USER}'@'localhost' IDENTIFIED BY '${DB_PASS}';
GRANT ALL PRIVILEGES ON ${DB_NAME}.* TO '${DB_USER}'@'localhost';
FLUSH PRIVILEGES;
EOF

# Store DB_PASS securely in AWS Secrets Manager
aws secretsmanager create-secret \
    --name "${SECRET_NAME}" \
    --region "${REGION}" \
    --description "Slurm MySQL DB password" \
    --secret-string "${DB_PASS}"

• It is not clear that the secretsmanager call will work from the config script... keep an eye on that

parallel cluster yaml

NOTE The passwordsecretarn needs to exist before using this cluster config.... so consider putting the ARN generator in the daylily-create-ephemeral-cluster script

Scheduling:
  Scheduler: slurm
  SlurmSettings:
    Database:
      Uri: localhost  # "mysql://<HeadNode-IP or hostname>/slurm_acct_db"
      UserName: "slurm"
      PasswordSecretArn: "arn:aws:secretsmanager:<region>:<account-id>:secret:<your-secret>"