Queue adapters and executable templates

Table of Contents Queue adapters Using the configuration files to define template variables Example Using default values Example Leaving values undefined Executable templates

Queue adapters

These are templates that FireWorks uses to request appropriate resources when submitting jobs to the queue. Adapters for various queuing systems are found in the AaronJr/templates directory, with names of the form QUEUETYPE_qadpter.template. However, these should be checked for accuracy and any missing job script directives. Since we do not have access to every queuing system for testing, we would greatly appreciate any errors found in the template files provided by AaronJr be reported through the Issues tab on GitHub. If you need to edit the queue adapter template, you may copy the file to your $AARONLIB/templates folder and make any necessary changes there.

Using the configuration files to define template variables

The appropriate queue adapter template file is chosen based on the value of the queue_type option in the [HPC] section of your configuration file. For example: queue_type=SLURM will tell AaronJr to look for a queue adapter template file named SLURM_qadapter.template in first your $AARONLIB/templates directory then, if not found, in the AaronJr/templates directory.

The queue adapter template contains variables of the form $${option_name} that will be filled in using options in the [Job] section of the configuration file. An exception to this is the $${job_name} option, which will be automatically generated by AaronJr for you. See the [Job] section configuration help for commonly used options. If you need to use additional options not listed in the help page, AaronJr will automatically use the corresponding option=value pairs defined in your [Job] section to fill out the template.

The qadapter_options.py command line script can be used to both ensure AaronJr will use the correct template as well as to list the options found within that should be set by the user in their configuration's [Job] section:

$ qadapter_options.py NotAQueueType
Could not find NotAQueueType_qadapter.template in $AARONLIB/templates or AaronJr/templates
$ qadapter_options.py SLURM
Using $AARONLIB/templates/SLURM_qadapter.template
[Job] options found:
  queue
  nodes
  ppn
  wall
  memory

Example

The SLURM_qadapter.template is as follows:

#!/bin/bash
#SBATCH --partition=$${queue}
#SBATCH --job-name=$${job_name}
#SBATCH --ntasks=$${nodes}
#SBATCH --cpus-per-task=$${ppn}
#SBATCH --time=$${wall}:00:00
#SBATCH --mem=$${memory}

cd $${launch_dir}
$${rocket_launch}

Thus, the [HPC] and [Job] sections should look something like this:

[HPC]
queue_type = SLURM

[Job]
queue = batch
nodes = 1
ppn = 6
wall = 12
memory = 12GB

Using default values

Default values for these options can be set in the user's default configuration $AARONLIB/config.ini which can be overridden by options in the project configuration file. Function parsing is also available (see here for more info).

Example

Snippet from $AARONLIB/config.ini:

[Job]
queue = batch
nodes = 1
wall = 12
memory = %{ $ppn * $nodes * 2 }GB

Snippet from small_job.ini (will set memory = 2GB):

[Job]
ppn = 1
wall = 2

Snippet from large_job.ini (will set memory = 96GB):

[Job]
nodes = 2
ppn = 24
wall = 24

Leaving values undefined

It is best to go ahead and put additional job script directives that you may not always use in the queue adapter template. If the option is left undefined (i.e.: it is not in your configuration files), the line for that job script directive is removed from the parsed template. For example, say you want to receive a notification from the SLURM queue whenever jobs are completed but only for certain projects. These lines would be added to SLURM_qadapter.template:

#SBATCH --mail-type=$${mail_type}
#SBATCH --mail_user=$${mail_user}

Now, for projects where you DO want to receive these notifications, simply define mail_type=END and mail_user=myemail@something.com in the [Job] section of the configuration file. For projects where you DO NOT want to receive these notifications, leave the mail_type and mail_user option=value pairs out of the configuration file entirely.

Executable templates