model-ensembleris a tool to configure and automate running model ensembles on High-Performance Computing (HPC) clusters.
It reduces the manual configuration of individual ensemble runs, by using a common configuration to individually generate templates
for each run.
It also provides pre-and post processing functionality to allow for common tasks to be applied to the ensemble, before and after the individual runs.
model-ensembler is developed to be extendable to various HPC backends,currently supporting SLURM and running locally.
To install model-ensembler:
python -m venv venv
source venv/bin/activate
pip install model-ensembler
To check it has installed correctly, you can run:
model_ensemble_check [dummy|slurm]
Under the examples/ folder you will find example configs and templates that we can run on a local machine:
examples/
├── template_job/
│ ├── inputfile.j2
│ ├── pre_run.sh.j2
│ ├── slurm_run.sh.js
│ └── post_run.sh.j2
└── ensemble_config.ymlThe command model_ensemble is provided to execute the ensemble.
The --help flag can be used to find out more information:
model_ensemble --helpIts use is as follows:
model_ensemble configuration {slurm, dummy}Here configuration refers to our configuration file, and {slurm, dummy}
are the HPC backend options (where dummy is the options to run locally).
Applying this to our examples/, and running locally:
model_ensemble examples/sanity-check.yml dummyYou are now running a model ensemble!
For further usage instructions and an overview of model-ensembler, please refer to the documentation.
Current plans are captured now in the github issues. There's nothing in the long term that I'm focusing on for this tool, except to maintain it and see if I can promote the usage a bit more.
This tool was merely to help out with a single support ticket for a weather model run, but the concept had potential and it was easier than deploying something more substantial! If there are better approaches or tools that do something similar, very keen to look at them!
Certainly, things like Airflow and job arrays have similar concepts, but are either more heavyweight/less suitable deployment wise or not abstracted enough for simplifying lives, respectively!!!
Compare Cylc to model-ensembler.
The model-ensembler is part of a wider family of tools for Environmental Forecasting:
- download-toolbox: A toolbox of downloaders for environmental data.
- preprocess-toolbox: A toolbox for processing downloaded datasets according to common approaches for environmental data.
© British Antarctic Survey 2021-2025