Tutorial for Snakemake versions 7 and 8

Preparation on Minerva

Install a Snakemake 8 environment by doing the following:

mamba create -n snakemake8 -c bfh python=3.11 snakemake \
  snakemake-executor-plugin-lsf=0.2.5 \
  snakemake-storage-plugin-http snakemake-storage-plugin-ftp

Also make sure your main environment has Snakemake 7.32.4 installed:

mamba install snakemake=7.32.4

Finally, make sure you have profiles for Snakemake 8. The lab setup script now installs Snakemake 8 profiles by default. However, if you don't want to run the entire script, you can just run the following and respond to the prompts:

smp_url=https://raw.githubusercontent.com/marcoralab/lab_operations/main/scripts/setup_snakemake_profiles.py
wget -O setup_snakemake_profiles.py $smp_url && \
  python3 setup_snakemake_profiles.py && \
  rm setup_snakemake_profiles.py

Running Snakemake

Use the following basic commands for running Snakemake:

• Dry run: snakemake -np
• Local execution with Snakemake 7: snakemake --profile local
• Local execution with Snakemake 8: snakemake --profile local8
• LSF execution with Snakemake 7: snakemake --profile lsf
• LSF execution with Snakemake 8: snakemake --profile lsf8

Make sure that you do not run Snakemake locally without first using an interactive LSF job by running ijob or using the vscode_minerva script. Running Snakemake locally can cause Minerva to become unresponsive. HPC Admins will complain to both you and me, and Snakemake will be killed.

Make sure you are not on one of the following nodes:

• li03c02
• li03c03
• li03c04
• dataxfer-10 or similar

When to use each version

Snakemake 7

Use Snakemake 7 when you have already written your pipeline in Snakemake 7 or you are using a Snakemake 7 module and the following are true:

• You are not sharing the pipeline outside of Mount Sinai
• You are not making significant changes or rewrites to the code
• You have not encountered an issue that is difficult to solve without a bug report or significant investment of time

In other words, if you've already made the pipeline and it's gonna be a waste of time to upgrade, don't bother.

Snakemake 8

Use Snakemake 8 when writing new pipelines or preparing existing ones for sharing:

• You are starting a new pipeline
• You are sharing a pipeline outside of Mount Sinai (Our Snakemake 7 pipelines only work with LSF)
• You need to submit a bug to the Snakemake project (Snakemake 7 is no longer maintained)

Major differences in day-to-day usage

Memory specification

Memory is specified per-thread in Snakemake 7 but per-job in Snakemake 8.

So if you want a 4 core job with a total of 16 GB of memory, you would do the following:

Snakemake 7:

rule my_rule:
    threads: 4
    resources:
        mem_mb=4000

Snakemake 8:

rule my_rule:
    threads: 4
    resources:
        mem_mb=16000

You can also use mem_mb_per_cpu:

rule my_rule:
    threads: 4
    resources:
        mem_mb_per_cpu=4000

Walltime specification

time_min can be used in both versions of Snakemake. However, a walltime of "HH:MM" is valid in Snakemake 7 but should not be used in Snakemake 8. Instead, use time_min with the number of minutes or runtime with a number and unit (e.g. runtime="1h", runtime="5h", runtime="6d"). The runtime parameter is not usable in Snakemake 7.

lsf.yaml and workflow profiles:

Previously in Snakemake 7, the lsf.yaml file was used to specify extra LSF parameters. An example might look like this:

some_rule:
  - "-R 'himem'"
  - "-q long"

You can still theoretically use something similar in Snakemake 8. Place a config.yaml in profiles/default with contents like this:

set-resources:
    some_rule:
        lsf_queue: long
        lsf_extra: "-R 'himem'"

A profile with a different name can be used by adding --workflow-profile to your command.

However, this is currently broken for lsf_extra. Instead, add the resources to your rule in the Snakefile:

rule some_rule:
    resources:
        mem="500G",
        runtime="8d",
        lsf_queue="long",
        lsf_extra="-R 'himem'"

Remote storage providers

Snakemake 8 has a different way of specifying remote files. Please see the example pipelines for details.