Storage space

[bnwolford]

I'm dealing with limited storage space so I'm trying to profile the space I need to run the pipeline. What is the storage space needed for an average run of the pipeline? What are the storage space needs a function of—the genotype data, the scores calculated, etc?

[nebfield]

Some stats from an example run using v1.3.2:

Input data:

• HGDP in a single file (downloaded from plink resources)
• Compressed variant information file
• No annotations in the information file (i.e. drop the INFO column when you recode a VCF)
• 78,097,677 million variants
• 929 samples
• ~4.4GB

Run:

$ nextflow run main.nf -profile docker --platform arm64 --input ~/Documents/data/hgdp/combined/samplesheet.csv --target_build GRCh38 --vzs --pgs_id PGS000802,PGS001229,PGS000049

Final output:

• work/: 0.6GB
• results/: 5.8MB

The work/ directory will at times be larger than 0.6GB as the workflow runs but we try to clean up temporary data as the processes complete.

Generally speaking I think the total number of variants across different scoring files and the number of variants in your target genomes are the biggest factors that increase storage space.

In results/match we include a log file that describes every variant across all input scoring files with metadata about matching, which can get big.

If you test this run with singularity you'll probably find the work directory is larger, because nextflow will stage the singularity images there by default. They shouldn't be more than ~3GB.

[bnwolford]

Great, thank you!

Is there a way to set a custom /tmp directory? The system /tmp directory I have is not large enough. I tried to reset TMP, TMPDIR, SINGULARITYENV_TMP, and SINGULARTIYENV_TMPDIR to a custom directory but I'm still getting no space left on device when the extraction starts in /tmp

[nebfield]

How far along do you get in the workflow when you get no space left on device? Does it trigger on a specific process?

[bnwolford]

Seems like it fails when extracting the image. I cleared what /tmp stuff I had rights to delete and got it to progress this far, it had been failing slightly earlier but still on the image extraction.

(base) bwolford@cvdpgx-home:~/work/pgs_calc$ nextflow run pgscatalog/pgscalc \
>     -profile singularity \
>     --input samplesheet.csv \
>     --trait_efo EFO_0001645 \
>     --target_build GRCh37

N E X T F L O W  ~  version 22.10.6
Launching `https://github.com/pgscatalog/pgscalc` [reverent_legentil] DSL2 - revision: c42dd1dee7 [main]


------------------------------------------------------
  pgscatalog/pgsc_calc v1.3.2
------------------------------------------------------
Core Nextflow options
  revision       : main
  runName        : reverent_legentil
  containerEngine: singularity
  launchDir      : /mnt/work/pgs_calc
  workDir        : /mnt/work/pgs_calc/work
  projectDir     : /home/bwolford/.nextflow/assets/pgscatalog/pgscalc
  userName       : bwolford
  profile        : singularity
  configFiles    : /home/bwolford/.nextflow/assets/pgscatalog/pgscalc/nextflow.config

Input/output options
  input          : samplesheet.csv
  pgs_id         : null
  pgp_id         : null
  trait_efo      : EFO_0001645
  target_build   : GRCh37
  genotypes_cache: null

Institutional config options
  hostnames      : [:]

Max job request options
  max_cpus       : 2
  max_memory     : 16.GB

!! Only displaying parameters that differ from the pipeline defaults !!
------------------------------------------------------
If you use pgscatalog/pgsc_calc for your analysis please cite:

* The Polygenic Score Catalog
  https://doi.org/10.1038/s41588-021-00783-5

* The nf-core framework
  https://doi.org/10.1038/s41587-020-0439-x

* Software dependencies
  https://github.com/pgscatalog/pgsc_calc/blob/master/CITATIONS.md
------------------------------------------------------
[-        ] process > PGSCATALOG_PGSCALC:PGSCALC:DOWNLOAD_SCOREFILES                -
executor >  local (2)
executor >  local (2)
[d2/0027c9] process > PGSCATALOG_PGSCALC:PGSCALC:DOWNLOAD_SCOREFILES ([pgs_id:, pgp_id:, trait_efo:EFO_0001645]) [  0%] 0 of 1
executor >  local (2)
[d2/0027c9] process > PGSCATALOG_PGSCALC:PGSCALC:DOWNLOAD_SCOREFILES ([pgs_id:, pgp_id:, trait_efo:EFO_0001645]) [  0%] 0 of 1
[bf/92320c] process > PGSCATALOG_PGSCALC:PGSCALC:INPUT_CHECK:SAMPLESHEET_JSON (samplesheet.csv)                  [100%] 1 of 1, failed: 1 ✘
[-        ] process > PGSCATALOG_PGSCALC:PGSCALC:INPUT_CHECK:COMBINE_SCOREFILES                                  -
[-        ] process > PGSCATALOG_PGSCALC:PGSCALC:MAKE_COMPATIBLE:PLINK2_RELABELBIM                               -
[-        ] process > PGSCATALOG_PGSCALC:PGSCALC:MAKE_COMPATIBLE:PLINK2_RELABELPVAR                              -
[-        ] process > PGSCATALOG_PGSCALC:PGSCALC:MAKE_COMPATIBLE:PLINK2_VCF                                      -
[-        ] process > PGSCATALOG_PGSCALC:PGSCALC:MAKE_COMPATIBLE:MATCH_VARIANTS                                  -
[-        ] process > PGSCATALOG_PGSCALC:PGSCALC:MAKE_COMPATIBLE:MATCH_COMBINE                                   -
[-        ] process > PGSCATALOG_PGSCALC:PGSCALC:APPLY_SCORE:PLINK2_SCORE                                        -
[-        ] process > PGSCATALOG_PGSCALC:PGSCALC:APPLY_SCORE:SCORE_AGGREGATE                                     -
[-        ] process > PGSCATALOG_PGSCALC:PGSCALC:APPLY_SCORE:SCORE_REPORT                                        -
[-        ] process > PGSCATALOG_PGSCALC:PGSCALC:DUMPSOFTWAREVERSIONS                                            -
Pulling Singularity image oras://dockerhub.ebi.ac.uk/gdp-public/pgsc_calc/singularity/pgscatalog_utils:amd64-0.3.1 [cache /mnt/work/pgs_calc/work/singularity/dockerhub.ebi.ac.uk-gdp-public-pgsc_calc-singularity-pgscatalog_utils-amd64-0.3.1.img]
Execution cancelled -- Finishing pending tasks before exit
WARN: Singularity cache directory has not been defined -- Remote image will be stored in the path: /mnt/work/pgs_calc/work/singularity -- Use env variable NXF_SINGULARITY_CACHEDIR to specify a different location
Error executing process > 'PGSCATALOG_PGSCALC:PGSCALC:INPUT_CHECK:SAMPLESHEET_JSON (samplesheet.csv)'

Caused by:
  Process `PGSCATALOG_PGSCALC:PGSCALC:INPUT_CHECK:SAMPLESHEET_JSON (samplesheet.csv)` terminated with an error exit status (255)

Command executed:

  samplesheet_to_json.py samplesheet.csv out.json

  cat <<-END_VERSIONS > versions.yml
  SAMPLESHEET_JSON:
      python: $(echo $(python --version 2>&1) | cut -f 2 -d ' ')
  END_VERSIONS

Command exit status:
  255

Command output:
  (empty)

Command error:
  INFO:    Converting SIF file to temporary sandbox...
  FATAL:   while extracting /mnt/work/pgs_calc/work/singularity/dockerhub.ebi.ac.uk-gdp-public-pgsc_calc-singularity-pgscatalog_utils-amd64-0.3.1.img: root filesystem extraction failed: extract command failed: WARNING: passwd file doesn't exist in container, not updating
  WARNING: group file doesn't exist in container, not updating
  WARNING: Skipping mount /etc/hosts [binds]: /etc/hosts doesn't exist in container
  WARNING: Skipping mount /etc/localtime [binds]: /etc/localtime doesn't exist in container
  WARNING: Skipping mount proc [kernel]: /proc doesn't exist in container
  WARNING: Skipping mount /home/bwolford/miniconda3/var/singularity/mnt/session/tmp [tmp]: /tmp doesn't exist in container
  WARNING: Skipping mount /home/bwolford/miniconda3/var/singularity/mnt/session/var/tmp [tmp]: /var/tmp doesn't exist in container
  WARNING: Skipping mount /home/bwolford/miniconda3/var/singularity/mnt/session/etc/resolv.conf [files]: /etc/resolv.conf doesn't exist in container

  Write on output file failed because No space left on device

  FATAL ERROR:writer: failed to write file /image/root/usr/bin/git
  Parallel unsquashfs: Using 16 processors
  39250 inodes (45521 blocks) to write

  : exit status 1

Work dir:
  /mnt/work/pgs_calc/work/bf/92320ca5771ea0e3a390d858f7bca4

Tip: view the complete command output by changing to the process work dir and entering the command `cat .command.out`


ERROR: No scores calculated!

[nebfield]

Setting NXF_SINGULARITY_CACHEDIR might be helpful!

[bnwolford]

That didn't but I did some more googling and found this. So I installed squashfs-tools. Different error messages now, but another no space left on device error. Still working on getting people to clear out their stuff in /tmp. 🤞🏻

Error executing process > 'PGSCATALOG_PGSCALC:PGSCALC:INPUT_CHECK:COMBINE_SCOREFILES (1)'

Caused by:
  Process `PGSCATALOG_PGSCALC:PGSCALC:INPUT_CHECK:COMBINE_SCOREFILES (1)` terminated with an error exit status (255)

Command executed:

  combine_scorefiles -s PGS001229_22.txt             -t GRCh37             -o scorefiles.txt.gz             -l log_scorefiles.json             -v

  cat <<-END_VERSIONS > versions.yml
  COMBINE_SCOREFILES:
      pgscatalog_utils: $(echo $(python -c 'import pgscatalog_utils; print(pgscatalog_utils.__version__)'))
  END_VERSIONS

Command exit status:
  255

Command output:
  (empty)

Command error:
  INFO:    Converting SIF file to temporary sandbox...
  FATAL:   while extracting /mnt/work/pgs_calc/work/singularity/dockerhub.ebi.ac.uk-gdp-public-pgsc_calc-singularity-pgscatalog_utils-amd64-0.3.1.img: root filesystem extraction failed: failed to copy content in staging file: write /tmp/rootfs-340126977/archive-700666325: no space left on device

Work dir:
  /mnt/work/pgs_calc/work/01/89dc48816f20dcc8babf0b779f6824

Tip: you can try to figure out what's wrong by changing to the process work dir and showing the script file named `.command.sh`


ERROR: No scores calculated!

[nebfield]

Does your local singularity installation always convert the SIF file to a sandbox? I don't normally see that happen. Was your local singularity build configured with setuid support?

[bnwolford]

I haven't used singularity in this environment before. I just installed it via conda and haven't run any specific configurations. Do I want setuid support, or is that potentially the problem?

[nebfield]

Normally when I run singularity in HPC environments like CSC Puhti or at the EBI, the containers aren't converted to sandboxes.

On these HPCs typically somebody with superuser powers has compiled and installed singularity system-wide. setuid support means that unprivileged users can run and build containers directly.

I think building sandboxes would use quite a lot of extra storage compared with a normal run. Do you have superuser access in the environment? You could try using the docker profile or building singularity.