Files not being called properly from an array in singularity env file?

[yaaminiv]

Maybe more of a question for @kubu4...

How can I properly format an array of file name prefixes/ensure those files are called in a for loop?

I'm running the program BAT_mapping, and I need to load a singularity file on our HPC to run it. Basically I'm aligning paired-end read files to a genome (script found here, relevant section below). I'm binding directories (--bind) and providing a file with variables (--env-file)

for f in "${FASTQ[@]}"
do
  singularity exec --env-file /vortexfs1/home/yaamini.venkataraman/03-alignment-envfile.txt --bind /vortexfs1/home/naluru/:/naluru,/vortexfs1/scratch/yaamini.venkataraman:/scratch /vortexfs1/home/naluru/bat_latest.sif \
  BAT_mapping \
  -g $GENOME \
  -q ${TRIMMED}/${f}_1_val_1.fq.gz \
  -p ${TRIMMED}/${f}_2_val_2.fq.gz \
  -i $INDICES \
  -o ${SINGMAPPED}/${f} \
  -t 16 \
  -F 2
done

I'm using the following code in the environment file to create an array of file prefixes:

FASTQ=(/scratch/02-trimgalore/190626_I114_FCH7TVNBBXY_L2_20-N4 /scratch/02-trimgalore/190626_I114_FCH7TVNBBXY_L2_20-S1 /scratch/02-trimgalore/190626_I114_FCH7TVNBBXY_L2_20-S3 /scratch/02-trimgalore/190626_I114_FCH7TVNBBXY_L2_20-S4 /scratch/02-trimgalore/190626_I114_FCH7TVNBBXY_L2_5-N1 /scratch/02-trimgalore/190626_I114_FCH7TVNBBXY_L2_5-N2 /scratch/02-trimgalore/190626_I114_FCH7TVNBBXY_L2_5-S3 /scratch/02-trimgalore/190626_I114_FCH7TVNBBXY_L2_5-S4 /scratch/02-trimgalore/190626_I114_FCH7TVNBBXY_L2_OC-S1 /scratch/02-trimgalore/190626_I114_FCH7TVNBBXY_L3_20-N2 /scratch/02-trimgalore/190626_I114_FCH7TVNBBXY_L3_5-N3 /scratch/02-trimgalore/190626_I114_FCH7TVNBBXY_L3_5-S2 /scratch/02-trimgalore/190626_I114_FCH7TVNBBXY_L3_OC-N1 /scratch/02-trimgalore/190626_I114_FCH7TVNBBXY_L3_OC-N2 /scratch/02-trimgalore/190626_I114_FCH7TVNBBXY_L3_OC-N4 /scratch/02-trimgalore/190626_I114_FCH7TVNBBXY_L3_OC-N5 /scratch/02-trimgalore/190626_I114_FCH7TVNBBXY_L3_OC-S2 /scratch/02-trimgalore/190626_I114_FCH7TVNBBXY_L3_OC-S3 /scratch/02-trimgalore/190626_I114_FCH7TVNBBXY_L4_20-N1 /scratch/02-trimgalore/190626_I114_FCH7TVNBBXY_L4_20-S2 /scratch/02-trimgalore/190626_I114_FCH7TVNBBXY_L4_5-S1 /scratch/02-trimgalore/190626_I114_FCH7TVNBBXY_L4_OC-S5)

When I run my code, I get no alignment output so I don't think the files are being called properly. How can I set the variable properly so I can call it in the singularity container? When I run code interactively, I'm able to create the array and reference it, so I'm not sure why it's not working in my for loop:

Singularity> FASTQ=(/scratch/02-trimgalore/190626_I114_FCH7TVNBBXY_L2_20-N4 /scratch/02-trimgalore/190626_I114_FCH7TVNBBXY_L2_20-S1 /scratch/02-trimgalore/190626_I114_FCH7TVNBBXY_L2_20-S3 /scratch/02-trimgalore/190626_I114_FCH7TVNBBXY_L2_20-S4 /scratch/02-trimgalore/190626_I114_FCH7TVNBBXY_L2_5-N1 /scratch/02-trimgalore/190626_I114_FCH7TVNBBXY_L2_5-N2 /scratch/02-trimgalore/190626_I114_FCH7TVNBBXY_L2_5-S3 /scratch/02-trimgalore/190626_I114_FCH7TVNBBXY_L2_5-S4 /scratch/02-trimgalore/190626_I114_FCH7TVNBBXY_L2_OC-S1 /scratch/02-trimgalore/190626_I114_FCH7TVNBBXY_L3_20-N2 /scratch/02-trimgalore/190626_I114_FCH7TVNBBXY_L3_5-N3 /scratch/02-trimgalore/190626_I114_FCH7TVNBBXY_L3_5-S2 /scratch/02-trimgalore/190626_I114_FCH7TVNBBXY_L3_OC-N1 /scratch/02-trimgalore/190626_I114_FCH7TVNBBXY_L3_OC-N2 /scratch/02-trimgalore/190626_I114_FCH7TVNBBXY_L3_OC-N4 /scratch/02-trimgalore/190626_I114_FCH7TVNBBXY_L3_OC-N5 /scratch/02-trimgalore/190626_I114_FCH7TVNBBXY_L3_OC-S2 /scratch/02-trimgalore/190626_I114_FCH7TVNBBXY_L3_OC-S3 /scratch/02-trimgalore/190626_I114_FCH7TVNBBXY_L4_20-N1 /scratch/02-trimgalore/190626_I114_FCH7TVNBBXY_L4_20-S2 /scratch/02-trimgalore/190626_I114_FCH7TVNBBXY_L4_5-S1 /scratch/02-trimgalore/190626_I114_FCH7TVNBBXY_L4_OC-S5)
Singularity> echo "${FASTQ[0]}"
/scratch/02-trimgalore/190626_I114_FCH7TVNBBXY_L2_20-N4

[kubu4]

My initial guess would be to simply move the for loop (for f in "${FASTQ[@]}") to inside the Singularity call. E.g.:

singularity exec --env-file /vortexfs1/home/yaamini.venkataraman/03-alignment-envfile.txt \
--bind /vortexfs1/home/naluru/:/naluru,/vortexfs1/scratch/yaamini.venkataraman:/scratch /vortexfs1/home/naluru/bat_latest.sif
for f in "${FASTQ[@]}"
do
BAT_mapping \
  -g $GENOME \
  -q ${TRIMMED}/${f}_1_val_1.fq.gz \
  -p ${TRIMMED}/${f}_2_val_2.fq.gz \
  -i $INDICES \
  -o ${SINGMAPPED}/${f} \
  -t 16 \
  -F 2
done

But, I haven't refreshed my understanding of container usage in quite some time, so I've forgotten the concept of how to operate "within" the container after starting it...

[yaaminiv]

In your second error, it looks like something went wrong when trying to load the module

I killed my HPC node and reloaded the module, but got the same error...I'll do some digging to figure out if and how to fix it

[kubu4]

Looking at your envfile, it's what I thought. You won't be able to populate an array in that fashion. You'd need to do so via an interactive terminal or via a script.

So, with what we've experienced above, I'd opt for creating a script and calling that script with your Singularity image execution:

singularity exec --env-file /vortexfs1/home/yaamini.venkataraman/03-alignment-envfile.txt --bind /vortexfs1/home/naluru/:/naluru,/vortexfs1/scratch/yaamini.venkataraman:/scratch /vortexfs1/home/naluru/bat_latest.sif \
bat_mapping.sh

bat_mapping.sh would look something like this:

#!/bin/bash

# Populate FastQ array
FASTQ=(/scratch/02-trimgalore/190626_I114_FCH7TVNBBXY_L2_20-N4 /scratch/02-trimgalore/190626_I114_FCH7TVNBBXY_L2_20-S1 /scratch/02-trimgalore/190626_I114_FCH7TVNBBXY_L2_20-S3 /scratch/02-trimgalore/190626_I114_FCH7TVNBBXY_L2_20-S4 /scratch/02-trimgalore/190626_I114_FCH7TVNBBXY_L2_5-N1 /scratch/02-trimgalore/190626_I114_FCH7TVNBBXY_L2_5-N2 /scratch/02-trimgalore/190626_I114_FCH7TVNBBXY_L2_5-S3 /scratch/02-trimgalore/190626_I114_FCH7TVNBBXY_L2_5-S4 /scratch/02-trimgalore/190626_I114_FCH7TVNBBXY_L2_OC-S1 /scratch/02-trimgalore/190626_I114_FCH7TVNBBXY_L3_20-N2 /scratch/02-trimgalore/190626_I114_FCH7TVNBBXY_L3_5-N3 /scratch/02-trimgalore/190626_I114_FCH7TVNBBXY_L3_5-S2 /scratch/02-trimgalore/190626_I114_FCH7TVNBBXY_L3_OC-N1 /scratch/02-trimgalore/190626_I114_FCH7TVNBBXY_L3_OC-N2 /scratch/02-trimgalore/190626_I114_FCH7TVNBBXY_L3_OC-N4 /scratch/02-trimgalore/190626_I114_FCH7TVNBBXY_L3_OC-N5 /scratch/02-trimgalore/190626_I114_FCH7TVNBBXY_L3_OC-S2 /scratch/02-trimgalore/190626_I114_FCH7TVNBBXY_L3_OC-S3 /scratch/02-trimgalore/190626_I114_FCH7TVNBBXY_L4_20-N1 /scratch/02-trimgalore/190626_I114_FCH7TVNBBXY_L4_20-S2 /scratch/02-trimgalore/190626_I114_FCH7TVNBBXY_L4_5-S1 /scratch/02-trimgalore/190626_I114_FCH7TVNBBXY_L4_OC-S5)

# Run BAT_mapping on each pair of FastQs
for f in "${FASTQ[@]}"
do
BAT_mapping \
  -g $GENOME \
  -q ${TRIMMED}/${f}_1_val_1.fq.gz \
  -p ${TRIMMED}/${f}_2_val_2.fq.gz \
  -i $INDICES \
  -o ${SINGMAPPED}/${f} \
  -t 16 \
  -F 2
done

But, you likely need to have that script loaded in your container prior to execution (bind it somehow??). Also, if you do create that script, don't forget to make it executable before trying to run it (chmod +x bat_mapping.sh).

[yaaminiv]

That worked!! Thanks for the suggestion. Final scripts below:

• SLURM script: https://github.com/yaaminiv/killifish-hypoxia-RRBS/blob/main/code/03-alignment.sh
• File with variables: https://github.com/yaaminiv/killifish-hypoxia-RRBS/blob/main/code/03-alignment-envfile.txt
• Mapping script: https://github.com/yaaminiv/killifish-hypoxia-RRBS/blob/main/code/03-BAT-mapping.sh

[kubu4]

If you have a sec, do you mind pasting the actual Singularity command you ran to execute everything in the container?

[kubu4]

NM! I see it in the SLURM script (duh)!

[kubu4]

I killed my HPC node and reloaded the module, but got the same error...I'll do some digging to figure out if and how to fix it

Don't spend much time worrying about it. After doing some internet sleuthing, it seems that this shouldn't be much of a concern.