Best way to run R code that requires a bit more memory?

[aspencoyle]

Hi all!

Alright, I analyzed a single crab using WGCNA (well, mostly - ran into some issues due to the lack of replication, but nothing major), and so decided to move on to analyzing all ambient-temperature crab over time (unfiltered libraries, so both host and parasite).

However, at a certain point (specifically the adjancency() function), it throws an error - it requires 16.2 gb of memory, and I just don't have enough on my local machine.

Since using R code on Mox seems like a bit of a hassle, and my memory requirement isn't massive, is there a way to run R code on Gannet or Roadrunner remotely? Or should I just make the jump on up to Mox?

[sr320]

I suggest getting Rstudio running on Mox. In the long-term this is likely a worthwhile endevour.

[kubu4]

This isn't really viable for jobs, as it requires a tunnel to remain open for the duration of the processing. If the connection to Mox is interrupted, the job would get killed.

[kubu4]

Although, now that I've typed that, it's possible that RStudio Server might work. I can mess around with seeing if we can run Rstudio Server sometime this week.

[kubu4]

UW IT passed along this info on running RStudio Server with a Singularity container on SLURM:

https://www.rocker-project.org/use/singularity/

UW IT (currently) doesn't have their own documentation ready yet, so suggested utilizing that example as a guide.

I'll look into it a bit more and see what I can do...

[kubu4]

Since using R code on Mox seems like a bit of a hassle

Not a hassle at all. Exactly the same as any other script you might run. The only thing that needs to be altered is the "shebang".

Example. Our usual Mox SBATCH scripts look like this:

#!/bin/bash
## Job Name
#SBATCH --job-name=20210406_pgen_blobtools_Panopea-generosa-v1.0
## Allocation Definition
#SBATCH --account=srlab
#SBATCH --partition=srlab
## Resources
## Nodes
#SBATCH --nodes=1
## Walltime (days-hours:minutes:seconds format)
#SBATCH --time=30-00:00:00
## Memory per node
#SBATCH --mem=500G
##turn on e-mail notification
#SBATCH --mail-type=ALL
#SBATCH --mail-user=samwhite@uw.edu
## Specify the working directory for this job
#SBATCH --chdir=/gscratch/scrubbed/samwhite/outputs/20210406_pgen_blobtools_Panopea-generosa-v1.0

The very first line (#!/bin/bash) is called a "shebang" and tells the computer what program to use to run the commands in the script. So, this shebang tells the computer to run the script using bash.

To tell the computer to use R instead, change that line to something like:

#!/gscratch/srlab/programs/R-3.6.2/bin/R

Then, it's ready to roll!

[sr320]

And FTR here is a list of computers available with memory info
https://docs.google.com/spreadsheets/d/1mtIITcjqZVEQtynYZFdOdx51uXTiXP7Jvvzv_SnWCDY/edit#gid=0

[yaaminiv]

FWIW, the instructions for running R Studio interactively on mox are likely out of date, because I couldn't find the specific module they loaded in these instructions.

Hyak documentation for running R on mox is pretty solid! https://wiki.cac.washington.edu/display/hyakusers/Hyak+R+programming

[kubu4]

Believe it or not, I have managed to get RStudio Server to function on Mox!!!

Here's the script to use to launch from from SBATCH.

• After launching, you'll need to follow the instructions that are printed in the SLURM output file.
• Before using, you'll likely need to change the R_LIBS_USER location specified in your ~/.Renviron file. Here's how my ~/.Renviron file currently looks:

[samwhite@mox2 rstudio_server_test]$ cat ~/.Renviron 
# Set local library installation path
# R_LIBS_USER=/gscratch/srlab/sam/R_libs
R_LIBS_USER=/gscratch/srlab/sam/R_libs_singularity

I we can adjust the script to set to use a specific ~/.Renviron file at some point, to avoid the need to manually modify and existing ~/.Renviron file.

Finally, keep in mind, it's current state is minimally configured (e.g. no tidyverse or methylkit yet), but does have DESeq2. So, it's more "proof-of-concept" right now. I'm in the process of retracing my steps and creating documentation on how to do this, so that people can update the Singularity container with new stuff on their own.

SBATCH script (obviously, adjust partition, account , --mail-user= and --chdir):

#!/bin/bash
## Job Name
#SBATCH --job-name=rstudio_server_test
## Allocation Definition
#SBATCH --account=
#SBATCH --partition=
## Resources
## Nodes
#SBATCH --nodes=1
## Walltime (days-hours:minutes:seconds format)
#SBATCH --time=0-08:00:00
## Memory per node
#SBATCH --mem=100G
#SBATCH --signal=USR2
##turn on e-mail notification
#SBATCH --mail-type=ALL
#SBATCH --mail-user=
## Specify the working directory for this job
#SBATCH --chdir=

module load singularity

export PASSWORD=$(openssl rand -base64 15)
# get unused socket per https://unix.stackexchange.com/a/132524
# tiny race condition between the python & singularity commands
readonly PORT=$(python -c 'import socket; s=socket.socket(); s.bind(("", 0)); print(s.getsockname()[1]); s.close()')
cat 1>&2 <<END
1. SSH tunnel from your workstation using the following command:

   ssh -N -L 8787:${HOSTNAME}:${PORT} ${USER}@LOGIN-HOST

   and point your web browser to http://localhost:8787

2. log in to RStudio Server using the following credentials:

   user: ${USER}
   password: ${PASSWORD}

When done using RStudio Server, terminate the job by:

1. Exit the RStudio Session ("power" button in the top right corner of the RStudio window)
2. Issue the following command on the login node:

      scancel -f ${SLURM_JOB_ID}
END


##NOTE##
# This is a local drive location I can write, you should be able
# to just set to a subfolder of your HPC home/scratch directory
export TMPDIR="/gscratch/scrubbed/${USER}/rstudio-tmp"

mkdir -p "$TMPDIR/tmp/rstudio-server"
uuidgen > "$TMPDIR/tmp/rstudio-server/secure-cookie-key"
chmod 0600 "$TMPDIR/tmp/rstudio-server/secure-cookie-key"

mkdir -p "$TMPDIR/var/lib"
mkdir -p "$TMPDIR/var/run"

# User-installed R packages go into their home directory
if [ ! -e ${HOME}/.Renviron ]
then
  printf '\nNOTE: creating ~/.Renviron file\n\n'
  echo 'R_LIBS_USER=~/R/%p-library/%v' >> ${HOME}/.Renviron
fi

# This example bind mounts the /project directory on the host into the Singularity container.
# By default the only host file systems mounted within the container are $HOME, /tmp, /proc, /sys, and /dev.
singularity exec \
--bind="$TMPDIR/var/lib:/var/lib/rstudio-server" \
--bind="$TMPDIR/var/run:/var/run/rstudio-server" \
--bind="$TMPDIR/tmp:/tmp" \
--bind=/gscratch/scrubbed/${USER} \
/gscratch/srlab/programs/singularity_containers/rstudio-4.0.2-mox \
rserver --www-port ${PORT} --auth-none=0 --auth-pam-helper-path=pam-helper

[yaaminiv]

What would the hostname/port be? I assume user is my UW net ID

Never mind, realize the slurm out produces that for you. This is what I get:

ssh -N -L 8787:n2221:33264 yaaminiv@LOGIN-HOST

What should the LOGIN-HOST be? My first thought was mox.hyak.edu but that did not work:

edited to include specific slurm output and error from using mox.hyak.edu

[kubu4]

@mox.hyak.uw.edu

[yaaminiv]

@mox.hyak.uw.edu

facepalm

[aspencoyle]

Hey! So I got my script all ready to run, and when loading libraries, both tidyverse and DESeq2 load just fine, but WGCNA is not! Any clue what's going on there? Used the same slurm script as your second one (with relevant inputs changed).

Screenshot just in case it helps:

[kubu4]

Dang, sorry! Not sure what happened. I've created an updated container: rstudio-4.0.2.sjw-v1.0

I've also tested to make sure it has WGCNA! I also updated the xlibs thingy which was hindering graphics from rendering in R Markdown, so that's fixed, too!

Here's the updated script to use referencing that new container:

#!/bin/bash
## Job Name
#SBATCH --job-name=rstudio_server_test
## Allocation Definition
#SBATCH --account=
#SBATCH --partition=
## Resources
## Nodes
#SBATCH --nodes=1
## Walltime (days-hours:minutes:seconds format)
#SBATCH --time=0-08:00:00
## Memory per node
#SBATCH --mem=100G
#SBATCH --signal=USR2
##turn on e-mail notification
#SBATCH --mail-type=ALL
#SBATCH --mail-user=
## Specify the working directory for this job
#SBATCH --chdir=

module load singularity

export PASSWORD=$(openssl rand -base64 15)
# get unused socket per https://unix.stackexchange.com/a/132524
# tiny race condition between the python & singularity commands
readonly PORT=$(python -c 'import socket; s=socket.socket(); s.bind(("", 0)); print(s.getsockname()[1]); s.close()')
cat 1>&2 <<END
1. SSH tunnel from your workstation using the following command:

   ssh -N -L 8787:${HOSTNAME}:${PORT} ${USER}@LOGIN-HOST

   and point your web browser to http://localhost:8787

2. log in to RStudio Server using the following credentials:

   user: ${USER}
   password: ${PASSWORD}

When done using RStudio Server, terminate the job by:

1. Exit the RStudio Session ("power" button in the top right corner of the RStudio window)
2. Issue the following command on the login node:

      scancel -f ${SLURM_JOB_ID}
END


##NOTE##
# This is a local drive location I can write, you should be able
# to just set to a subfolder of your HPC home/scratch directory
export TMPDIR="/gscratch/scrubbed/${USER}/rstudio-tmp"

mkdir -p "$TMPDIR/tmp/rstudio-server"
uuidgen > "$TMPDIR/tmp/rstudio-server/secure-cookie-key"
chmod 0600 "$TMPDIR/tmp/rstudio-server/secure-cookie-key"

mkdir -p "$TMPDIR/var/lib"
mkdir -p "$TMPDIR/var/run"

# User-installed R packages go into their home directory
if [ ! -e ${HOME}/.Renviron ]
then
  printf '\nNOTE: creating ~/.Renviron file\n\n'
  echo 'R_LIBS_USER=~/R/%p-library/%v' >> ${HOME}/.Renviron
fi

# This example bind mounts the /project directory on the host into the Singularity container.
# By default the only host file systems mounted within the container are $HOME, /tmp, /proc, /sys, and /dev.
singularity exec \
--bind="$TMPDIR/var/lib:/var/lib/rstudio-server" \
--bind="$TMPDIR/var/run:/var/run/rstudio-server" \
--bind="$TMPDIR/tmp:/tmp" \
--bind=/gscratch/scrubbed/${USER} \
/gscratch/srlab/programs/singularity_containers/rstudio-4.0.2.sjw-v1.0 \
rserver --www-port ${PORT} --auth-none=0 --auth-pam-helper-path=pam-helper