Myriad StarDist pipeline

Pipeline for computing image segmentations using StarDist on UCL high performance computing system Myriad.

Note

Make sure you're connected to the UCL network when accessing Myriad either via a UCL VPN or on campus via UCL Eduroam. Alternatively you can first connect to the UCL SSH gateway.

How to run pipeline

Clone this repository to your home directory on Myriad

cd ~
git clone https://github.com/UCL/myriad-stardist-pipeline
cd myriad-stardist-pipeline

Load modules with required system level dependencies. We use a pre-built version of TensorFlow optimized for use with the GPU nodes on Myriad here rather than installing the package separately. The module versions below are the latest available on Myriad at the time of writing, and the tensorflow module comes bundled with a Python 3.9 install.

module purge
module load compilers/gnu/10.2.0
module load openblas/0.3.13-openmp/gnu-10.2.0
module load tensorflow/2.11.0/gpu

Create a Python virtual environment

python3 -m venv venv

Activate environment, upgrade pip and install required Python dependencies

source venv/bin/activate
python3 -m pip install --upgrade pip
python3 -m pip install -r requirements.txt

Transfer required models and images to subdirectories models and images within myriad-stardist-pipeline with instructions below. The directories should have the following structure:

.
├── images
│   ├── image_1.tif
│   ├── image_2.tif
│   └── ...
└── models
    ├── model_1
    ├── model_2
    └── ...

On your local machine, you can arrange your files into the above directory structure and zip/compress them as normal.

You can then run commands in the shell to transfer them to the myriad-stardist-pipeline directory in your Myriad home directory using scp:

scp models.zip ucaXXXX@myriad.rc.ucl.ac.uk:~/myriad-stardist-pipeline/
scp images.zip ucaXXXX@myriad.rc.ucl.ac.uk:~/myriad-stardist-pipeline/

Note

ucaXXXX is your UCL username and this command is similar to ssh command you used to connect to Myriad.

On Myriad, you can check the contents of your myriad-stardist-pipeline directory and any other folder by running the ls command:

cd ~/myriad-stardist-pipeline
ls

You should see the two .zip files you just transferred.

Tip

You can also check which folder/directory you're currently in by running the pwd command.

Unzip the files in your myriad-stardist-pipeline directory using the unzip command:

unzip models.zip
unzip images.zip

You can check the contents of the models and images directories to confirm that the files have been unzipped correctly:

cd models
ls
cd ../images
ls

Submit job script by running

cd ~/myriad-stardist-pipeline
qsub job_script.sh

On completion the outputs of the job will be written to a directory outputs_{job_id} in your Scratch space where {job_id} is a unique identifier for the job.