Skip to content

rename vep data env var #898

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Sep 19, 2024
Merged

rename vep data env var #898

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
1f0ea89
rename vep data env var
bpblanken Sep 17, 2024
a79be81
lint
bpblanken Sep 17, 2024
bbc9492
Merge branch 'dev' of github.com:broadinstitute/seqr-loading-pipeline…
bpblanken Sep 17, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions v03_pipeline/bin/dataproc_vep_init.bash
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,9 +52,9 @@ EOF
gcc -Wall -Werror -O2 /vep.c -o /vep
chmod u+s /vep

gcloud storage cp gs://seqr-luigi/releases/$ENVIRONMENT/latest/bin/download_vep_data.bash /download_vep_data.bash
chmod +x /download_vep_data.bash
./download_vep_data.bash $REFERENCE_GENOME
gcloud storage cp gs://seqr-luigi/releases/$ENVIRONMENT/latest/bin/download_vep_reference_data.bash /download_vep_reference_data.bash
chmod +x /download_vep_reference_data.bash
./download_vep_reference_data.bash $REFERENCE_GENOME

gcloud storage cp gs://seqr-luigi/releases/$ENVIRONMENT/latest/bin/vep /vep.bash
chmod +x /vep.bash
Expand Down
12 changes: 6 additions & 6 deletions v03_pipeline/bin/download_vep_data.bash → ...line/bin/download_vep_reference_data.bash
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
set -eux

REFERENCE_GENOME=$1
VEP_DATA=/seqr/vep_data
VEP_REFERENCE_DATASETS_DIR=${VEP_REFERENCE_DATASETS_DIR:-/seqr/vep-reference-data}

case $REFERENCE_GENOME in
GRCh38)
Expand Down Expand Up @@ -43,20 +43,20 @@ case $REFERENCE_GENOME in
exit 1
esac

if [ -f $VEP_DATA/$REFERENCE_GENOME/_SUCCESS ]; then
if [ -f $VEP_REFERENCE_DATASETS_DIR/$REFERENCE_GENOME/_SUCCESS ]; then
echo "Skipping download because already successful"
exit 0;
fi

mkdir -p $VEP_DATA/$REFERENCE_GENOME;
mkdir -p $VEP_REFERENCE_DATASETS_DIR/$REFERENCE_GENOME;
for vep_reference_data_file in ${VEP_REFERENCE_DATA_FILES[@]}; do
if [[ $vep_reference_data_file == *.tar.gz ]]; then
echo "Downloading and extracting" $vep_reference_data_file;
gsutil cat $vep_reference_data_file | tar -xzf - -C $VEP_DATA/$REFERENCE_GENOME/ &
gsutil cat $vep_reference_data_file | tar -xzf - -C $VEP_REFERENCE_DATASETS_DIR/$REFERENCE_GENOME/ &
else
echo "Downloading" $vep_reference_data_file;
gsutil cat $vep_reference_data_file $VEP_DATA/$REFERENCE_GENOME/ &
gsutil cat $vep_reference_data_file $VEP_REFERENCE_DATASETS_DIR/$REFERENCE_GENOME/ &
fi
done;
wait
touch $VEP_DATA/$REFERENCE_GENOME/_SUCCESS
touch $VEP_REFERENCE_DATASETS_DIR/$REFERENCE_GENOME/_SUCCESS
4 changes: 2 additions & 2 deletions v03_pipeline/bin/vep
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
set -eux

REFERENCE_GENOME=$1
VEP_DATA=/seqr/vep_data
VEP_REFERENCE_DATASETS_DIR=${VEP_REFERENCE_DATASETS_DIR:-/seqr/vep-reference-data}
VEP_DOCKER_IMAGE="gcr.io/seqr-project/vep-docker-image"

case $REFERENCE_GENOME in
Expand All @@ -17,5 +17,5 @@ case $REFERENCE_GENOME in
esac

shift # Remove the REFERENCE_GENOME arg.
docker run --platform linux/amd64 -i -v $VEP_DATA/$REFERENCE_GENOME:/opt/vep/.vep/:ro $VEP_DOCKER_IMAGE:$REFERENCE_GENOME \
docker run --platform linux/amd64 -i -v $VEP_REFERENCE_DATASETS_DIR/$REFERENCE_GENOME:/opt/vep/.vep/:ro $VEP_DOCKER_IMAGE:$REFERENCE_GENOME \
/opt/vep/src/ensembl-vep/vep $@
5 changes: 5 additions & 0 deletions v03_pipeline/lib/model/environment.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,10 @@
'REFERENCE_DATASETS_DIR',
'/seqr/seqr-reference-data',
)
VEP_REFERENCE_DATASETS_DIR = os.environ.get(
'VEP_REFERENCE_DATASETS_DIR',
'/seqr/vep-reference-data',
)

# Allele registry secrets :/
ALLELE_REGISTRY_SECRET_NAME = os.environ.get('ALLELE_REGISTRY_SECRET_NAME', None)
Expand Down Expand Up @@ -50,3 +54,4 @@ class Env:
PROJECT_ID: str | None = PROJECT_ID
REFERENCE_DATASETS_DIR: str = REFERENCE_DATASETS_DIR
SHOULD_REGISTER_ALLELES: bool = SHOULD_REGISTER_ALLELES
VEP_REFERENCE_DATASETS_DIR: str = VEP_REFERENCE_DATASETS_DIR
9 changes: 6 additions & 3 deletions v03_pipeline/lib/vep.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,10 +2,10 @@

import hail as hl

from v03_pipeline.lib.model import DatasetType, ReferenceGenome
from v03_pipeline.lib.model import DatasetType, Env, ReferenceGenome

VEP_CONFIG_URI = Template(
'file:///seqr/vep_data/$reference_genome/vep-$reference_genome.json',
'file://$vep_reference_datasets_dir/$reference_genome/vep-$reference_genome.json',
)


Expand All @@ -18,7 +18,10 @@ def run_vep(
return ht
return hl.vep(
ht,
config=VEP_CONFIG_URI.substitute(reference_genome=reference_genome.value),
config=VEP_CONFIG_URI.substitute(
vep_reference_datasets_dir=Env.VEP_REFERENCE_DATASETS_DIR,
reference_genome=reference_genome.value,
),
name='vep',
block_size=1000,
tolerate_parse_error=True,
Expand Down
Loading