Using Hail with dsub

[buutrg]

Hi all, I am trying to use hail via dsub to extract a subset of variants in All of Us server. I think this is the most relevant image I can use https://github.com/DataBiosphere/terra-docker/tree/master/terra-jupyter-hail

But it result in error that pyspark is not found. I tried to install pyspark from https://dlcdn.apache.org/spark/spark-3.1.3/spark-3.1.3-bin-hadoop3.tgz. Now it says No FileSystem for scheme "gs".

May I ask do you have any idea how to use hail via dsub?
Your help is really appreciated!

[drmurdock]

Hi @buutrg, did you ever solve this issue? I too am trying to use hail via dsub with the latest terra docker image (terra-jupyter-aou:2.2.14). Reading the dockerfile it seems pyspark is not installed, so I get the following error:

ModuleNotFoundError: No module named 'pyspark'

Were you able to figure this out? Options are limited since the VM can't access the internet to install it. Can hail even run in a dsub batch job? I'm thinking not. I'd prefer not having to run it from a jupyter notebook.

Thanks!