Dagster with SLURM submissions on HPC

[Andrew-S-Rosen]

I'll preface this by saying that I'm familiar with workflow tools in general but not yet with dagster.

Most of the tasks I want to run are to be executed on one of a variety of HPC machines, each of which has a SLURM job scheduling system. Can dagster be used to execute tasks in this way? I searched the docs for "SLURM" and didn't get much back except the Dask executor. The Dask executor can indeed be used to submit various job submission scripts, but this is all done locally, right? In other words, one would have to be using dagster on the same machine in which you wish to submit the Dask-generated SLURM job, right? The thing that makes this seem potentially a bit messy is that if you have tasks that you want to run on several different SLURM-hosted machines, you'd have to SSH into each one to execute jobs.

Is there a better/cleaner way to achieve this goal?

[ghost]

+1

[Andrew-S-Rosen]

I wouldn't hold your breath on this one, but good to see someone else interested. 😄

[RBrearton]

I'd also like to add that this would be a very important feature for my use case. We would like to use dagster to orchestrate HPC workflows that are scheduled using slurm.

I appreciate that slurm integration might not be an incredibly common use case in the open source world, but I suspect that there are quite a lot of industry users that would be very interested in this. Given that dagster is VC backed and ultimately needs to make a profit, perhaps it would be reasonable to consider prioritizing a feature like this?

[ghost]

I agree. A lot of people use tools like snakemake and nextflow, but I'm pretty sure if Dagster provided plugins for common HPC schedulers like SLURM, PBS, Torque, etc. there would be a ton of support for that.

[kazewong]

+1 on this plugin. While I am not sure how much this is a priority for the dagster team, I am wondering how difficult it would be to add a plugin targeting slurm. https://github.com/dagster-io/dagster/tree/master/python_modules/libraries/dagster-k8s is probably an example we can look at, another one is https://github.com/danielgafni/dagster-ray.

[prometheusred]

I'm doing research for a similar slurm-based, HPC use case now. And I'm confused by this post. I see the dask-jobqueue integration can generate sbatch commands... couldn't you just use the appropriate parameterized directives (and appropriate slurm setup) to send jobs to any given part of you slurm cluster? e.g. to a given partition?

Anyway, I'll be trying to figure out the way this Dask integration works (e.g. SlurmCluster vs SlurmRunner, also managing logs, managing job status in a dag, etc...) but might also be interested in developing a more custom and direct (non-dask) Slurm integration too as I learn more for my current project....

[geoHeil]

I would also be interested in a SLURM integration.

So far I was using dagster pipes with

• local
• databricks
• EMR

for remote compute and bootstrapping of the environment (see https://georgheiler.com/post/paas-as-implementation-detail/ and also the OSS library here https://github.com/ascii-supply-networks/ascii-hydra/tree/main/src/pipelines/ascii_library_demo

I would imagine a solution around:

• dagster pipes (termination + log forwarding + bootstrapping) with the particular HPC environment
• package manager https://pixi.sh/latest/
• packaging of the environement https://github.com/Quantco/pixi-pack

further I would Imagine that dagster would then spin up and start a ray cluster for the assets/computation and then ray would delegate to slurm for the resource management https://docs.ray.io/en/latest/cluster/vms/user-guides/community/slurm.html

[geoHeil]

Is here anyone around who might be interested in collaborating on an integration of a modern data orchestrator like dagster with supercomputers/slurm? I am thinking of organizing a team to work on that https://www.openhackathons.org/s/siteevent/a0CUP000013Tp8f2AC/se000375 @prometheusred ? @kazewong ? @Andrew-S-Rosen ?

see dagster-slurm for a list of tasks. Even if you do not have time to work hands on-but have experience with SLURM based systems it would be great if you can review the task list if something critical is missing dagster-slurm

[geoHeil]

And https://github.com/ascii-supply-networks/dagster-slurm to eventually hold the code

[kazewong]

I am glad to see someone is taking initiative to make this integration. Right now I am a bit overwhelmed in work so I won't be able to contribute other than reviewing the code from time to time, but I will certainly keep an eye on this.

[prometheusred]

I am interested especially if i can make it relevant to my current project. I’ll have to figure out if / how much time commitment I can actually make based on that. I haven’t looked closely at your notes or suggested tools (yet) - but I don’t have experience with pixi or ray specifically. I don’t know much much it matters but my slurm environment is RHEL based and on-prem though I will have integrations with snowflake.

[geoHeil]

would it make sense to ahve a quick chat about the envisioned architecture with both of you?