Adding MPI Support

[jtronge]

I'm creating this discussion to talk about how we can add some level of MPI support to BEE. I'm thinking this would require some modifications to the Worker code in order to wrap steps with the proper mpirun command. It may also require some modifications to the container runtime classes, but we might be able to get around that.

Also, we'll probably need to have some sort of new hint or requirement for MPI. The CWL team published a paper last year about a cwltool:MPIRequirement extension that they experimented with. Perhaps we can use this requirement to some extent, or come up with something else that's better suited to HPC.

[pagrubel]

For updating for Slurm and Charliecloud it looks fairly simple see:
https://hpc.github.io/charliecloud/tutorial.html#non-interactive-mpi-hello-world

[jtronge]

Using Singularity+MPI looks to be fairly straightforward based on their docs: https://sylabs.io/guides/3.3/user-guide/mpi.html. It does look like the container build process, such as for including the MPI libraries, may be more complicated.

[jtronge]

I wanted to make a note about Quincy's comment for using prefixes for requirements/hints in order to avoid polluting the CWL namespace. I found an example of this with Arvados. They use an arv: prefix. I think we might be able to do something like beeflow:SomeRequirement like they are doing.

[jtronge]

After looking over the worker and container runtime code some more, I think we may need to modify the container runtime interface. Right now the container code simply generates script text to be appended onto the job submission scripts. This makes prepending to the main command difficult within the worker. An example of why this seems to be necessary is in the Charliecloud link that Pat posted above. You can see that there are some commands that just set up the environment, like the module load commands and then other commands that need to have srun prepended in order to run on the allocated nodes, if I understand the example correctly.

I'm thinking that maybe instead of generating text, the container runtime could instead generate a list of special Command objects with specific properties (in a new method commands(task)). Then the worker would use each special command to build the final submission script. Does that seem like the best way to do this?

[rstyd]

Hey Jake,
I'll have to check out the code a bit more, but that sounds like a solid plan to me. The granularity is definitely too low at the moment.

[pagrubel]

This does sound like a good idea. I'm not sure what specific properties would be, something like pre-pend?

[jtronge]

I'm thinking that Command object might have some sort of type property. We might have an environment type that corresponds to environment commands like module load charliecloud and then a compute or all-node type that would correspond the ch-tar2dir ... and ch-run ... commands that should be prepended.

[pagrubel]

We'd also need an order
for instance on the main compute node you may run
srun ch-tar2dir
then
srun --cpus-per-task=1 ch-run ...

[jtronge]

Yeah. The container runtime will probably need to return an in-order list of Command objects. These will then be concatenated in order by the worker.