Remote Nextflow Execution #8
Description
A few users have requested that Napari can be run locally, but still execute commands on HPC. This is opposed to the current setup, where there is an expectation that Napari is run on the HPC using Open OnDemand or some other kind of virtual desktop.
The original design wanted to avoid this functionality for the following reasons:
- One group of intended users are non-computational imaging experts, who may be unfamiliar with HPC, ssh etc. This group will not use the Nextflow backend in isolation, and always use the Napari plugin or other front-end.
- For data that is opened in-memory, HPC can provide a lot more resource (memory) for this. Note, with proper lazy loading and/or NGFF, this is less of an issue.
- It adds several points of failure, that can be specific to the user and/or institutional setup, potentially adding confusion and increasing developer burden.
However, the flexibility of having the option to run locally and still submit to HPC is clearly attractive, hence multiple requests. This could be achieved by simply allowing users to specify a ssh key and a destination, and then if specified integrate into the nxf_cmd to use ssh to evoke the command.
This could cause issues depending on institutional setup, potentially adding load to login nodes etc. but this would be considered an advanced feature where those who use it are expected to be aware of this.