Relationship between resource pool and hpc_connect resources

[tjfulle]

As of 7e3758b, batching was removed from Canary's core and moved to the canary_hpc plugin. The canary_hpc plugin interacts with hpc_connect and sets Canary's config.resource_pool. This poses a problem for setting resources on the command line during batched runs. Eg,

canary -c resource_pool:gpus:4 run -b spec=...

The canary_hpc plugin sets the config.resource_pool based on the hpc_connect backend's resources, so the specification of gpus:4 above is currently lost.

Idea:

In canary_hpc.Conductor.setup, look for configuration modifications and apply them to the hpc_connect backend.

[mdmosby]

We had a good discussion on this topic and generally agreed that HPCC should fully own population of the resource pool for a batch/job-submission workflow; however some details might be devils

• Multi-platform config file for supporting a site-wide configuration
  • Agreed that we need this, but need to develop a spec (human read/write important)
• how to manage oversubscription? E.g., GPUs?
  • maybe OK to continue with -c resource_pool:slots_per_gpu:N?

[tjfulle]

As @mdmosby relayed above - the resource set used to submit jobs to hpc_connect should be managed by hpc_connect and not canary. That removes some complexity in the interaction between canary, hpc_connect, and the resource pool.

Two recent developments aid in the resolution of the discussion:

• sandialabs/hpc-connect@030c796: plugin hooks for defining the default hpc_connect resource set. A plugin to read a multi-platform config file, given by the HPC_CONNECT_HOSTFILE environment variable was implemented.
• The --oversubscribe TYPE=N option was implemented in 22b1ef8

I believe all that is left on this issue is to properly document the resource set relationship.