Using AWS secrets in `includeConfig` doesn't work

[bentsherman]

We added the ability to use secrets in the includeConfig path in #4840 . This works for local secrets, but not AWS secrets, because AWS secrets requires the xpack-amzn plugin and the config is resolved before plugins are loaded, since the config may include additional plugins.

To fix this I think we would need to load plugins in two phases, adding an "early" phase which happens before the config is resolved.

Not sure how much demand there is for this use case, but it would include any AWS Batch run in Seqera Platform that loads a private config using a secret credential.

cc @markpanganiban

[pditommaso]

Bit ugly, because plugins depends on the config 🤦

[pditommaso]

Likely assuming the AWS config does not rely in the config setting it's possible to load the AWS plugin to resolve the secrets

[pditommaso]

There are two possible approaches:

1. Fetching the branch to be used via the provider API, likely here
2. or using git remote via jgit (see here)

Tagging @tom-seqera for visibility

[ewels]

Bonus functionality that could be unlocked if we can get AWS credentials in for config loading: the ability to load a Nextflow config file from s3:// (in addition to local / http etc).

[jpfeuffer]

Not an expert but I have the feeling that not even access to secrets in pipeline scripts is possible without xpack-amzn being loaded before discovering the secrets providers.
At least I cannot get secrets to work on the head node (with AWS batch) which is a serious limitation for us.
No secrets in onComplete, no conditional steps based on secrets, etc.
Happy to open another issue though, if it is mistakenly unrelated.

[pditommaso]

Another possible solution to this problem. Re-implement secrets plugin over class Java ServiceLoader instead of Pf4j to avoid the circular dependency

[adamrtalbot]

Not an expert but I have the feeling that not even access to secrets in pipeline scripts is possible without xpack-amzn being loaded before discovering the secrets providers.
At least I cannot get secrets to work on the head node (with AWS batch) which is a serious limitation for us.
No secrets in onComplete, no conditional steps based on secrets, etc.
Happy to open another issue though, if it is mistakenly unrelated.

This isn't currently supported but there is a feature request here.

[bentsherman]

Another possible solution to this problem. Re-implement secrets plugin over class Java ServiceLoader instead of Pf4j to avoid the circular dependency

I don't think this will make a difference because the AWS secrets provider still lives in the nf-amazon plugin, so you still need to load nf-amazon before you can pick up the secrets provider