Usage of secrets in Workflow Script on AWS Batch and Platform

[FriederikeHanssen]

Bug report

• Calling workflow secrets in the onComplete block does not work locally and on AWS Batch (submitted with Platform)
• Calling workflow secrets in the workflow body does not appear to work on AWS Batch (submitted with Platform)

Expected behavior and actual behavior

I would like to use secrets as described here in both the workflow and onComplete block.

Steps to reproduce the problem

I created a small example here: https://github.com/FriederikeHanssen/mre/blob/secrets/main.nf

For testing locally:

1. Set a secret: nextflow secrets set RIKE_SECRET "Hello world"
2. Run pipeline: (tested with both, version that originally released it and newer version):
   - NXF_VER=24.03.0-edge nextflow run main.nf
   - NXF_VER=24.10.5 nextflow run .

For testing on Platform + AWS Batch

1. Set up a workspace etc. Define a RIKE_SECRET. Add the pipeline to the launchpad, ensure it is part of the Workflow secrets.
2. Run pipeline

Program output

Locally:

❯ nextflow run .

 N E X T F L O W   ~  version 24.10.5

Launching `./main.nf` [special_montalcini] DSL2 - revision: d462ac85b4

The secret in Workflow is: Hello world
ERROR ~ Failed to invoke `workflow.onComplete` event handler

 -- Check script 'main.nf' at line: 6 or see '.nextflow.log' file for more details

AWS Batch & platform:

The secret in Workflow is: null

ERROR ~ Failed to invoke `workflow.onComplete` event handler
-- Check script '.nextflow/assets/FriederikeHanssen/mre/main.nf' at line: 6 or see 'nf-1Sc7D9V8WrFVRF.log' file for more details

Environment

• Nextflow version: 24.10.5 & 24.03.0-edge
• Java version: [?]
• Operating system: macOS & AWS Batch
• Bash version: zsh 5.9 (arm64-apple-darwin24.0)

Additional context

[jpfeuffer]

Related to #5312 and the associated community feature request. We are also desperately looking for that functionality.
I think at least the docs should be fixed in the meantime.

Edit: wrong issue number

[bentsherman]

Per @tom-seqera , the issue is very likely that the secrets subsystem is now being initialized before the config loader (so that config files can use secrets), but this causes the secrets providers to be registered before the nf-amazon plugin can register the AWS secrets provider, so it never gets registered and AWS secrets don't work.

Additionally, the AWS secrets provider has apparently not implemented some methods that are needed to use AWS secrets in the pipeline script.

[pditommaso]

@FriederikeHanssen do you have the nextflow logs?

[FriederikeHanssen]

For the local run: nextflow_local.log

I don't get a log for the run on platform, but this is the Execution log:

The secret in Workflow is: null
Monitor the execution with Seqera Platform using this URL: https://cloud.seqera.io/orgs/scidev/workspaces/testing/watch/4unVNNJ5uEIZTr
ERROR ~ Failed to invoke `workflow.onComplete` event handler
-- Check script '.nextflow/assets/FriederikeHanssen/mre/main.nf' at line: 6 or see 'nf-4unVNNJ5uEIZTr.log' file for more details

[pditommaso]

For local run is not expected to work. I'm interesting instead in a launch via Platform, in that case secrets should work

[FriederikeHanssen]

Send you the link. For local, I think the docs need some updating then https://www.nextflow.io/docs/latest/secrets.html#pipeline-script:

This feature is only available when using the local or grid executors (Slurm, Grid Engine, etc). The AWS Batch executor allows the use of secrets when deploying the pipeline execution via Seqera Platform.

cc @christopher-hakkaart

[pditommaso]

The main on master looks empty https://github.com/FriederikeHanssen/mre/blob/main/main.nf

update: got it. it's a secrets branch

[pditommaso]

@bentsherman the problem you are mentioning is related to the use of secrets in the config, in the workflow script it should be ok. Here there's a problem with onComplete handler (local executor) and and general one when using batch + xpack

[bentsherman]

My theory (and Tom's) is that the run command is loading the secrets providers before loading plugins, which causes the AWS secrets provider to never be loaded:

nextflow/modules/nextflow/src/main/groovy/nextflow/config/ConfigBuilder.groovy

Line 340 in 431767a

binding.put('secrets', SecretsLoader.secretContext())

nextflow/modules/nextflow/src/main/groovy/nextflow/cli/CmdRun.groovy

Lines 328 to 340 in 431767a

	// create the config object
	final builder = new ConfigBuilder()
	.setOptions(launcher.options)
	.setCmdRun(this)
	.setBaseDir(scriptFile.parent)
	final config = builder .build()
	// check DSL syntax in the config
	launchInfo(config, scriptFile)
	// -- load plugins
	final cfg = plugins ? [plugins: plugins.tokenize(',')] : config
	Plugins.load(cfg)

This means the script would never receive the AWS secrets provider either

@FriederikeHanssen what needs to be updated in the docs? it says that AWS secrets are supported only when launching via platform

[pditommaso]

The circular dep is related to the config. The problem here is with the workflow script

[bentsherman]

The problems are related. Think about how the config's early loading of secret providers affects the script later on:

nextflow/modules/nextflow/src/main/groovy/nextflow/script/BaseScript.groovy

Line 98 in 431767a

binding.setVariable( 'secrets', SecretsLoader.secretContext() )

nextflow/modules/nextflow/src/main/groovy/nextflow/secret/SecretsLoader.groovy

Lines 40 to 48 in 431767a

	SecretsProvider load() {
	if( provider )
	return provider
	synchronized (this) {
	if( provider )
	return provider
	return provider = load0()
	}
	}

[FriederikeHanssen]

@bentsherman I understand this one here: "This feature is only available when using the local or grid executors " That is should either work on local/grid or on AWS but there only with platform

[pditommaso]

@bentsherman Don't believe so, because this works

https://github.com/FriederikeHanssen/mre/blob/secrets/main.nf#L3

and this does not

https://github.com/FriederikeHanssen/mre/blob/secrets/main.nf#L6

[bentsherman]

@FriederikeHanssen can you try this instead:

    workflow.onComplete = { // add the equals sign
        log.info("The secret on Complete is: ${secrets.RIKE_SECRET}")
    }