Merge branch 'master' into 6158_azcopy_hides_error_message

Signed-off-by: adamrtalbot <12817534+adamrtalbot@users.noreply.github.com>

Author: adamrtalbot

docs/aws.md

@@ -348,14 +348,13 @@ The grandparent directory of the `aws` tool will be mounted into the container a
 
 ### Docker installation
 
-Docker is required by Nextflow to execute tasks on AWS Batch. The **Amazon ECS-Optimized Amazon Linux 2 (AL2) x86_64 AMI** has Docker installed, however, if you create your AMI from a different AMI that does not have Docker installed, you will need to install it manually.
+Docker is required by Nextflow to execute tasks on AWS Batch. The **Amazon ECS-optimized Amazon Linux 2023 AMI** has Docker installed, however, if you create your AMI from a different AMI that does not have Docker installed, you will need to install it manually.
 
 The following snippet shows how to install Docker on an Amazon EC2 instance:
 
 ```bash
 # install Docker
 sudo yum update -y
-sudo amazon-linux-extras install docker
 sudo yum install docker
 
 # start the Docker service
@@ -365,21 +364,20 @@ sudo service docker start
 sudo usermod -a -G docker ec2-user
 ```
 
-You may have to reboot your instance for the changes to `ec2-user` to take effect.
+You must logging out and logging back in again to use the new `ec2-user` permissions.
 
 These steps must be done *before* creating the AMI from the current EC2 instance.
 
 ### Amazon ECS container agent installation
 
 The [ECS container agent](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ECS_agent.html) is a component of Amazon Elastic Container Service (Amazon ECS) and is responsible for managing containers on behalf of ECS. AWS Batch uses ECS to execute containerized jobs, therefore it requires the agent to be installed on EC2 instances within your Compute Environments.
 
-The ECS agent is included in the **Amazon ECS-Optimized Amazon Linux 2 (AL2) x86_64 AMI** . If you use a different base AMI, you can also install the agent on any EC2 instance that supports the Amazon ECS specification.
+The ECS agent is included in the **Amazon ECS-optimized Amazon Linux 2023 AMI** . If you use a different base AMI, you can also install the agent on any EC2 instance that supports the Amazon ECS specification.
 
 To install the agent, follow these steps:
 
 ```bash
-sudo amazon-linux-extras disable docker
-sudo amazon-linux-extras install -y ecs
+sudo yum install ecs-init
 sudo systemctl enable --now ecs
 ```
 

docs/config.md

@@ -248,7 +248,7 @@ With the above configuration:
 - All processes will use 4 cpus (unless otherwise specified in their process definition).
 - Processes annotated with the `hello` label will use 8 cpus.
 - Any process named `bye` (or imported as `bye`) will use 16 cpus.
-- Any process named `bye` (or imported as `bye`) invoked by a workflow named `mysub` with use 32 cpus.
+- Any process named `bye` (or imported as `bye`) invoked by a workflow named `mysub` will use 32 cpus.
 
 (config-profiles)=
 

docs/reference/config.md

@@ -417,7 +417,7 @@ The following settings are available:
 `azure.batch.poolIdentityClientId`
 : :::{versionadded} 25.05.0-edge
   :::
-: Specify the client ID for an Azure [managed identity](https://learn.microsoft.com/en-us/entra/identity/managed-identities-azure-resources/overview) that is available on all Azure Batch node pools. This identity will be used for task-level authentication to Azure services. See {ref}`azure-managed-identities` for more details.
+: Specify the client ID for an Azure [managed identity](https://learn.microsoft.com/en-us/entra/identity/managed-identities-azure-resources/overview) that is available on all Azure Batch node pools. This identity will be used by Fusion to authenticate to Azure storage. If set to `'auto'`, Fusion will use the first available managed identity.
 
 `azure.managedIdentity.clientId`
 : Specify the client ID for an Azure [managed identity](https://learn.microsoft.com/en-us/entra/identity/managed-identities-azure-resources/overview). See {ref}`azure-managed-identities` for more details. Defaults to environment variable `AZURE_MANAGED_IDENTITY_USER`.

plugins/nf-azure/src/main/nextflow/cloud/azure/fusion/AzFusionEnv.groovy

@@ -64,7 +64,13 @@ class AzFusionEnv implements FusionEnv {
         // If pool has a managed identity, ONLY add the MSI client ID
         // DO NOT add any SAS token or reference cfg.storage().sasToken
         if (managedIdentityId) {
-            result.FUSION_AZ_MSI_CLIENT_ID = managedIdentityId
+            // Fusion will try and pick up a managed identity that is available.
+            // We recommend explicitly setting the config item to the managed ID so you know which one is being used.
+            // However if set to 'true' it will use whichever is available.
+            // This can be helpful if the pools have different managed identities.
+            if (managedIdentityId != 'auto') {
+                result.FUSION_AZ_MSI_CLIENT_ID = managedIdentityId
+            }
             // No SAS token is added or generated
             return result
         }

plugins/nf-azure/src/test/nextflow/cloud/azure/fusion/AzFusionEnvTest.groovy

@@ -243,4 +243,26 @@ class AzFusionEnvTest extends Specification {
         env.size() == 2  // Only account name and managed identity
     }
 
+    def 'should not provide explicit managed identity when pool identity is set to true'() {
+        given:
+        def NAME = 'myaccount'
+        Global.session = Mock(Session) {
+            getConfig() >> [azure: [
+                storage: [accountName: NAME],
+                batch: [poolIdentityClientId: 'auto']
+            ]]
+        }
+
+        when:
+        def config = Mock(FusionConfig)
+        def fusionEnv = new AzFusionEnv()
+        def env = fusionEnv.getEnvironment('az', config)
+
+        then:
+        env.AZURE_STORAGE_ACCOUNT == NAME
+        !env.FUSION_AZ_MSI_CLIENT_ID
+        !env.AZURE_STORAGE_SAS_TOKEN  // SAS token should NOT be present
+        env.size() == 1  // Only account name
+    }
+
 }