[Terraform] Image creation fails after 1 hour

[mfontana-elem]

ParallelCluster version: v3.11.0
aws-parallelcluster version: v1.1.0

When trying to build an image via Terraform, after exactly 1 hour, the creation on the Terraform side fails. I have verified this 3 times. If I read the relevant CloudWatch logs, the build is still executing at the moment of Terraform failure, and after more time, it finishes correctly. However, the resource is left in a tainted state on Terraform side, forbidding IaC management.

Relevant Terraform output:

module.images.module.images["hpc6a.48xlarge"].aws-parallelcluster_image.main: Still creating... [1h0m1s elapsed]
╷
│ Error: Image create failed to complete.
│ 
│   with module.images.module.images["hpc6a.48xlarge"].aws-parallelcluster_image.main,
│   on ../../../modules/common/images/image/image.tf line 26, in resource "aws-parallelcluster_image" "main":
│   26: resource "aws-parallelcluster_image" "main" {
│ 
│ Error: 403 Forbidden

My first reflex was to think there was a 1 hour timeout on the Terraform resource code, so I created #6891. However, while inspecting the code, I realized it SHOULD have a 3 hours timeout.

Hence, my only suspect is that an assumed role is reaching its max duration or something like that. How can I go and try to debug this problem?

Best regards,

[mfontana-elem]

On closer inspection, it seems the problem is that the Terraform provider assumes the Lambda role for a duration of 3600 seconds. Relevant CloudTrail logs:

    "requestParameters": {
        "roleArn": "arn:aws:iam::XXXXXXXXXXXXXXXXX:role/ParallelClusterLambdaRole-XXXXXX",
        "roleSessionName": "tracing",
        "durationSeconds": 3600
    },

If I understand correctly, my admin aws session is making the API call and assuming the ParallelClusterLambdaRole. Then, this role is being utilized for polling until the cloud formation stack is completed. However, the default duration of 3600 is being used, and I cannot find a way to modify that externally when looking at the source code for the provider. Also, I don't see code for trying to refresh the role credentials in the provider configuration. Hence, after 1 hour of building the image, this polling fails with a 403 Forbidden.

Note that I've manually increased the max duration parameter for the relevant IAM Roles. Yet, it seems like the Terraform provider is using the typical 3600 duration instead of going for the max duration.

Is this a bug (or at least an oversight) in aws-parallelcluster provider or am I understanding something wrong?

Best regards,

[mfontana-elem]

After 1 hour, I don't get any AccessDenied messages in CloudTrail except for this:

        "sessionContext": {
            "sessionIssuer": {
                "type": "Role",
                "principalId": "XXXXXXXXXXXXXXXX",
                "arn": "arn:aws:iam::XXXXXXXXX:role/ParallelClusterLambdaRole-XXXXXXXX",
                "accountId": "XXXXXXXXXXX",
                "userName": "ParallelClusterLambdaRole-XXXXXXXXX"
            },
       ....
    "errorMessage": "User: arn:aws:sts::XXXXXXXXX:assumed-role/ParallelClusterLambdaRole-XXXXX/ParallelClusterAPIv3-13-0-ParallelClusterFunction-XXXXXXX is not authorized to perform: cloudformation:DescribeStackResource on resource: arn:aws:cloudformation:eu-north-1:XXXXXXXX:stack/ami-XXXXXXXX/XXXXXXXXXX because no identity-based policy allows the cloudformation:DescribeStackResource action",

Indeed, the role generated by the aws-tf/parallelcluster module (following this) does not have a DescribeStackResource, but rather a DescribeStackResources (note the s) permission.

However, this error is triggered every 10 seconds since I run the apply, so to explain a failure after 60 minutes some very long error count threshold (~ 360 errors) should be considered. And I don't see such threshold in the code. Also, it doesn't explain why if my build + test + publish pipeline takes less than an hour, terraform apply succeeds.

There are probably some API operations not being logged in CloudTrail and those are the ones 403'ing after an hour. Although I do see some AssumeRole operations before expiration, it seems the provider might not be refreshing credentials correctly.

---

P.S: Manually adding DescribeStackResource to the ParallelClusterLambdaRole's policies don't solve this. But they do result in some empty "Reason: " messages to become populated (for instance, VCpuLimitExceeded in my playground account). If you want I can't create a separate issue for adding this to ParallelClusterLambdaRole on creation of the API stack.

[hehe7318]

Hi @mfontana-elem,

Thanks for your investigation, it seems like a STS-token expiration:

• The provider assumes ParallelClusterLambdaRole-* for the default 3600 seconds.
• It keeps polling the Image Builder stack with the same credentials.
• After one hour the token expires and the next call returns 403 Forbidden – Terraform marks the resource tainted even though the AMI is still being built and will eventually succeed.

Regarding this public official document: https://sdk.amazonaws.com/swift/api/awsiam/1.3.32/documentation/awsiam/createroleinput/maxsessionduration

The maximum session duration (in seconds) that you want to set for the specified role. If you do not specify a value for this setting, the default value of one hour is applied. This setting can have a value from 1 hour to 12 hours. Anyone who assumes the role from the CLI or API can use the DurationSeconds API parameter or the duration-seconds CLI parameter to request a longer session. The MaxSessionDuration setting determines the maximum duration that can be requested using the DurationSeconds parameter. If users don’t specify a value for the DurationSeconds parameter, their security credentials are valid for one hour by default.

I checked our template, https://us-east-1-aws-parallelcluster.s3.us-east-1.amazonaws.com/parallelcluster/3.11.0/templates/policies/policies.yaml, it doesn't explicitly set a MaxSessionDuration. So 3600 seconds is applied.

I will talk to my team on Monday next week. We will create a backlog task to investigate further and fix it. Sorry for the inconvenience.

Best Regards,
Xuanqi He

[mfontana-elem]

Hi Xuanqi,

Indeed I tried manually changing the MaxSessionDuration, but if the API calls don't assume the role with extra duration, the sessions still default to 3600.

The best alternative would be for the provider to re-assume the role on expired tokens.

If that's somehow not attainable, perhaps some high max duration has to be set for the role (on creation) and assume the role with a session duration equal (or slightly larger) than the resource timeout.

Best regards,

P.S: Although harmless, I guess it would also be nice to fix the 403 on calling DescribeStackResource.

[hehe7318]

Indeed I tried manually changing the MaxSessionDuration, but if the API calls don't assume the role with extra duration, the sessions still default to 3600.

Yes, because that's the parameter controls the maximum value. In ParallelCluster API also need to explicitly set DurationSeconds parameter to increase the session duration time.

Discussing with my team, it should can be fixed in next release 3.14.0.