Create a role and assume it right away with NewRoleExistsWaiter

[gforien]

I am trying to create an IAM role and assume it.

Because this is an async operation, I must wait for the role creation. However, the waiter NewRoleExistsWaiter responds right away without error, and so the subsequent AssumeRole operation fails.

Please find my code snippet below

details

package yourpackage

import (
	"context"
	"fmt"
	"math/rand"
	"testing"
	"time"

	"github.com/aws/aws-sdk-go-v2/aws"
	"github.com/aws/aws-sdk-go-v2/config"
	"github.com/aws/aws-sdk-go-v2/credentials/stscreds"
	"github.com/aws/aws-sdk-go-v2/service/iam"
	"github.com/aws/aws-sdk-go-v2/service/sts"
	"github.com/stretchr/testify/require"
)

func TestCreateAssume(t *testing.T) {
	config1, _ := config.LoadDefaultConfig(context.TODO())
	iamClient := iam.NewFromConfig(config1)

	// 1. Define new role parameters (the role ARN is known beforehand)
	testRoleArn := "yourARN"
	testRoleName := fmt.Sprintf("test-role-%d", rand.Int())
	testRoleAssumePolicy := fmt.Sprintf(`{
        "Version": "2012-10-17",
        "Statement": [
            {
                "Effect": "Allow",
                "Principal": {
                    "AWS": "%s"
                },
                "Action": "sts:AssumeRole"
            }
        ]
	}`, testRoleArn)


    // 2. Create the new role
	res, err := iamClient.CreateRole(context.TODO(), &iam.CreateRoleInput{
		RoleName:                 &testRoleName,
		AssumeRolePolicyDocument: &testRoleAssumePolicy,
	})
	require.NoError(t, err)
	responseTestRole := res.Role


	// 3. Wait for the new role to be available

	// 3.1. Wait manually
	// time.Sleep(20 * time.Second)

	// 3.2. Or use a waiter
	testRoleWaiter := iam.NewRoleExistsWaiter(iamClient)
	maxWaitTime := 20 * time.Second
	out, err := testRoleWaiter.WaitForOutput(context.TODO(), &iam.GetRoleInput{
		RoleName: responseTestRole.RoleName,
	}, maxWaitTime)
	fmt.Print(out)        // <- set breakpoint here
	require.NoError(t, err)


    // 4. Configure a new CredentialProvider (AssumeRole)
	baseStsClient := sts.NewFromConfig(config1)
	assumeRoleProvider := stscreds.NewAssumeRoleProvider(baseStsClient, *responseTestRole.Arn,
		func(opts *stscreds.AssumeRoleOptions) {
			opts.RoleSessionName = "new-session"
		},
	)
	config2 := *aws.NewConfig()
	config2.Credentials = aws.NewCredentialsCache(assumeRoleProvider)
	config2.Region = config1.Region


    // 5. Test credential process with STS.GetCallerIdentity
	testStsClient := sts.NewFromConfig(config2)
	_, err = testStsClient.GetCallerIdentity(context.TODO(), &sts.GetCallerIdentityInput{})
	require.NoError(t, err)
}

Expected result

The test is ok. ✅

Actual result

Error 403 Unauthorized

operation error STS: GetCallerIdentity
failed to sign request: failed to retrieve credentials: failed to refresh cached credentials
operation error STS: AssumeRole
https response error StatusCode: 403
RequestID: xxxxxx
api error AccessDenied: User: xxxxxx is not authorized to perform: sts:AssumeRole on resource: yyyyyy

(403 Unauthorized is not an IAM permissions-related issue in this context, it's the assumeRole operation failing

Circumvention

You can verify that waiting manually works in this scenario by uncommenting section 3.1.

Am I using the wrong waiter ? I did not find an appropriate waiter in package sts or stscreds

Thank you for your time

[RanVaknin]

Hi @gforien

TLDR: It's not possible without a workaround.

Am I using the wrong waiter ? I did not find an appropriate waiter in package sts or stscreds

I think you are pretty much spot on in your diagnosis.
The IAM waiter works as expected. In IAM the role is created and immediately available so that's why your waiter returns so quickly (you can enable logs and see the 200 status getRole response).
I believe what you are experiencing is just propagation delay time between STS and IAM. Since STS does not offer the same waiter it cannot be used to rely on the waiter from a standpoint of different services.

You can implement some hacky solutions to get it to work, for example sleeping your main thread like you did here, or a little bit more graceful (but still hacky) is implement a retry mechanism for that AssumeRole operation to retry on AccessDenied errors.

As a side note, In my experience developing on AWS I never had to create a role and immediately assume it.
Can you share your actual use case for "on demand" role creation and assumption?

Thanks,
Ran~

[gforien]

Hello Ran,
Thank you for your answer

I write end-to-end terratests on terraform modules for AWS. I needed my tests to be isolated, reproducible, and ran in parallel. Also each test scenario runs with its own set of minimal IAM permissions. The simplest method I found was to create a new role at test setup, assume it for the test, and destroy it at teardown.

I wrote an update with the relevant methods I used to wait on the STS credentials retrieval, so I took the liberty to mark that other comment as the answer. Please correct me if I'm wrong.

You seem to say that using a retry mechanism on error would be hacky ? I don't understand why. It seems to me that this is precisely how others waiters work (retrying on error, with exp backoff and jitter).

Thanks,
Gabriel

[gforien]

I ended up using my own waiter.

If you encounter the same problem, it's fairly simple to use the underlying ComputeDelay() method from smithy-go, which provides the exponential backoff with jitter that you need. Here is an example. I waited on method CredentialsCache.Retrieve. I did not use CredentialsCacheOptions.

[gforien]

(of course the operation + waiting on error are enclosed in a for loop, otherwise you would wait only once)

[github-actions[bot]]

Hello! Reopening this discussion to make it searchable.