Assume role with aws s3 source

[ambroserb3]

I'm wondering if I'm missing something with the auth.assume role option for an S3 source?

This is our vector.toml file:

[sources.mlp-data-lake]
    type = "aws_s3"
    region = "$REGION"
    auth.assume_role = "$IAM_ROLE" 
    sqs.queue_url = "$SQS_QUEUE"

[sinks.opensearch]
    type = "elasticsearch" # required
    inputs = ["mlp-data-lake"] # required
    auth.strategy = "aws"
    aws.region = "$REGION"
    endpoints = ["$OS_DOMAIN"]

This works locally without assume_role (we use SSO locally), but in our cluster we're baffled. Assume role simply doesn't work. We worked around this by creating an IAM User and just adding the AKID/SKID to the env variables on the pod, but we couldn't figure out why the assume_role option wasn't working. We're able to assume the role from the CLI when we exec into the pod, we also tried creating a k8s service account that assumes the role. The role does have the correct permissions in its access policy.

But it kept giving us this error:

The credential provider was not enabled: no providers in chain provided credentials

until we created the service account, then the error changed to:

An error occurred while loading credentials: Error { code: "AccessDenied", message: "Not authorized to perform sts:AssumeRoleWithWebIdentity"

But we triple checked this iam role access policy it definitely has this permission.

I was wondering where assume_role falls into this hierarchy in the docs:

Vector checks for AWS credentials in the following order:

1. The access_key_id and secret_access_key options.
2. The AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables.
3. The AWS credentials file (usually located at ~/.aws/credentials).
4. The IAM instance profile (only works if running on an EC2 instance with an instance profile/role). Requires IMDSv2 to be enabled. For EKS, you may need to increase the metadata token response hop limit to 2.

Like am I missing something here, do we still need one of these when assuming a role?

Has anyone else run into this issue with auth.assume_role and found a solution?

[jszwedko]

Hey @stephanrb3 !

One thing you could try is setting the environment variable VECTOR_LOG to aws_config=debug. This will enable debug logging by the AWS SDK that might make it clearer where it is retrieving credentials from.

The assume role should be applied after credentials are loaded. If unspecified in the Vector config, as you have it, it should use the order mentioned by the SDK here: https://docs.aws.amazon.com/sdk-for-rust/latest/dg/credentials.html.