Issues when trying to override CLI credentials on an EC2 instance

[rankol]

I am creating an IAM User and then generating CLI credentials for that user. On an EC2 instance I would like to use these credentials for CLI commands instead of using the EC2 instance profile permissions. I attempt to create a new credentials file and update it with the following commands.

touch creds
export AWS_SHARED_CREDENTIALS_FILE="creds"
aws configure set aws_access_key_id $ACCESS_KEY_ID --profile test
aws configure set aws_secret_access_key $SECRET_ACCESS_KEY --profile test

I can read the creds file and I see a [test] block with credentials below that.

I try to run aws sts get-caller-identity --region <REGION> --profile test to confirm that my user's credentials are being used which gives the following error:

An error occurred (AccessDenied) when calling the GetCallerIdentity operation: The security token included in the request is invalid.

Is there something I'm doing incorrectly here? Is there a security token associated to the default profile or EC2 instance profile that is being used with the credentials for the test profile?

One command which worked during testing was aws sts get-caller-identity --region <REGION> --profile default. This seems to update the creds file with a [default] block and an associated set of new credentials for what I'm assuming is the EC2 instance profile. Do I need those default credentials in the new credentials file?

[RyanFitzSimmonsAK]

Hi @rankol, thanks for reaching out. Having a token invalid error on get-caller-identity is definitely strange, since that command doesn't require any permissions. Do you have aws_session_token in your profile or AWS_SESSION_TOKEN as an environment variable? If so, try clearing it.

I was able successfully make my profile using the same process you followed, but it might be worth it for you to try another method. Could you try making your profile by importing CSV credentials? Here's the associated documentation for that, specifically the section on using named profiles and aws configure import, but I'll go over the steps.

1. Generate CSV credentials from the IAM web console.
2. Edit them in Notepad so that the User Name is included (in the order User Name, Access key ID, Secret access key)
3. Copy the CSV over to your EC2 instance.
4. Import the profile using aws configure import --csv file://myCreds.csv

If you want to use these credentials for all commands without needing to specify them using --profile, you can set them as the default using export AWS_PROFILE=userName.

Let me know if that works for you, or if you have any follow-up questions. Thanks!

[rankol]

Hi @RyanFitzSimmonsAK, sorry for the late response. Just to follow up I was able to resolve this by adding in a sleep 30 command to the bash script in my setup.

I was using Hashicorp Vault to generate dynamic credentials for a temporary IAM user and it seems that attempting to use these credentials too quickly will result in the error I mentioned in my original post. This seems to not error out when waiting for a short amount of time before attempting to use the credentials from the newly created user.

Edit: Somewhat of a similar issue to this. I can't find the exact documentation that mentioned IAM User credentials reach eventual consistency after some time.

hashicorp/vault#687