ServiceAccount token expiration

[akunafin]

After the upgrade of the kubernetes cluster to 1.21 we've got a notification from AWS that some of our applications do not refresh the service account tokens. It's related to BoundServiceAccountTokenVolume feature that is enabled by default starting from 1.21.
It looks like that there is a need to upgrade the client sdk. Or maybe there is another workaround that you can suggest.

[jszwedko]

Hi @akunafin !

The token refresh feature was added to Vector as of version 0.21.0. Are you running an older version of Vector?

[jszwedko]

Thanks @akunafin ! I'd try 0.21.1 since there were a couple of bug fixes included which may or may not be relevant.

[akunafin]

After upgrade from 0.20.1 to 0.21.1 I've started getting the following error for every S3 sink that I have:

2022-05-17T15:05:17.459519Z ERROR sink{component_kind="sink" component_id=phoenix_s3_debug component_type=aws_s3 component_name=phoenix_s3_debug}:request{request_id=0}: vector_core::stream::driver: Service call failed. error=ConstructionFailure(CredentialsLoadingError(ProviderError { cause: ServiceError { err: AssumeRoleWithWebIdentityError { kind: Unhandled(Error { code: Some("AccessDenied"), message: Some("Not authorized to perform sts:AssumeRoleWithWebIdentity"), request_id: Some("5123babe-155a-4137-ad03-86f359be98a5"), extras: {} }), meta: Error { code: Some("AccessDenied"), message: Some("Not authorized to perform sts:AssumeRoleWithWebIdentity"), request_id: Some("5123babe-155a-4137-ad03-86f359be98a5"), extras: {} } }, raw: Response { inner: Response { status: 403, version: HTTP/1.1, headers: {"x-amzn-requestid": "5123babe-155a-4137-ad03-86f359be98a5", "content-type": "text/xml", "content-length": "299", "date": "Tue, 17 May 2022 15:05:16 GMT"}, body: SdkBody { inner: Once(Some(b"<ErrorResponse xmlns=\"https://sts.amazonaws.com/doc/2011-06-15/\">\n  <Error>\n    <Type>Sender</Type>\n    <Code>AccessDenied</Code>\n    <Message>Not authorized to perform sts:AssumeRoleWithWebIdentity</Message>\n  </Error>\n  <RequestId>5123babe-155a-4137-ad03-86f359be98a5</RequestId>\n</ErrorResponse>\n")), retryable: true } }, properties: SharedPropertyBag(Mutex { data: PropertyBag, poisoned: false, .. }) } } })) request_id=0

When I switch to the previous version it works fine. Am I missing some change that might be relevant?

[jszwedko]

Interesting, are you using web identity for authentication with 0.20.1? One difference with 0.21.1 is that all AWS components now use the IMDSv2 for metadata credentials. Previously they used IMDSv1.

[akunafin]

I've found a configuration mistake in the AWS role. Don't know how it worked with 0.20.1, but after I've fixed it, it works well with 0.21.1.
Also after the upgrade I don't see stale tokens anymore. Thanks for your help!

[jszwedko]

Great, thanks for confirming!