Add ability to refresh service account token to AWS Container Insights Kueue Receiver

[rvasahu-amazon]

Description

This PR:

1. Adds a new component BearerTokenProvider with a default behaviour of loading the service account token from the cluster.
2. Updates the awsContainerInsightKueueReceiver to use a BearerTokenProvider for both initially retrieving the bearer token and refreshing it as necessary.
3. Separates scrape configuration and prometheus receiver setup into helper functions from the KueuePrometheusScraper constructor.
4. Other minor documentation and renaming changes.

Link to tracking issue

Tracking issues are internal.

Testing

Unit testing was added for the BearerTokenProvider.

Manual end-to-end validation is pending. This should not be merged before that is complete.

Documentation

No documentation was added.

[chadpatel]

A flaw in our current architecture which I didn't really think about until seeing your PR is that the BearerToken is provided at sub-receiver initialization. I think updating the config to use a CredentialsFile/BearerTokenFile would fix it and not require the restart

diff --git a/receiver/awscontainerinsightskueuereceiver/internal/kueue_scraper/kueue_prometheus_scraper.go b/receiver/awscontainerinsightskueuereceiver/internal/kueue_scraper/kueue_prometheus_scraper.go
index 5138b91d47..541a32c21a 100644
--- a/receiver/awscontainerinsightskueuereceiver/internal/kueue_scraper/kueue_prometheus_scraper.go
+++ b/receiver/awscontainerinsightskueuereceiver/internal/kueue_scraper/kueue_prometheus_scraper.go
@@ -88,6 +88,10 @@ func NewKueuePrometheusScraper(opts KueuePrometheusScraperOpts) (*KueuePrometheu
                        TLSConfig: configutil.TLSConfig{
                                InsecureSkipVerify: true,
                        },
+                       Authorization: &configutil.Authorization{
+                               Type:            "Bearer",
+                               CredentialsFile: "/var/run/secrets/kubernetes.io/serviceaccount/token",
+                       },
                },
                ScrapeInterval:  model.Duration(kmCollectionInterval),
                ScrapeTimeout:   model.Duration(kmCollectionInterval),
@@ -114,12 +118,6 @@ func NewKueuePrometheusScraper(opts KueuePrometheusScraperOpts) (*KueuePrometheu
                MetricRelabelConfigs: GetKueueRelabelConfigs(opts.ClusterName),
        }

-       if opts.BearerToken != "" {
-               scrapeConfig.HTTPClientConfig.BearerToken = configutil.Secret(opts.BearerToken)
-       } else {
-               opts.TelemetrySettings.Logger.Warn("bearer token is not set, kueue metrics will not be published")
-       }
-
        promConfig := prometheusreceiver.Config{
                PrometheusConfig: &prometheusreceiver.PromConfig{
                        ScrapeConfigs: []*config.ScrapeConfig{scrapeConfig},

Better would be to use the path containerinsightscommon.BearerToken instead of hardcoding. And then I think we can remove the BearerToken option.

Trying to balance good enough here (which I know I was pushing you on last week) with also the smallest change possible

[rvasahu-amazon]

A flaw in our current architecture which I didn't really think about until seeing your PR is that the BearerToken is provided at sub-receiver initialization. I think updating the config to use a CredentialsFile/BearerTokenFile would fix it and not require the restart.

Hey @chadpatel, as discussed offline that seems plausible. Testing that now and I'll update this PR when that's complete.

Better would be to use the path containerinsightscommon.BearerToken instead of hardcoding. And then I think we can remove the BearerToken option.

Is the containerinsightscommon module from amazon-cloudwatch-agent available to this package? If not I think it'd be possible to pass it as a config field, but that would require a PR to that package as well.

[chadpatel]

Is the containerinsightscommon module from amazon-cloudwatch-agent available to this package? If not I think it'd be possible to pass it as a config field, but that would require a PR to that package as well.

I don't know. I think the best solution is to hard code it and we can figure out the right approach when we backport this change to our other usages of the bearer token for container insights

[chadpatel]

We attempted the CredentialsFile approach on a screen share and no luck. Due to urgency we will need to go with the approach in this PR

[rvasahu-amazon]

We attempted the CredentialsFile approach on a screen share and no luck. Due to urgency we will need to go with the approach in this PR

Yep. As discussed, testing that on my end and will look to see if there's a way to manually induce a token refresh for a quicker turnaround on that.

[rvasahu-amazon]

Cancelling this PR as was able to get CredentialsFile approach to succeed. Issue was our validation procedure was faulty. Will raise a new PR with those changes instead.