Add OIDC method to Kafka authentication

[richscott]

This change adds support for authentication via OIDC to the Kafka client. It provides an implementation of the sarama.AccessTokenProvider interface, supporting the client_credentials Grant Type, and a background token refresh.

[atoulme]

CLA is signed, getting reviews in.

[github-actions]

Welcome, contributor! Thank you for your contribution to opentelemetry-collector-contrib.

Important reminders:

• Please review our Contributing Guidelines.
• Don't forget to sign the Contributor License Agreement (CLA) if you haven't already.

A maintainer will review your pull request soon. Thank you for helping make OpenTelemetry better!

[axw]

@richscott thanks for the PR. I've skimmed the code, and the code looks good.

My main question at the moment is: does all of this OAuth2 logic have live in the Kafka components? There appears to be a significant overlap with what already exists in oauth2clientauthextension.

I think what would be ideal is if the Kafka components could be configured with the SASL/OAUTHBEARER mechanism, and the configuration is just a reference to an instance of oauth2clientauthextension. oauth2clientauthextension would need to be modified to expose an API for the Kafka client code to invoke to obtain a token.

Aside from that, franz-go is about to become the default and in future only client implementation. I would recommend that we start with franz-go, and perhaps not even bother with Sarama.

[axw]

FYI franz-go will soon be the default for both receiver and exporter, and when all the kinks are ironed out (in the not too distant future) the Sarama client will be removed.

[richscott]

@axw Thanks, and I agree completely with all your points/questions. This whole effort was animated by a request from another team of my employer, and they suggested this implementation to extend the Sarama client mechanism to do OIDC, and notably, to get the client_secret from a file (in usage scenario of using a Kubernetes pod secret file, which is dynamic and rotated frequently). I believe they may be using an older revision that does not have these facilities. I see now that the oauth2clientextension already (very closely) has this support, including client_id_file and client_secret_file. (Full disclosure: I'm new to this project/repo, and working with OIDC as well.) Also, as the franz-go client migration will occur soon(ish), I'll go ahead and change this code to use that in lieu of Sarama. I think I can excise a lot of this, work through the call-chain from the franz_client.go constructors, and then modify configkafka.SASLConfig and its users to use the oauth2clientextension.

[axw]

Thanks @richscott! That sounds good. I'll move this PR back to draft in the mean time.

[richscott]

Hi, @axw - I have substantially reworked the branch, as you suggested:

• The OIDC logic has been cut down, the code that maintained a background goroutine to periodically refresh tokens, etc - all that has been ripped out and replaced by references to types/functions in the repo's extension/oauth2clientauthextension package, so the changes are considerably smaller/simpler. I had to change a couple small things in the oauth2clientauthextension package code - make a public ClientCredentialsConfig so it was available to internal/kafka/oidc_client.go, added a TokenSource() constructor, etc, but other than that I have not modified anything in oauth2clientauthextension.
• The OIDC code remaining in internal/kafka now uses franz-go in lieu of the soon-deprecated sarama.
• I had to add a replace directive to the go.mod in each kafka-related component package in the repo, so the compiler would find and use the latest in the repo source, rather than downloading, so it would build.
• I made a few other miscellaneous tweaks to satisfying golangci-lint.

Running the golint and gotest Make targets are successful, and an external toy integration program I wrote to just verify that the client does indeed connect with a OIDC token to a Kafka instance, do a few basic list/delete/create topic operations, sleep for a bit, then repeat infinitely, does work correctly - it refreshes the token as needed, etc.

Probably there will be more work needed on this - I will mark this ready for review again and welcome comment/suggestions from everyone.

[axw]

@richscott thanks for the changes. I'm glad to see most of the logic now lives in oauth2clientauthextension. I'd like to see if we can go even further.

What I have in mind is that users will configure oauth2clientauthextension independently of the Kafka components, and then reference it via a new config attribute in the Kafka components. Something like this:

extensions:
  oauth2client:
    client_id_file: /path/to/client_id_file
    client_secret: /path/to/client_secret_file

exporters:
  kafka:
    auth:
      sasl:
        mechanism: OAUTHBEARER
        oauthbearer_token_source: oauth2client

The "oauthbearer_token_source" config setting would be the ID of an extension that implements a token source interface along these lines:

import "golang.org/x/oauth2"

type TokenSource interface {
    Token(context.Context) (*oauth2.Token, error)
}

This way we're not duplicating any oauth2 config, and would be closer to how the oauth2client extension works with HTTP/gRPC-based exporters -- see https://github.com/open-telemetry/opentelemetry-collector/blob/main/config/configauth/README.md

[axw]

Is this a mechanism name understood by anything else? How is this different to SASL/OAUTHBEARER?

[richscott]

This was a early-development mistake - I'll be definitely pulling this out.

[richscott]

This is now used, with the conversion of the other logic from sarama to franz-go, so it remains in.

[axw]

My question was more about the choice of string, "OIDCFILE". I'm not aware of a SASL mechanism with that name, and couldn't find anything in searches. I think if we go with my other suggestion, we could use OAUTHBEARER.

[atoulme]

sorry please fix the conflicts and mark ready for review again.