feat: Add support for passing additional parameters to OIDC providers

[thomaspurchas]

Some OIDC providers support passing additional parameters in authorization requests to support features like multi-audience tokens, or tokens where the audience and client ID are different. An example of how providers implement this can be found here.

This change adds support for passing additional static parameters in the OIDC authorization request, so that Kargo can request tokens with different audiences, or multiple audiences. Being able to customise the audience in the returned authentication token, makes it possible for Kargo to request a token that's both recognised by Kargo and recognised as an authentication token for the underlying Kubernetes cluster. Which in turn opens up the possibility of using K8s authentication tokens with Kargo, and having Kargo operations directly authenticate with the user token, rather than having to manage it own RBAC and service account layer to provide access control.

[netlify]

✅ Deploy Preview for docs-kargo-io canceled.

Name	Link
🔨 Latest commit	a016196
🔍 Latest deploy log	https://app.netlify.com/projects/docs-kargo-io/deploys/687e85dc4bf72200088bbdb2

[krancour]

I see this must be related to #4197.

I want to tread lightly here, but I think this approach has a lot of potential.

Some things I'd be interested in seeing:

• Update the chart to make is easy to set OIDC_ADDITIONAL_PARAMETERS at install time.

• Update the CLI to also use this feature. I know I mentioned in feat: Allow users with OIDC JWTs to fall through to bearerToken auth #4197 that the CLI can already use the token from your kubeconfig, but since this is generic enough to use for other purposes, I wouldn't want the CLI excluded from this improvement.

• It would be great to see the multi-audience token demo'ed and working as you're hoping it will so we don't go down this road only to find some other blocker.

rather than having to manage it own RBAC and service account layer to provide access control.

I just want to set expectations that the mapping of OIDC users to ServiceAccounts won't go away. This would be an alternative. The reason we need to keep this feature is that in many (possibly even a majority of) orgs, many (or even most) Kargo users actually don't have any access to the cluster and the whole system of mapping users to SAs is really the only way to accommodate such users.

I'm excited to see where this could go and will close #4197 because this seems like a much better direction.

[krancour]

@thomaspurchas I'm confused by one thing... this PR was opened prior to #4197. What was #4197 meant to achieve that couldn't be done better through this approach? I feel like I must be missing something.

[thomaspurchas]

@krancour this PR makes it possible to get multi audience tokens. #4197 makes it possible to use multi audience tokens. So they’re “matched pair” of PRs. But I figured it would be better to introduce them as two separate PRs because they’re kinda doing slightly different things, and touch very different (but related) parts of the codebase.

I just want to set expectations that the mapping of OIDC users to ServiceAccounts won't go away. This would be an alternative.

Yeah I wasn’t expecting the existing SA approach to ever go away. Taking advantage of OIDC and RBAC like this requires a very specific cluster and IDP setup that probably isn’t viable for everyone.

Update the chart to make is easy to set OIDC_ADDITIONAL_PARAMETERS at install time.

I can add that to this PR.

Update the CLI to also use this feature. I know I mentioned in

I can certainly take a look, but I’m not sure what value it would bring. I wasn’t aware the CLI was capable of doing its own OIDC handshake, but if it is, it should be easy to extend this change to the CLI.

It would be great to see the multi-audience token demo'ed and working as you're hoping it will so we don't go down this road only to find some other blocker.

I’m not sure how best to do this. I can tell you that I have this working internally, and it’s been working well for little while now (indeed the only reason #4197 exists, is because I discovered the JWT limitation while trying to get all of this working). But I’m not sure how to create a public demo, as we use our own in-house IDP rather than any of big IDP services. Our IDP isn’t doing anything non-standard to support this, it’s all bog standard OIDC/oAuth, I’m just not familiar enough with any external IDP to set up something I could publicly demo.

[krancour]

I think we mutually agree #4197 just doesn't make sense without this PR, so especially with #4197 being a one-liner, let's just deal with everything here. 🙏

I can certainly take a look, but I’m not sure what value it would bring.

What I meant originally about not excluding the CLI from this feature was not excluding it from the generic ability to pass additional parameters during the authorization code flow.

Since then, I've realized a reason that the CLI can benefit specifically from multi-audience tokens...

Its current ability to use a k8s token to authenticate with the Kargo API server is predicated on the existence of kubeconfig, that kubeconfig's current context being the correct one, etc. On top of that, the token often doesn't appear directly in the kubeconfig because there is very often an auth plugin at play. The trickery involved in actually obtaining and using that token in such cases is complex.

I wouldn't have any of that go away really, because that "client config" option will remain useful in cases where OIDC just isn't set up at all, but outside of that case, the multi-audience tokens would simply be a much more straightforward way of doing the same thing without relying on kubeconfig or any kind of tomfoolery.

I’m not sure how best to do this. I can tell you that I have this working internally, and it’s been working well for little while now

I’m just not familiar enough with any external IDP to set up something I could publicly demo.

When the PR looks fully ready, I can stand up a throwaway cluster and IDP to help us validate it all.

[thomaspurchas]

@krancour PR updated with CLI, and Helm chart support!

Also #4197 merged in.

[krancour]

Thanks @thomaspurchas. This is looking pretty good. I will review more thoroughly in the coming week, but I think we're nicely aligned on the approach now.

[thomaspurchas]

@krancour do you know when you might have time to look at this?

[thomaspurchas]

@krancour is there anything I can to help you review this PR? If you can give me some pointers on OIDC providers that support multi-audience setups, I might be able to put together a test setup for you.

[krancour]

@thomaspurchas sorry for the delay. I've finally gotten around to reviewing this carefully and the code looks just fine in theory, but after spending hours experimenting and scouring docs, I couldn't make this work end-to-end with any publicly available IDP. This unfortunately leaves me with doubts about the practicality of what you're trying to enable with this change.

we use our own in-house IDP rather than any of big IDP services. Our IDP isn’t doing anything non-standard to support this...

The spec for sure allows the aud claim to be an array, so I can readily accept your claim (pun intended) that your homegrown IDP is doing nothing non-standard, but even with that being the case, it would seem you have access to a capability that nearly no one else does.

The unfortunate thing here, I think, is that probably no one but you benefits from this change, and having no access to an IDP with this unique capability, it's not possible to thoroughly vet that change.

I'm sorry to say, but I'm reluctant to move forward with this unless you can show this working end-to-end with some publicly available IDP.

[krancour]

Tentatively closing this.

[thomaspurchas]

@krancour could you give me a list of public IDPs that you've tried? I'm not super familiar with the IDP landscape, so a list providers you know about would give me a good starting point to see if I can make this working using a public IDP.

Also if we're closing this, can we reopen #4197? You don't need multi-audience support for the JWT issued to Kargo to be accepted by the k8s RBAC layer. k8s can be configured to accept JWTs from multiple issuers at the same time. So there is nothing preventing someone from configuring their k8s cluster to accept the JWTs already being issued to Kargo.

I think it's also worth noting that there may be other reasons for wanting to pass additional parameters in the auth request to an OIDC provider, beyond just multi-audience tokens. For example, it's often used to provide hints to the IDP's login flow. Auth0 have some docs on how a number of IDP systems accept additional parameters.

My use-case specific uses multi-audience tokens, so that's what I've tested. But these two bits of functionality individually add important functionality that is broadly usable by anyone. It just that in my specific case, I'm using these two components together in a manner that is less common than expected.

[krancour]

I experimented extensively with AKS + Entra ID and KinD with a standalone Dex. After no success with those, I resorted to reading docs and consulting Copilot for other IDPs. There was little point struggling with any IDP unless their docs explicitly noted support for this.

I'll concede that Ory claims to support this and with enough fiddling, I possibly could have gotten k8s to work with Ory, but it's neither easy nor common to use k8s distros from the public cloud providers with any IDP other than their own.

I'm not at all saying this is impossible to do. I'm saying it's very hard to validate these changes and the hours that I've already sunk into this exceed the perceived benefit.

[thomaspurchas]

I'm saying it's very hard to validate these changes and the hours that I've already sunk into this exceed the perceived benefit.

Can we find another way to validate the parameter passing that doesn't rely on multi-audience support? And separately find a way to validate the change in the RBAC code, that also doesn't require multi-audience support?

It'll be two seperate, completely unrelated, test scenarios. Each validating one specific part of this change, and also proving the independent value of the two changes?

For example, Dex allows clients to pass an additional connector_id parameter in the auth URL, which will cause Dex to automatically redirect the client to the specified connector, rather than displaying a list of connectors for the user to choose from. For some reason this functionality isn't documented, but it does exist.

Microsoft Entra supports prompt, login_hint and domain_hint, as additional parameters that can be passed to the authorize URL.

Support for additional parameters makes it possible for people to configure Kargo to pass parameters like connection_id/prompt/domain_hint etc to IDPs and create a better user experience by having parts of the login flow either skipped completely, or prefilled. Which I think is enough value alone to support additional auth parameters.

For the RBAC fallthrough, that can be achieve with a Kind cluster and Kargo using Dex, with the Kind cluster configured to accept JWTs issued to either audience. I can try and put together a Kind cluster config that sets this up, it might take me a little while.

[krancour]

Can we find another way to validate the parameter passing that doesn't rely on multi-audience support? And separately find a way to validate the change in the RBAC code, that also doesn't require multi-audience support?

I don't think it will come to that...

I can try and put together a Kind cluster config that sets this up, it might take me a little while.

I actually started working on that lastnight, but I'm using Ory since it purports to support the exact scenario you care about. I'll re-open this, but wrapping this up is low priority.

with the Kind cluster configured to accept JWTs issued to either audience

Isn't that actually a tad different than the scenario you were initially concerned with?

Making the k8s API server accept tokens for two different audiences (which I'm not positive is possible, but let's find out) and making the IDP issue multi-audience tokens seem like distinctly different approaches to the same problem. If the former is even possible with KinD, I'm highly skeptical that most other k8s distros would permit that.

[thomaspurchas]

Big block of text below, so want to prefix this with an offer to have call to talk through this stuff, might be easier than going back and forth via Github. If that's of interest, just say so, and I can reach out via Discord.

Isn't that actually a tad different than the scenario you were initially concerned with?

It is, but the changes needed to support it are very similar. #4197 is all that's needed to support this case.

Making the k8s API server accept tokens for two different audiences (which I'm not positive is possible, but let's find out)

It's explicitly supported in k8s 1.3 and above via the auth configuration file. The k8s team added the auth config file to support this exact use case, the example config in the k8s docs show how to accept multiple audencies from the same issuer.

The functionality is currently not exposed via hosted services like EKS or GKE, but I from talking to EKS support staff (and bugs that broken our clusters) that EKS now uses the authentication config files under-the-hood, so EKS has the technical capability to support multiple OIDC issuers (and the UI even hints at the support), it's just disabled in the UI at the moment.

Isn't that actually a tad different than the scenario you were initially concerned with?

It is different. What I'm trying to highlight is that the two changes here:

1. Support for passing additional parameter to OIDC providers
2. feat: Allow users with OIDC JWTs to fall through to bearerToken auth #4197 - Allowing auth to fallthrough to k8s RBAC if the OIDC token isn't recognised by Kargo

both have very important uses outside of just the multi-audience ID token scenario. Multi-audience tokens aren't the only scenario where you need these two features.

For Support for passing additional parameter to OIDC providers you also need this functionality for using IdP specific features, exposed via additional query parameters, like:

• Specifying a connector_id to Dex, so the Dex can automatically redirect the user to the correct IdP system (out of multiple connected) that Dex is federating.
• Specifying the domain_hint for Microsoft Entra, so it can automatically redirect the user to whatever backend SSO system has been configured for their company, without having to enter an email address first, just to trigger the redirect.
• Specifying the prompt for Microsoft Entra, so it's possible to force the user to go through the login flow, even if they already have a valid Entra session.

For #4197 - Allowing auth to fallthrough to k8s RBAC if the OIDC token isn't recognised by Kargo you need this functionality to allow people to use the K8s RBAC system directly, if they have an auth setup that allows them to issue OIDC tokens that are recognised by both k8s and Kargo. There's three different ways to make that happen:

• Simply configure Kargo and K8s to accept and request the same audience token. Not best practice, but totally workable.
• Configure your IdP to issue multi-audience tokens so that Kargo can request a ID token that includes the audience used by the K8s cluster.
• Configure the k8s cluster to accept the audience use in tokens issued to Kargo.

It's only the multi-audience token scenario that requires both of these changes to happen at the same time. I could go a create two new PRs for these changes, and provide you with two brand new, perfectly valid and useful, reasons for making each change. I think there's sufficient justification for these changes, even if you can't get the multi-audience config working, and I think it's worth evaluating these changes based on those other uses, because they're much easier to test and validate. Multi-audience support then becomes a happy accident, as a consequence of adding support for features that are independently valuable.

[krancour]

I understand everything you said and appreciate you cluing me in on a few more obscure config options that I wasn't yet aware of. But I'm asking you to please understand this is not a priority for our team. Jumping on a call is the last thing on my radar right now. It doesn't matter whether you view this as one change or two individually useful changes that happen to also work great together. For something we don't see as a priority, I've sunk a lot of time already into validating this, and I think it should be clear from me re-opening the PR that I am graciously willing to put more time into it still, but I'm asking for you to understand that it's happening on a best effort basis as time permits.

[thomaspurchas]

For something we don't see as a priority, I've sunk a lot of time already into validating this, and I think it should be clear from me re-opening the PR that I am graciously willing to put more time into it still, but I'm asking for you to understand that it's happening on a best effort basis as time permits.

I appreciate the time your putting into this, and I'm not trying to pressure you into going faster and apologise if it's coming across like that. I just trying to highlight some options for validating this change that might be quicker and easier to do, rather than trying to get the multi-audience tokens working end-to-end.

Ultimately it would be a shame if these changes were rejected simply because use-case I presented with them is too obscure, when there are plenty of non-obscure use-cases that could be used instead.

I have very little visibility into what your team considers a priority, so I can only guess at what might provide reasonable justification for prioritising, or not prioritising a specific feature. But regardless of that, we'll continue to run a fork of Kargo internally anyway, even once this is merged, because our internal processes make contributing to OSS projects difficult (it took me a month to get approval to contribute these changes). I just want to make sure I've given you everything you need to review this change, on whatever timeline works best for you.