Document secure Kserve authentication via automated tests

[madmecodes]

Pull Request Template for Kubeflow Manifests

✏️ Summary of Changes

This PR addresses a security issue with KServe inference endpoints by applying oauth2-proxy authentication to the cluster-local-gateway. Currently, KServe endpoints lack proper authentication because they use the cluster-local-gateway which has an ALLOW policy instead of proper oauth2-proxy authentication.

Changes include:

1. Modified the cluster-local-gateway AuthorizationPolicy to use oauth2-proxy authentication instead of allowing all traffic without authentication
2. Updated the KServe test workflow to:
   • Use proper Kubeflow user namespaces instead of generic test namespaces
   • Test with tokens from the appropriate service accounts
   • Apply predictor-specific AuthorizationPolicy for tests

This ensures KServe inference endpoints are properly secured and require authentication, just like other Kubeflow services accessed through the istio-ingressgateway.

📦 Dependencies

No direct dependencies on other PRs. This PR combines improvements that were planned for PR #2936 with the security fix.

🐛 Related Issues

• Fixes Enable and document for Kubeflow 1.10 Kserve secure inferencing from inside and outside the cluster with tokens #2811 (Secure KServe inferencing with proper authentication)
• Related to KSERVE: update the kserve m2m test with real Kubeflow namespaces #2936 (Update KServe tests with real Kubeflow namespaces)
• Helps unblock ISTIO CNI by default #2907 (ISTIO CNI by default)

✅ Contributor Checklist

• I have tested these changes with kustomize. See Installation Prerequisites.
• All commits are signed-off to satisfy the DCO check.

---

You can join the CNCF Slack and access our meetings at the Kubeflow Community website. Our channel on the CNCF Slack is here #kubeflow-platform.

[madmecodes]

PS: okay i will look into the failed CI/CD runs, fixing them

[juliusvonkohout]

/retest

[madmecodes]

/retest

namespace creation is failing, i am debugging that, i will tag u once the issue is solved, sorry for the inconvenience :(

[juliusvonkohout]

#3062 is merged, so you can just rebase to master and have way less complexity

[madmecodes]

if KF-Profile is before KF-multi-tenancy
created a dependency issue.

common/kubeflow-namespace/base creates kubeflow namespace
common/user-namespace/base creates kubeflow-user-example-com namespace and profile controllers

If KF-profile is after Multi Tenancy

@juliusvonkohout Could you please review the installation order, and guide a bit, still facing the namespace issue.

[juliusvonkohout]

@akagami-harsh can you cross check whether this file is auto generated?

[akagami-harsh]

The gateway-authorizationpolicy.yaml file appears to be a manually created file that doesn't change when running the synchronize-istio-manifests.sh script.

[madmecodes]

working on Auth Policy, so that unwanted namespace get access denied

PS: found
In Istio, when a request enters through the ingress gateway (typically in the istio-system namespace), the source namespace evaluated by the AuthorizationPolicy is the namespace of the ingress gateway’s service account, not the namespace associated with the token in the Authorization header. Here’s why unauthorized access isn’t blocked

[madmecodes]

@juliusvonkohout Currently Auth Policy is allowing access to models, via any namespace, if it has a valid JWT we need to change it to so that only specified namespaces in AuthPolicy can access them, am i thinking in the right direction?

i am testing it locally, when its too strict, it forbids all if too permissive, allows all (with valid JWT), working on a middle path, am i working on right problem?

[juliusvonkohout]

@juliusvonkohout Currently Auth Policy is allowing access to models, via any namespace, if it has a valid JWT we need to change it to so that only specified namespaces in AuthPolicy can access them, am i thinking in the right direction?

i am testing it locally, when its too strict, it forbids all if too permissive, allows all (with valid JWT), working on a middle path, am i working on right problem?

Yes, that is what we want. Only mebers/serviceaccounts of the same namespace should be able to access it. Maybe you can work with the allowing the default-editor, default and default-viewer serviceaccount of the same namespace as principals in the authorizationpolicy.

[juliusvonkohout]

See also https://github.com/kubeflow/manifests/blob/master/apps/pipeline/upstream/base/installs/multi-user/istio-authorization-config.yaml

[juliusvonkohout]

Let me do some changes, merge the PR and then you can create a follow-up PR for the remaining stuff @madmecodes

• The dex test is just a bit flaky. Lets try to make it more robust.
• Do not hardcode "kubeflow-user-example-com" in the authorizationpolicy, but make it pick up the namespace where the authorizationpolicy is deployed in automatically.
• You need to make sure that you cannot inference with a token from another namespace e.g. kubeflow-user-example-com-attacker. For path-based and host-based access.

[madmecodes]

Let me do some changes, merge the PR and then you can create a follow-up PR for the remaining stuff @madmecodes

• The dex test is just a bit flaky. Lets try to make it more robust.
• Do not hardcode "kubeflow-user-example-com" in the authorizationpolicy, but make it pick up the namespace where the authorizationpolicy is deployed in automatically.
• You need to make sure that you cannot inference with a token from another namespace e.g. kubeflow-user-example-com-attacker. For path-based and host-based access.

yes, i am working on this only, facing some problems, troubleshooting them, i will create a follow up PR for these 3 things as mentioned once the PR is merged.

1. Make the Dex test more robust (fix flakiness)
2. Avoid hardcoding namespaces in AuthorizationPolicy
3. Ensure tokens from other namespaces can't access models

[juliusvonkohout]

/lgtm
/approve

[google-oss-prow]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: juliusvonkohout

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [juliusvonkohout]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[juliusvonkohout]

Please make sure that you rebase to master before you create the new branch / PR.

[juliusvonkohout]

In the follow up PR you can either remove or secure the cluster-local-gateway. Whatever works for kserve and keeps all security tests green.