Skip to content

Add VirtualService conflict troubleshooting for KServe path-based Routing #3247

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Sep 30, 2025
+201 −174
Merged

Add VirtualService conflict troubleshooting for KServe path-based Routing #3247

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
54fcc2a
docs: Add VirtualService conflict troubleshooting for KServe path-bas…
madmecodes Sep 22, 2025
0c40f4e
update: merged kserve tests into one file
madmecodes Sep 30, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
88 changes: 0 additions & 88 deletions .github/workflows/kserve_jwt_authentication_test.yaml
View file
Open in desktop

This file was deleted.

83 changes: 0 additions & 83 deletions .github/workflows/kserve_secure_test.yaml
View file
Open in desktop

This file was deleted.

145 changes: 143 additions & 2 deletions .github/workflows/kserve_test.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,25 +3,29 @@ on:
pull_request:
paths:
- tests/install_KinD_create_KinD_cluster_install_kustomize.sh
- .github/workflows/kserve_m2m_test.yaml
- .github/workflows/kserve_test.yaml
- applications/kserve/**
- apps/kserve/**
- tests/kserve/**
- tests/kserve_test.sh
- tests/kserve_install.sh
- tests/kserve_jwt_authentication_test.sh
- common/istio*/**
- common/oauth2-proxy/**
- tests/oauth2-proxy_install.sh
- common/cert-manager/**
- tests/istio*
- common/knative/**
- tests/knative_install.sh
- tests/*authentication*test.sh
- tests/final_validation.sh

permissions:
contents: read
actions: read

jobs:
build:
test-basic-kserve:
runs-on: ubuntu-latest
steps:
- name: Checkout
Expand Down Expand Up @@ -93,3 +97,140 @@ jobs:

- name: Apply Pod Security Standards restricted levels
run: ./tests/PSS_enable.sh

test-jwt-authentication:
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v4

- name: Install KinD, Create KinD cluster and Install kustomize
run: ./tests/install_KinD_create_KinD_cluster_install_kustomize.sh

- name: Install kubectl
run: ./tests/kubectl_install.sh

- name: Create kubeflow namespace
run: kustomize build common/kubeflow-namespace/base | kubectl apply -f -

- name: Install Istio CNI
run: ./tests/istio-cni_install.sh

- name: Install oauth2-proxy
run: ./tests/oauth2-proxy_install.sh

- name: Install knative CNI with secure cluster-local-gateway
run: ./tests/knative_install.sh

- name: Verify secure cluster-local-gateway configuration
run: |
kubectl get authorizationpolicy,requestauthentication -n istio-system | grep cluster-local-gateway
kubectl get requestauthentication cluster-local-gateway-jwt -n istio-system -o yaml
kubectl get authorizationpolicy cluster-local-gateway -n istio-system -o yaml
kubectl get authorizationpolicy cluster-local-gateway-require-jwt -n istio-system -o yaml

- name: Setup python 3.12
uses: actions/setup-python@v4
with:
python-version: 3.12

- name: Port forward
run: ./tests/port_forward_gateway.sh

- name: Wait for cluster-local-gateway to be ready
run: |
kubectl wait --for=condition=Available --timeout=120s deployment/cluster-local-gateway -n istio-system
sleep 100

- name: Run Basic JWT Authentication Tests
run: |
export KSERVE_INGRESS_HOST_PORT=localhost:8080
curl -s -o /dev/null -w "%{http_code}" -H "Host: test.example.com" "http://localhost:8080/" | grep -q "403"

- name: Run Knative Service JWT Authentication Tests
run: |
export KSERVE_INGRESS_HOST_PORT=localhost:8080
./tests/knative_authentication_test.sh

- name: Test External Access Configuration
run: |
export KSERVE_INGRESS_HOST_PORT=localhost:8080
./tests/kserve_setup_external_access.sh kubeflow-user-example-com secure-model-predictor
# Test external access pattern
TOKEN=$(kubectl -n kubeflow-user-example-com create token default-editor)
RESPONSE=$(curl -s -o /dev/null -w "%{http_code}" \
-H "Authorization: Bearer $TOKEN" \
-H "Content-Type: application/json" \
"http://localhost:8080/kserve/kubeflow-user-example-com/secure-model-predictor/" \
2>/dev/null || echo "404")
if [ "$RESPONSE" != "404" ] && [ "$RESPONSE" != "200" ] && [ "$RESPONSE" != "503" ]; then
exit 1
fi

- name: Apply Pod Security Standards restricted levels
run: ./tests/PSS_enable.sh

test-secure-authentication:
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v4

- name: Install KinD, Create KinD cluster and Install kustomize
run: ./tests/install_KinD_create_KinD_cluster_install_kustomize.sh

- name: Install kubectl
run: ./tests/kubectl_install.sh

- name: Create kubeflow namespace
run: kustomize build common/kubeflow-namespace/base | kubectl apply -f -

- name: Install Istio CNI
run: ./tests/istio-cni_install.sh

- name: Install oauth2-proxy
run: ./tests/oauth2-proxy_install.sh

- name: Install cert-manager
run: ./tests/cert_manager_install.sh

- name: Install knative CNI (with secure cluster-local-gateway)
run: ./tests/knative_install.sh

- name: Install KServe
run: ./tests/kserve_install.sh

- name: Install KF Multi Tenancy
run: ./tests/multi_tenancy_install.sh

- name: Install kubeflow-istio-resources
run: kustomize build common/istio/kubeflow-istio-resources/base | kubectl apply -f -

- name: Create KF Profile
run: ./tests/kubeflow_profile_install.sh

- name: Setup python 3.12
uses: actions/setup-python@v4
with:
python-version: 3.12

- name: Port forward
run: ./tests/port_forward_gateway.sh

- name: Verify JWT authentication policies are applied
run: |
kubectl get authorizationpolicy cluster-local-gateway-require-jwt -n istio-system
kubectl get requestauthentication cluster-local-gateway-jwt -n istio-system
kubectl get authorizationpolicy cluster-local-gateway -n istio-system
kubectl get deployment cluster-local-gateway -n istio-system
kubectl wait --for=condition=Available deployment/cluster-local-gateway -n istio-system --timeout=120s
kubectl get pods -n istio-system -l app=cluster-jwks-proxy | grep -q Running || kubectl get pods -n istio-system -l app=cluster-jwks-proxy

- name: Wait for configurations to propagate
run: sleep 60

- name: Run KServe secure authentication tests
run: ./tests/kserve_jwt_authentication_test.sh kubeflow-user-example-com

- name: Apply Pod Security Standards restricted levels
run: ./tests/PSS_enable.sh
16 changes: 15 additions & 1 deletion applications/kserve/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,4 +2,18 @@

For KServe installation and usage, see the [GitHub Actions tests](.github/workflows/kserve_test.yaml) which demonstrate working configurations.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you list all the tests and explain what they are for? I mean we have already a tests for kserve path based access.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

or better jus tmerge
kserve_jwt_authentication_test.yaml
kserve_secure_test.yaml
kserve_test.yaml

Into a single file.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You have to find out which ordering makes sense.


For complete documentation, visit the [official KServe website](https://kserve.github.io/website/).
For complete documentation, visit the [official KServe website](https://kserve.github.io/website/).

## Integration with KubeFlow

When using KServe with path-based routing in a KubeFlow deployment, you may encounter VirtualService conflicts that result in 404 errors when accessing KServe InferenceServices.

**Common Issues:**
- KServe InferenceServices return 404 errors when accessed via their configured domain
- Conflicts between KubeFlow's wildcard VirtualServices and KServe's specific-host VirtualServices

**Solution:** See the [Istio troubleshooting guide](../../common/istio/README.md#virtualservice-conflicts-with-kserve-path-based-routing) for detailed resolution steps.

**Related Documentation:**
- [KServe Path-Based Routing Configuration](https://kserve.github.io/website/docs/admin-guide/configurations#path-template)
- [Upstream Istio Issue](https://github.com/istio/istio/issues/57404)
Loading
Loading