401 No user detected error when accessing Jupyter Notebook (with Keycloak)

[GH-00]

Validation Checklist

• Is this a Kubeflow issue?
• Are you posting in the right repository ?
• Did you follow the Kubeflow installation guideline ?
• Is the issue report properly structured and detailed with version numbers?
• Is this for Kubeflow development ?
• Would you like to work on this issue?
• You can join the CNCF Slack and access our meetings at the Kubeflow Community website. Our channel on the CNCF Slack is here #kubeflow-platform.

Version

1.9

Describe your issue

When I try to access the Jupyter Notebook using browser, I get this error.
{"log":"No user detected.","status":401,"success":false,"user":null}

I replaced dex with Keycloak as Idp and using oauth2-proxy.
API curl test was successful.

Here are my istio-ingressgateway and jupyter web app logs.

[2024-12-10T02:54:49.977Z] "GET /api/dashboard-links HTTP/1.1" 304 - via_upstream - "-" 0 0 50 49 "10.10.0.216" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36" "b05621c3-f6a4-413f-862a-913209df5e05" "211.175.140.50:31635" "10.233.88.23:4180" outbound|80||oauth2-proxy.oauth2-proxy.svc.cluster.local 10.233.88.11:33264 10.233.88.11:8080 10.10.0.216:6792 - -
[2024-12-10T02:54:49.974Z] "GET /api/activities/testuser HTTP/1.1" 304 - via_upstream - "-" 0 0 68 66 "10.10.0.216" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36" "adc97f19-3395-4c1e-be6e-d76843d332ff" "211.175.140.50:31635" "10.233.88.23:4180" outbound|80||oauth2-proxy.oauth2-proxy.svc.cluster.local 10.233.88.11:39080 10.233.88.11:8080 10.10.0.216:45815 - -
[2024-12-10T02:54:49.976Z] "GET /api/workgroup/exists HTTP/1.1" 304 - via_upstream - "-" 0 0 69 68 "10.10.0.216" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36" "0a92d9bc-2a2a-44ae-b838-133d03a7ede0" "211.175.140.50:31635" "10.233.88.14:4180" outbound|80||oauth2-proxy.oauth2-proxy.svc.cluster.local 10.233.88.11:43604 10.233.88.11:8080 10.10.0.216:64148 - -
[2024-12-10T02:54:50.042Z] "GET /assets/favicon.ico HTTP/1.1" 304 - via_upstream - "-" 0 0 5 5 "10.10.0.216" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36" "50de2262-bed4-4afa-9423-3597865c1215" "211.175.140.50:31635" "10.233.88.14:4180" outbound|80||oauth2-proxy.oauth2-proxy.svc.cluster.local 10.233.88.11:41014 10.233.88.11:8080 10.10.0.216:24129 - -
[2024-12-10T02:54:50.034Z] "GET /pipeline/apis/v1beta1/runs?page_size=5&sort_by=created_at%20desc&resource_reference_key.type=NAMESPACE&resource_reference_key.id=testuser HTTP/1.1" 304 - via_upstream - "-" 0 0 16 8 "10.10.0.216" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36" "dcb1d031-a7f7-4c2c-a8bb-0946abf1a8a0" "211.175.140.50:31635" "10.233.88.23:4180" outbound|80||oauth2-proxy.oauth2-proxy.svc.cluster.local 10.233.88.11:33264 10.233.88.11:8080 10.10.0.216:53698 - -
[2024-12-10T02:54:50.088Z] "GET /assets/favicon-32x32.png HTTP/1.1" 304 - via_upstream - "-" 0 0 9 9 "10.10.0.216" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36" "5c0eefd1-80b7-456f-acf4-fa43dcd4b58b" "211.175.140.50:31635" "10.233.88.23:4180" outbound|80||oauth2-proxy.oauth2-proxy.svc.cluster.local 10.233.88.11:33264 10.233.88.11:8080 10.10.0.216:24129 - -
[2024-12-10T02:54:50.083Z] "GET /api/workgroup/env-info HTTP/1.1" 304 - via_upstream - "-" 0 0 21 21 "10.10.0.216" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36" "7fd4e48c-804c-404c-95bd-6bfe87f38871" "211.175.140.50:31635" "10.233.88.14:4180" outbound|80||oauth2-proxy.oauth2-proxy.svc.cluster.local 10.233.88.11:41014 10.233.88.11:8080 10.10.0.216:53698 - -
[2024-12-10T02:54:50.148Z] "GET /api/workgroup/get-contributors/testuser HTTP/1.1" 304 - via_upstream - "-" 0 0 16 15 "10.10.0.216" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36" "c0e8d0ce-8187-48d8-9a4c-1e109e291dc0" "211.175.140.50:31635" "10.233.88.14:4180" outbound|80||oauth2-proxy.oauth2-proxy.svc.cluster.local 10.233.88.11:41014 10.233.88.11:8080 10.10.0.216:53698 - -

2024-12-10 02:54:02,018 | kubeflow.kubeflow.crud_backend.errors.handlers | ERROR | HTTP Exception handled: 401 Unauthorized: No user detected.
127.0.0.6 - - [10/Dec/2024:02:54:02 +0000] "GET /api/namespaces/testuser/notebooks HTTP/1.1" 401 69 "http://211.175.140.50:31635/_/jupyter/?ns=testuser" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36"
2024-12-10 02:54:49,983 | kubeflow.kubeflow.crud_backend.errors.handlers | ERROR | HTTP Exception handled: 401 Unauthorized: No user detected.
127.0.0.6 - - [10/Dec/2024:02:54:49 +0000] "GET / HTTP/1.1" 401 69 "http://211.175.140.50:31635/_/jupyter/?ns=testuser" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36"
2024-12-10 02:54:50,015 | kubeflow.kubeflow.crud_backend.errors.handlers | ERROR | HTTP Exception handled: 401 Unauthorized: No user detected.
127.0.0.6 - - [10/Dec/2024:02:54:50 +0000] "GET /api/namespaces/testuser/notebooks HTTP/1.1" 401 69 "http://211.175.140.50:31635/_/jupyter/?ns=testuser" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36"
2024-12-10 04:06:59,702 | kubeflow.kubeflow.crud_backend.errors.handlers | ERROR | HTTP Exception handled: 401 Unauthorized: No user detected.
127.0.0.6 - - [10/Dec/2024:04:06:59 +0000] "GET / HTTP/1.1" 401 69 "http://211.175.140.50:31635/_/jupyter/?ns=testuser" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36"
2024-12-10 04:06:59,720 | kubeflow.kubeflow.crud_backend.errors.handlers | ERROR | HTTP Exception handled: 401 Unauthorized: No user detected.
127.0.0.6 - - [10/Dec/2024:04:06:59 +0000] "GET /api/namespaces/testuser/notebooks HTTP/1.1" 401 69 "http://211.175.140.50:31635/_/jupyter/?ns=testuser" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36"

Steps to reproduce the issue

1. This is my configuration of oauth2-proxy

apiVersion: v1
kind: ConfigMap
metadata:
  name: oauth2-proxy
  namespace: oauth2-proxy
  labels:
    app: oauth2-proxy
data:
  oauth2_proxy.cfg: |
    provider = "keycloak-oidc"
    oidc_issuer_url = "http://211.175.140.50:31122/realms/OIDC"
    scope = "profile email groups openid"
    upstreams = ["http://centraldashboard.kubeflow.svc.cluster.local", "http://jupyter-web-app-service.kubeflow.svc.cluster.local/jupyter"]
    client_id = "Kubeflow" 
    client_secret = "{...}"
    email_domains = [ "*" ]

    skip_oidc_discovery = false
    login_url = "http://{...}/realms/{...}/protocol/openid-connect/auth"
    redeem_url = "http://{...}/realms/{...}/protocol/openid-connect/token"
    oidc_jwks_url = "http://{...}/realms/{...}/protocol/openid-connect/certs"
    extra_jwt_issuers = "http://{...}/realms/{...}"

    skip_provider_button = true
    skip_jwt_bearer_tokens = false

    api_routes = [
      # Generic
      # NOTE: included because most background requests contain these paths
      "/api/",
      "/apis/",
      "/jupyter",

      # Kubeflow Pipelines
      # NOTE: included because KFP UI makes MANY background requests to these paths but because they are
      #       not `application/json` requests, oauth2-proxy will redirect them to the login page
      "^/ml_metadata",
    ]

    set_authorization_header = true
    pass_access_token = true
    pass_authorization_header = true
    set_xauthrequest = true
    cookie_name = "oauth2_proxy"

    cookie_secret = "{...}"
    cookie_secure = false           
    cookie_httponly = true          
    cookie_samesite = "lax"         
    cookie_expire = "24h"
    cookie_refresh = "1h"

    # cookie_csrf = "XSRF-TOKEN"
    # csrf_cookie_name =  "XSRF-TOKEN"
    code_challenge_method = "S256"

    redirect_url = "http://{...}/oauth2/callback"
    relative_redirect_url = true
    show_debug_on_error = true


  FORCE_HTTPS: "false"
  ALLOW_SELF_SIGNED_ISSUER: "false"
  M2M_TOKEN_ISSUER: "http://{keycloak IP:port}/realms/{realm}"
  ENABLE_M2M_TOKENS: "false"
  EXTRA_JWT_ISSUERS: "http://{keycloak IP:port}/realms/{realm}"

binaryData: {}

2. Jupyter-web-app virtiualservice

apiVersion: networking.istio.io/v1
kind: VirtualService
metadata:
  labels:
    app: jupyter-web-app
    kustomize.component: jupyter-web-app
  name: jupyter-web-app-jupyter-web-app
  namespace: kubeflow
spec:
  gateways:
  - kubeflow-gateway
  hosts:
  - '*'
  http:
  - match:
    - uri:
        prefix: /jupyter/
    rewrite:
      uri: /
    route:
    - destination:
        host: jupyter-web-app-service.kubeflow.svc.cluster.local
        port:
          number: 80
  - match:
    - uri:
        prefix: /oauth2/
    route:
    - destination:
        host: oauth2-proxy.oauth2-proxy.svc.cluster.local
        port:
          number: 80
  - match:
    - uri:
        prefix: /
    route:
    - destination:
        host: oauth2-proxy.oauth2-proxy.svc.cluster.local
        port:
          number: 80

3. Keycloak JWT requestauthentication

apiVersion: security.istio.io/v1
kind: RequestAuthentication
metadata:
  name: keycloak-jwt
  namespace: istio-system
spec:
  jwtRules:
  - audiences:
    - Kubeflow
    forwardOriginalToken: true
    fromHeaders:
    - name: Authorization
      prefix: 'Bearer '
    issuer: http://{keycloak IP:port}/realms/{realm}
    jwksUri: http://{keycloak IP:port}/realms/{realm}/protocol/openid-connect/certs
    outputClaimToHeaders:
    - claim: email
      header: kubeflow-userid
    - claim: groups
      header: kubeflow-groups
    - claim: sub
      header: x-auth-request-user
  selector:
    matchLabels:
      app: jupyter-web-app

Put here any screenshots or videos (optional)

No response

[juliusvonkohout]

Hello as a first step please use 1.9.1 or the master branch since there are relevant authentication fixes in 1.9.1/master

[juliusvonkohout]

Afterwards please check out my new documentation about replacing dex 2ebc576

[GH-00]

Afterwards please check out my new documentation about replacing dex 2ebc576

I deployed Kubeflow v1.9.1 again and followed the guideline you suggested.
But after logging in, I couldn't access the centraldashboard.

I just added the requestauthentication.keycloak-jwt.yaml and changed oauth2-proxy-configmap. The rest of the parts, except for this, are exactly the same as what is uploaded on GitHub. Would you happen to have any idea to solve this issue?

[2024-12-12T07:49:26.443Z] "GET / HTTP/1.1" 403 UAEX ext_authz_error - "-" 0 0 30006 - "10.10.0.216" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36" "27925417-8bdd-4605-bdc1-b26f1cfd08c2" "211.175.140.50:31635" "-" outbound|80||centraldashboard.kubeflow.svc.cluster.local - 10.233.88.20:8080 10.10.0.216:17213 - -
[2024-12-12T07:53:15.470Z] "- - HTTP/1.1" 400 DPE http1.codec_error - "-" 0 11 0 - "-" "-" "-" "-" "-" - - 10.233.88.20:8080 10.10.0.216:1706 - -
[2024-12-12T07:53:15.504Z] "- - HTTP/1.1" 400 DPE http1.codec_error - "-" 0 11 0 - "-" "-" "-" "-" "-" - - 10.233.88.20:8080 10.10.0.216:1830 - -
[2024-12-12T07:53:16.424Z] "GET / HTTP/1.1" 302 UAEX ext_authz_denied - "-" 0 424 3 - "10.10.0.216" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36" "eea1bd18-9378-4ac1-a9e6-f41eb8d184cf" "211.175.140.50:31635" "-" outbound|80||centraldashboard.kubeflow.svc.cluster.local - 10.233.88.20:8080 10.10.0.216:11035 - -
[2024-12-12T07:53:21.707Z] "GET /oauth2/callback?state=gjyYjm-9X3pV8w78stItLXBZ64zNF5K17AWENMtPEbc%3A%2F&session_state=52e77423-b93d-42b5-9269-36b0fb25187f&iss=http%3A%2F%2F211.175.140.50%3A31122%2Frealms%2FOIDC&code=d10d3090-8828-4157-a6e5-0e5284073c40.52e77423-b93d-42b5-9269-36b0fb25187f.2120cf89-39bb-4a25-b957-aba616326305 HTTP/1.1" 302 - via_upstream - "-" 0 24 23 22 "10.10.0.216" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36" "6b52b424-f556-4a7c-b83f-1049dd406d17" "211.175.140.50:31635" "10.233.88.4:4180" outbound|80||oauth2-proxy.oauth2-proxy.svc.cluster.local 10.233.88.20:60486 10.233.88.20:8080 10.10.0.216:11035 - -
[2024-12-12T07:53:21.740Z] "GET / HTTP/1.1" 403 UAEX ext_authz_error - "-" 0 0 30010 - "10.10.0.216" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36" "5a2d1fae-ebed-4868-83d4-0da1920d5c06" "211.175.140.50:31635" "-" outbound|80||centraldashboard.kubeflow.svc.cluster.local - 10.233.88.20:8080 10.10.0.216:11035 - -
[2024-12-12T07:55:53.263Z] "- - HTTP/1.1" 400 DPE http1.codec_error - "-" 0 11 0 - "-" "-" "-" "-" "-" - - 10.233.88.20:8080 10.10.0.216:11319 - -
[2024-12-12T07:55:53.287Z] "- - HTTP/1.1" 400 DPE http1.codec_error - "-" 0 11 0 - "-" "-" "-" "-" "-" - - 10.233.88.20:8080 10.10.0.216:19101 - -
[2024-12-12T07:55:54.103Z] "GET / HTTP/1.1" 302 UAEX ext_authz_denied - "-" 0 424 3 - "10.10.0.216" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36" "95e040ca-f3da-4a74-b2a3-5f7ebede39d1" "211.175.140.50:31635" "-" outbound|80||centraldashboard.kubeflow.svc.cluster.local - 10.233.88.20:8080 10.10.0.216:11620 - -
[2024-12-12T07:55:58.692Z] "GET /oauth2/callback?state=1w4qU92yzfl4xUDNLlhoGEdWQL7tctVC8HEiHBakIKk%3A%2F&session_state=7f025f1e-3573-40e6-bff8-0a91829cc518&iss=http%3A%2F%2F211.175.140.50%3A31122%2Frealms%2FOIDC&code=781eac84-a31c-41ab-8d58-31307f0e396c.7f025f1e-3573-40e6-bff8-0a91829cc518.2120cf89-39bb-4a25-b957-aba616326305 HTTP/1.1" 302 - via_upstream - "-" 0 24 130 129 "10.10.0.216" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36" "8a8aef63-b0b2-485d-9331-d7825bc876af" "211.175.140.50:31635" "10.233.88.14:4180" outbound|80||oauth2-proxy.oauth2-proxy.svc.cluster.local 10.233.88.20:36004 10.233.88.20:8080 10.10.0.216:11620 - -

It seems like auhtentication is successful. But I can't access the centraldashboard.

10.233.88.20:60486 - 6b52b424-f556-4a7c-b83f-1049dd406d17 - testuser@example.com [2024/12/12 07:53:21] [AuthSuccess] Authenticated via OAuth2: Session{email:testuser@example.com user:0e296eb0-03ed-4ed4-bb54-1a501bbb0235 PreferredUsername:testuser token:true id_token:true created:2024-12-12 07:53:21.729925554 +0000 UTC m=+3889.164129848 expires:2024-12-12 17:53:21.729340006 +0000 UTC m=+39889.163545138 refresh_token:true groups:[kubeflow-admin offline_access uma_authorization default-roles-oidc]}
10.233.88.20:60486 - 6b52b424-f556-4a7c-b83f-1049dd406d17 - - [2024/12/12 07:53:21] 211.175.140.50:31635 GET - "/oauth2/callback?state=gjyYjm-9X3pV8w78stItLXBZ64zNF5K17AWENMtPEbc%3A%2F&session_state=52e77423-b93d-42b5-9269-36b0fb25187f&iss=http%3A%2F%2F211.175.140.50%3A31122%2Frealms%2FOIDC&code=d10d3090-8828-4157-a6e5-0e5284073c40.52e77423-b93d-42b5-9269-36b0fb25187f.2120cf89-39bb-4a25-b957-aba616326305" HTTP/1.1 "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36" 302 24 0.022

[GH-00]

I solved the issue. If anyone else is facing with the same issue, check this out below.

• edit your oauth2-proxy-configmap

skip_auth_routes = [
      "^/oauth2/callback",
    ]

• add a new request authentication configuration file using your keycloak information.

[juliusvonkohout]

I solved the issue. If anyone else is facing with the same issue, check this out below.

* edit your oauth2-proxy-configmap

skip_auth_routes = [
      "^/oauth2/callback",
    ]

* add a new request authentication configuration file using your keycloak information.

see #2934 (comment)

[Epochex]

I'm having the same problem, although I complete dex validation to get into the notebook, I've built kubeflow a dozen times before, but this time I've reinstalled it several times with the same problem, I always fail to connect to kfp and report an error when triggering a pipeline build as follows:
Reason: Unauthorized HTTP response headers: HTTPHeaderDict({'content-type': 'application/json', 'date': 'Sat, 28 Dec 2024 13:51:04 GMT', 'content-length': '811', 'x-envoy-upstream-service-time': '3', 'server': 'envoy'}) HTTP response body: {"error":"List experiments failed: Failed to authorize with API: Failed to authorize with API: Unauthenticated: Failed to check authorization. User identity is empty in the request header: Unauthenticated: Request header error: there is no user identity header.: Request header error: there is no user identity header.","code":16,"message":"List experiments failed: Failed to authorize with API: Failed to authorize with API: Unauthenticated: Failed to check authorization. User identity is empty in the request header: Unauthenticated: Request header error: there is no user identity header.: Request header error: there is no user identity header.","details":[{"@type":"type.googleapis.com/google.rpc.Status","code":16,"message":"Failed to check authorization. User identity is empty in the request header"}]}

[GH-00]

I'm having the same problem, although I complete dex validation to get into the notebook, I've built kubeflow a dozen times before, but this time I've reinstalled it several times with the same problem, I always fail to connect to kfp and report an error when triggering a pipeline build as follows: Reason: Unauthorized HTTP response headers: HTTPHeaderDict({'content-type': 'application/json', 'date': 'Sat, 28 Dec 2024 13:51:04 GMT', 'content-length': '811', 'x-envoy-upstream-service-time': '3', 'server': 'envoy'}) HTTP response body: {"error":"List experiments failed: Failed to authorize with API: Failed to authorize with API: Unauthenticated: Failed to check authorization. User identity is empty in the request header: Unauthenticated: Request header error: there is no user identity header.: Request header error: there is no user identity header.","code":16,"message":"List experiments failed: Failed to authorize with API: Failed to authorize with API: Unauthenticated: Failed to check authorization. User identity is empty in the request header: Unauthenticated: Request header error: there is no user identity header.: Request header error: there is no user identity header.","details":[{"@type":"type.googleapis.com/google.rpc.Status","code":16,"message":"Failed to check authorization. User identity is empty in the request header"}]}

I resolved the issue by configuring PodDefault to access ml-pipline in the user namespace.
Please check out this document for more details.
https://www.kubeflow.org/docs/components/pipelines/user-guides/core-functions/connect-api/#serviceaccount-token-volume

And if you still have the same problem, then you might want to check your Authorizationpolicy. It seems similar to this issue. kubeflow/pipelines#6120