`hub_routespec` set via values.yaml doesn't apply correctly — shows "Launch Server" page instead of 404

[uqqasha-pmx]

Bug description

Setting hub.config.JupyterHub.hub_routespec: "/hub/" via Helm values.yaml in a z2jh v4.2.0 deployment does not behave as expected. Instead of returning a 404 page for requests to stopped user servers, it redirects to the Hub and shows the "Launch Server" UI.

However, when the same setting is applied by manually editing the hub ConfigMap and restarting the Hub pod, the behavior is correct and a 404 is returned as expected.

How to reproduce

1. Create a values.yaml file containing:

hub:
  config:
    JupyterHub:
      hub_routespec: "/hub/"

2. Deploy JupyterHub using Helm.
3. Start a user server, then stop it.
4. Try to access /user/<username>/... in the browser.

Expected behaviour

A 404 page should appear from the proxy, since hub_routespec is set to /hub/ and the user's server is not running. See: https://jupyterhub.readthedocs.io/en/5.3.0/howto/api-only.html#disabling-hub-ui

Actual behaviour

JupyterHub redirects to /hub/user/<username>/... and shows the "Your server is not running. Would you like to start it?" message with a "Launch Server" button — the same behavior as if hub_routespec had never been customized.

Your personal set up

• OS: OpenShift 4.17
• Helm Chart Version: 4.2.0
• Spawner: KubeSpawner
• Authenticator: GenericOAuthenticator (with Keycloak)

We also tested applying c.JupyterHub.hub_routespec = "/hub/" via jupyterhub_config.py and via extraConfig, and neither worked. The only working method was manually editing the Hub's ConfigMap and restarting the pod.

Logs

[W 2025-06-02 18:19:02.321 JupyterHub app:1146] Using custom route for Hub: /hub/. Requests for not-running servers may not be handled.
....
[I 2025-06-02 18:19:02.752 JupyterHub app:3711] Not starting proxy
[D 2025-06-02 18:19:02.752 JupyterHub proxy:925] Proxy: Fetching GET http://proxy-api:8001/api/routes
[D 2025-06-02 18:19:02.773 JupyterHub proxy:996] Omitting non-jupyterhub route '/'
[I 2025-06-02 18:19:02.774 JupyterHub app:3747] Hub API listening on http://:8081/hub/
[I 2025-06-02 18:19:02.774 JupyterHub app:3749] Private Hub API connect url http://hub:8081/hub/
[I 2025-06-02 18:19:02.774 JupyterHub app:3623] Starting managed service jupyterhub-idle-culler
[I 2025-06-02 18:19:02.774 JupyterHub service:423] Starting service 'jupyterhub-idle-culler': ['python3', '-m', 'jupyterhub_idle_culler', '--url=http://localhost:8081/hub/api', '--timeout=120', '--cull-every=30', '--concurrency=10', '--cull-users', '--cull-admin-users=false', '--remove-named-servers']
[I 2025-06-02 18:19:02.776 JupyterHub service:136] Spawning python3 -m jupyterhub_idle_culler --url=http://localhost:8081/hub/api --timeout=120 --cull-every=30 --concurrency=10 --cull-users --cull-admin-users=false --remove-named-servers
[D 2025-06-02 18:19:02.777 JupyterHub spawner:1693] Polling subprocess every 30s
[D 2025-06-02 18:19:02.777 JupyterHub proxy:389] Fetching routes to check
[D 2025-06-02 18:19:02.777 JupyterHub proxy:925] Proxy: Fetching GET http://proxy-api:8001/api/routes
[D 2025-06-02 18:19:02.780 JupyterHub proxy:996] Omitting non-jupyterhub route '/'
[D 2025-06-02 18:19:02.780 JupyterHub proxy:392] Checking routes
[I 2025-06-02 18:19:02.780 JupyterHub proxy:477] Adding route for Hub: /hub/ => http://hub:8081
[D 2025-06-02 18:19:02.780 JupyterHub proxy:925] Proxy: Fetching POST http://proxy-api:8001/api/routes/hub
[I 2025-06-02 18:19:02.839 JupyterHub app:3780] JupyterHub is now running, internal Hub API at http://hub:8081/hub/
[D 2025-06-02 18:19:02.840 JupyterHub app:3347] It took 1.004 seconds for the Hub to start

Despite logs indicating correct route setup, the actual behavior remains incorrect unless applied manually.

[minrk]

When you manually edit the configmap, what changes do you make?

[uqqasha-pmx]

When you manually edit the configmap, what changes do you make?

I simply add this line in ConfigMap under data.jupyterhub_config.py:

c.JupyterHub.hub_routespec = "/hub/"

and then restart the hub pod.
Given the documentation here, it should work by setting it through values.yaml but it doesn't.

[consideRatio]

Have you ensured you only have one entry for hub.config.JupyterHub in your values file? If you have set it twice, one may be silently ignored

[uqqasha-pmx]

Have you ensured you only have one entry for hub.config.JupyterHub in your values file? If you have set it twice, one may be silently ignored

Yes, I have this in values file:

hub:
  config:
    JupyterHub:
      hub_routespec: "/hub/"

When I deploy JupyterHub with this setting, the hub logs show this:

Logs

[W 2025-06-02 18:19:02.321 JupyterHub app:1146] Using custom route for Hub: /hub/. Requests for not-running servers may not be handled.
....
[D 2025-06-03 09:58:07.122 JupyterHub application:929] Loaded config file: /usr/local/etc/jupyterhub/jupyterhub_config.py
[W 2025-06-03 09:58:07.168 JupyterHub app:1146] Using custom route for Hub: /hub/. Requests for not-running servers may not be handled.
[I 2025-06-03 09:58:07.168 JupyterHub app:3354] Running JupyterHub version 5.3.0
[I 2025-06-03 09:58:07.168 JupyterHub app:3384] Using Authenticator: oauthenticator.generic.GenericOAuthenticator-17.3.0
[I 2025-06-03 09:58:07.168 JupyterHub app:3384] Using Spawner: kubespawner.spawner.KubeSpawner-7.0.0
[I 2025-06-03 09:58:07.169 JupyterHub app:3384] Using Proxy: jupyterhub.proxy.ConfigurableHTTPProxy-5.3.0
......
[I 2025-06-03 09:58:07.651 JupyterHub app:3711] Not starting proxy
[D 2025-06-03 09:58:07.652 JupyterHub proxy:925] Proxy: Fetching GET http://proxy-api:8001/api/routes
[D 2025-06-03 09:58:07.683 JupyterHub proxy:996] Omitting non-jupyterhub route '/'
[I 2025-06-03 09:58:07.684 JupyterHub app:3747] Hub API listening on http://:8081/hub/
[I 2025-06-03 09:58:07.684 JupyterHub app:3749] Private Hub API connect url http://hub:8081/hub/
.....
[D 2025-06-03 09:58:07.687 JupyterHub proxy:389] Fetching routes to check
[D 2025-06-03 09:58:07.687 JupyterHub proxy:925] Proxy: Fetching GET http://proxy-api:8001/api/routes
[D 2025-06-03 09:58:07.692 JupyterHub proxy:996] Omitting non-jupyterhub route '/'
[D 2025-06-03 09:58:07.692 JupyterHub proxy:392] Checking routes
[I 2025-06-03 09:58:07.692 JupyterHub proxy:477] Adding route for Hub: /hub/ => http://hub:8081
[D 2025-06-03 09:58:07.692 JupyterHub proxy:925] Proxy: Fetching POST http://proxy-api:8001/api/routes/hub
[I 2025-06-03 09:58:07.786 JupyterHub app:3780] JupyterHub is now running, internal Hub API at http://hub:8081/hub/
[D 2025-06-03 09:58:07.787 JupyterHub app:3347] It took 1.684 seconds for the Hub to start

When the singleuser server is stopped, and I try to access the URL of stopped server from browser, it does not show 404 page. And the hub logs show this:

[I 2025-06-03 10:01:27.253 JupyterHub log:192] 302 GET /user/:username/lab/ -> /hub/user/:username/lab/ (@::ffff:127.0.0.6) 0.79ms
[D 2025-06-03 10:01:27.270 JupyterHub user:496] Creating <class 'kubespawner.spawner.KubeSpawner'> for :username:
[W 2025-06-03 10:01:27.293 JupyterHub log:192] 424 GET /hub/user/:username/lab/ (:username@::ffff:127.0.0.6) 27.01ms
[D 2025-06-03 10:01:28.774 JupyterHub log:192] 200 GET /hub/health (@127.0.0.6) 0.55ms
[I 2025-06-03 10:01:28.867 JupyterHub log:192] 302 GET /user/:username/lab/ -> /hub/user/:username/lab/ (@::ffff:127.0.0.6) 0.76ms
[W 2025-06-03 10:01:28.883 JupyterHub log:192] 424 GET /hub/user/:username/lab/ (:username@::ffff:127.0.0.6) 4.92ms
[D 2025-06-03 10:01:28.916 JupyterHub log:192] 304 GET /hub/static/components/@fortawesome/fontawesome-free/webfonts/fa-solid-900.woff2 (@::ffff:127.0.0.6) 0.56ms

[minrk]

Make sure to set proxy.chp.defaultTarget as well to the backend that you want to serve your custom 404. That governs the behavior of unmatched URLs, and is the hub by default.

[uqqasha-pmx]

Make sure to set proxy.chp.defaultTarget as well to the backend that you want to serve your custom 404. That governs the behavior of unmatched URLs, and is the hub by default.

I understand the defaultTarget is used for fallback behavior, but I'm not serving a custom 404 page. I want the default proxy 404 page to appear for stopped servers when hub_routespec is set to /hub/. This works only when I patch the c.JupyterHub.hub_routespec manually in the ConfigMap and restart the Hub pod.

When I set the same config in values.yaml or via extraConfig, I still see logs that suggest the route / is removed, but I don’t get the 404, instead I get the "Launch Server" page.

It seems that in the Helm-based path, the Hub may still register itself on / regardless of hub_routespec, which cancels out the 404 behavior.

[uqqasha-pmx]

I implemented the workaround mentioned in the #3686 (comment) above and it was working reliably for a while.

However, recently I observed that the / route was somehow re-registered in the proxy without any changes to deployment or Helm upgrade. As a result, requests to deleted user servers are routed to /hub/..., and JupyterHub redirected to the “Launch Server” page instead of showing the proxy 404.

c.JupyterHub.hub_routespec = "/hub/" doesn't appear to produce consistent behavior. Either the route is being restored unexpectedly, or something else in setup is causing the proxy to serve / again. Not sure if I am missing an additional configuration step.

[minrk]

I observed that the / route was somehow re-registered

Yes, to reiterate with more detail, proxy.chp.defaultTarget sets the endpoint used for /. It defaults to the Hub, which is why you are getting the Hub's behavior as if hub_routespec were /—because the proxy config is setting this, not the hub config. You must set proxy.chp.defaultTarget to something else in order for requests to /user/something to be routed somewhere other than the Hub.

CHP doesn't have to have a default route, but this chart doesn't allow that (it probably should). Until then, you'll need to set it to something. It could be your nginx default backend, for example, for a generic 404:

proxy:
  chp:
    defaultTarget: http://RELEASE-ingress-nginx-defaultbackend

[uqqasha-pmx]

Thank you for pointing out in the right direction, I missed the default target setting of CHP in helm chart. I tested with setting it explicitly to an nginx backend and now it works fine.

[minrk]

Glad it worked! I think there's two action items here:

1. to allow the default route in the proxy to be truly unset to get CHP's default behavior, rather than needing to point it to something
2. make it clear in the docs that the proxy default route is set to the hub and what that means for enabling "api only mode"