Restart kernel takes exactly 60s

[linlol]

Description

I am using jupyter on some scenarios with quite bad performance powered by z2jh as below:

1. My frontend and backend are running at different region

In some cases, I can notice below behaviour during restarting

1. After/restart API finished, kernel takes exactly 60s to initialise (in devtool, I can see that client side send a few binary msg, and takes 1min to get response)

2. In backend log, I can see there is some timeout warning, and eventually work with exactly 60s (more precisely, a few milli secs longer, like 60050ms)

Given that it takes exactly 60s, I doubt it is some race condition that makes kernel manager reaches it timeout

However in this case it shall raise error instead of eventually work?

Reproduce

1. Go to '...'
2. Click on '...'
3. Scroll down to '...'
4. See error '...'

Expected behavior

Context

• Operating System and version: frontend is win 10, backend image is based on rhel 8 with host Ubuntu 21/22 on Azure kubernetes service
• Browser and version:
• Jupyter Server version: 2.14.1

Troubleshoot Output

Paste the output from running `jupyter troubleshoot` from the command line here.
You may want to sanitize the paths in the output.

Command Line Output

Paste the output from your command line running `jupyter lab` here, use `--debug` if possible.

Browser Output

Paste the output from your browser Javascript console here, if applicable.

[linlol]

I am still thinking about if it is

1. any of my extension code caused this race condition

Or

2. Any corner case that only happens in great network slowness (on my side due to the physical distance and authentication workflow, every request has about 180ms latency)

Pls feel free to leave any comments/guess/advise here, your intuition matters

[linlol]

more specifically, seems like when initialising the prepare function in channels.py

It somehow hits the give_up function, Which seems to mean it hit the timeout when calling request_kernel_info

Would take a further look

[linlol]

is this a known issue?

Am using jupyter_client 8.6.0 and ipykernel 6.29.4

[krassowski]

Can you provide more details/paste the actual logs?

[krassowski]

Also, does it happen with other kernels too?

[linlol]

hey @krassowski , thanks for following,

It simply raised the warning log in give_up function inside channels.py without any further unexpected.

After 60s timeout the kernel works as usual and sorry that I just tried ipykernel.

(And this issue could be reproduced on my side that I can relroduce it on different notebooks and different envs. But not sure if it is due to latency)

[linlol]

Would take further look at

1. Can I reproduce it locally (without jupyterhub) with same jupyter version (every version in jupyter --version I mean)

2. Can I reproduce the issue on different kernel (such as r kernel or xeus)

3. Can I get rid of this with some upgraded/downgrade jupyter_server/jupyter_client

[linlol]

hello @krassowski , take a further look that, seems like when I click reboot and run-all button when kernel is in "connecting" status, I would highly likely to reproduce this issue

Shall be some concurrency issue in jupyter for high latency which is not easy to reproduce locally (our jupyter server is running in Europe while jupyterlab frontend is running in Sg, so more likely to see something tricky)