Support for Jupyter hub

[rdalbuquerque]

Hello folks!

While developing with Jupyterhub and Dask, a documented and proposed way by Microsoft to run and monitor jobs is using kbatch, which is poorly maintained at the moment.

Is there any plans to integrate with jupyterhub?

Thanks in advance!

[irvinlim]

Hey there! I took a look at the main use case of kbatch, which seems to have been explained by these pages:

• https://words.yuvi.in/post/kbatch/
• https://planetarycomputer.microsoft.com/docs/overview/batch/

From what I can see, JupyterHub doesn't provide asynchronous job processing, and so kbatch was conceived to fill the gap so that users have a dead-simple CLI tool to submit the job from their notebook to a remote cluster (in this case, it's expected that JupyterHub users already are running on K8s, so it would be the same cluster). The kbatch tool basically uses the native batch/v1 Jobs and CronJobs under the hood.

On the other hand, Furiko aims to replace the batch/v1 Job/CronJob objects in Kubernetes by extending its functionality with a slightly more powerful feature set (see this page for more details). I guess you could treat it as a lower-level project (currently we don't have the bandwidth to expand its scope much further, unfortunately).

While there may be merit in Furiko to provide first-class support for JupyterHub, I believe it makes more sense for this to be developed as an integration (similar to how kbatch is an integration to Kubernetes itself). I'm not familiar with JupyterHub so unfortunately I don't think I can fully appreciate and understand what problems it can try and solve. Perhaps if you were so inclined to help contribute one, I would be more than happy to discuss the technical aspects with you!