Dask workers not starting

[sidravi1]

Hi - thanks for all your work on MPC. I'm really enjoying working on it.

I'm trying to setup a dask cluster as follows:

import dask_gateway

cluster = dask_gateway.GatewayCluster()
client = cluster.get_client()
cluster.scale(4)
print(cluster.dashboard_link)

but workers don't seem to start. I have waited for hours. Am i doing something wrong?

dashboard:

[glaroc]

Same problem here

[egagli]

I've also had the same problem the past 3 or 4 days

[TomAugspurger]

I'll take a closer look when I get back to a computer, but most likely you aren't doing anything wrong.

The Planetary Computer Hub is just a JupyterHub deployment on AKS (https://github.com/Microsoft/planetary-computer-hub). All the users get worker nodes from a shared pool of nodes. If that node pool reaches its maximum size, you'll see this behavior: i.e. zero feedback about what's going on, other than no workers showing up. The lack of feedback is really unfortunate, but I don't see an (easy) way to expose this information. By users of the Hub don't have access to the Kubernetes API, which is where this information resides

There are a few options:

1. Continue to wait, which doesn't feel like a great solution, especially since you don't have any insight into how long you'll wait. I also don't know what kind of fairness promises the Kubernetes scheduler gives to waiting Pods.
2. We implement some additional restrictions on jobs that can be run with Dask on the Hub (maximum lifetimes? Reduce the maximum number of cores / memory? etc.)
3. You deploy compute in your own Azure subscription, either your own Hub (if you need multi-user support) or one of the many options listed at https://planetarycomputer.microsoft.com/docs/concepts/computing/. Remember: the Hub isn't special with how it does things. As long as you put the compute next to the data (West Europe for most datasets) you'll get fast access to the data.

[glaroc]

Is there a way to show the current number of used nodes on the status page (https://planetarycomputer-status.microsoft.com/)? By the way, this page seems to be broken for me.

[sidravi1]

Thanks for your response.

1. Is it possible to report on load? Possibly on the status page? Btw, the status page often doesn't load for me as well.
2. Are unused clusters automatically terminated? Also, any way I can see clusters that I requested that might be still running?
3. We'll probably go down this path but I'm hoping to not setup my own infrastructure just yet.

Is there any chance additional nodes can be added to the pool?

Thanks again for all you do. I don't want to look a gift horse in the mouth. Just want to figure out a timeline when I will be able to get my workflow operational again.

[TomAugspurger]

Thanks for the report about the status page. We'll take a look at that as well.

I'm not sure about the feasibility of getting the Dask worker pool information into there.

Are unused clusters automatically terminated?

Yes, after (IIRC) 10-15 minutes of inactivity

Also, any way I can see clusters that I requested that might be still running?

Yes: https://gateway.dask.org/api-client.html?highlight=list_clusters#dask_gateway.Gateway.list_clusters

when I will be able to get my workflow operational again.

I would strongly recommend putting any kind of "production" workflow in your own infrastructure that you control. If only for things like control over the software environment.

Is there any chance additional nodes can be added to the pool?

Not sure on this ATM.