Prioritizing Certain Node Pools when using Cluster Autoscaler

[knuurr]

I have multiple autoscaler node pools on Hetzner, differing in server_type and location.
I’d like to prioritize certain node pools over others when scaling out.

Goal:

• Avoid creating new nodes in hel1, unless absolutely necessary.
• If Hetzner cannot provide resources in a preferred location (nbg1, fsn1), autoscaler should try another preferred location before using hel1.
• Prefer a solution at the autoscaler level (without modifying pod deployment templates).

Questions:

Can Cluster Autoscaler be configured to prefer some node pools over others? If so, how?
If not directly possible, what are the best workarounds? (e.g., taints/tolerations, priority classes, or other Kubernetes mechanisms?)
Is there a way to ensure a fallback strategy where hel1 is only used if no other locations have capacity?

Current setup:

Structure looks like this:

autoscaler_nodepools = [
    {
      name        = "worker-pool-ccx33-nbg1"
      server_type = "ccx33"
      location    = "nbg1"
      min_nodes   = 0
      max_nodes   = 3
      labels = {
        "node-type"                    = "worker"
        "node.kubernetes.io/role"      = "worker"
        "topology.kubernetes.io/region" = "nbg1"
        "node.hetzner.com/server-type" = "ccx33"
        "worker-pool"                  = "worker-small"
      }
    },
    {
      name        = "worker-pool-ccx33-fsn1"
      server_type = "ccx33"
      location    = "fsn1"
      min_nodes   = 0
      max_nodes   = 3
      labels = {
        "node-type"                    = "worker"
        "node.kubernetes.io/role"      = "worker"
        "topology.kubernetes.io/region" = "fsn1"
        "node.hetzner.com/server-type" = "ccx33"
        "worker-pool"                  = "worker-small"
      }
    },
    {
      name        = "worker-pool-ccx33-hel1"
      server_type = "ccx33"
      location    = "hel1"
      min_nodes   = 0
      max_nodes   = 3
      labels = {
        "node-type"                    = "worker"
        "node.kubernetes.io/role"      = "worker"
        "topology.kubernetes.io/region" = "hel1"
        "node.hetzner.com/server-type" = "ccx33"
        "worker-pool"                  = "worker-small"
      }
    },
    # ....
    ]

(we have some more but this is sufficient for context)

I have read docs for cluster autoscaler, but I am unsure if it's possible to apply it in this scenario

If something is unclear, don't hesitate to ask me here. Thanks in advance!

[dddsnn]

I had a similar problem and solved it using the autoscaler's priority expander. E.g., set

  cluster_autoscaler_extra_args = [
    "--expander=priority,least-waste"
  ]

and create a configmap like

apiVersion: v1
kind: ConfigMap
metadata:
  name: cluster-autoscaler-priority-expander
  namespace: kube-system
data:
  priorities: |-
    50:
      - .*nbg1
      - .*fsn1
    30:
      - .*hel1

So far, this seems to work for me.

The documentation says the configmap has to have exactly this name, and has to be created before the autoscaler can start. If you create it as a kubectl_manifest resource, I assume this even works relatively seamlessly when bootstrapping a new cluster, but I haven't tested that.

[knuurr]

Hi thanks for reply I still need to test it out. This looks really interesting. By any chance this doesn't mean in practice that those hel1 nodes will get replaced with higher priority ones, once they are available?

[dddsnn]

Not automatically I think. I'm pretty sure the only way the autoscaler has to see if nodes in a region are available is to order one and see if it succeeds. And I don't think you can get it to just do that exploratively, to try and increase the priority metric. The docs say it only adds nodes if pods fail to schedule, and if they're happily running in hel1 that's not the case.

But you can achieve this with a bit of manual intervention, by cordoning and draining a hel1 node. That node should be removed by the autoscaler (since it's underutilized). And if the workloads drained off of it don't fit in the rest of the cluster, it will order a new node (nbg1 or fsn1 if available).