Upgrading clusters problem

[clickersmudge]

Hi,
As described here:
https://docs.kubermatic.com/kubeone/v1.5/tutorials/upgrading-clusters/#upgrading-the-cluster

We were upgrading the cluster from version "1.24.7" to "1.24.8"

nano kubeone.yaml
  versions.kubernetes: "1.24.8"
kubeone apply --manifest kubeone.yaml --credentials credentials.yaml --tfjson tf.json --upgrade-machine-deployments

We got a lot of errors in the terminal during this process.

[xmudrii]

@clickersmudge Looks like there's an issue with evicting the CoreDNS pods. Can you share the output of kubectl get nodes -o wide and kubectl get pods -A?

[clickersmudge]

@xmudrii New cluster has been set up with version "1.24.1".

kubeone apply --credentials credentials.yaml --manifest kubeone.yaml --tfjson tf.json
INFO[21:03:04 CET] Determine hostname...                        
INFO[21:03:05 CET] Determine operating system...                
INFO[21:03:07 CET] Running host probes...                       
The following actions will be taken: 
Run with --verbose flag for more information.
	+ initialize control plane node "kubeone-control-plane-1" (192.168.0.3) using 1.24.1
	+ ensure machinedeployment "kubeone-pool1" with 1 replica(s) exists
	+ apply embedded addons
Do you want to proceed (yes/no): yes

kubectl get nodes -o wide
NAME                             STATUS   ROLES           AGE    VERSION   INTERNAL-IP   EXTERNAL-IP       OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
kubeone-control-plane-1          Ready    control-plane   8m9s   v1.24.1   192.168.0.3   167.235.148.156   Ubuntu 22.04.1 LTS   5.15.0-46-generic   containerd://1.6.10
kubeone-pool1-786ddf594f-fx5cr   Ready    <none>          94s    v1.24.1   192.168.0.4   5.75.141.58       Ubuntu 22.04.1 LTS   5.15.0-46-generic   containerd://1.6.10

kubectl get pods --all-namespaces
NAMESPACE            NAME                                               READY   STATUS    RESTARTS   AGE
kube-system          calico-kube-controllers-77559f8ddc-9dsjf           1/1     Running   0          7m14s
kube-system          canal-q5mz8                                        2/2     Running   0          2m1s
kube-system          canal-wbjfm                                        2/2     Running   0          7m15s
kube-system          cluster-autoscaler-c9b79c6dd-l8nb8                 1/1     Running   0          7m1s
kube-system          coredns-56fc8c7fff-bfcws                           1/1     Running   0          7m14s
kube-system          coredns-56fc8c7fff-sv76f                           1/1     Running   0          7m14s
kube-system          coredns-56fc8c7fff-zdqjk                           1/1     Running   0          6m49s
kube-system          etcd-kubeone-control-plane-1                       1/1     Running   0          8m34s
kube-system          hcloud-cloud-controller-manager-8585b85967-r2gd2   1/1     Running   0          7m8s
kube-system          hcloud-csi-controller-0                            5/5     Running   0          7m10s
kube-system          hcloud-csi-node-kqzlx                              3/3     Running   0          7m10s
kube-system          hcloud-csi-node-s9w7v                              3/3     Running   0          2m1s
kube-system          kube-apiserver-kubeone-control-plane-1             1/1     Running   0          8m34s
kube-system          kube-controller-manager-kubeone-control-plane-1    1/1     Running   0          7m45s
kube-system          kube-proxy-tl9gf                                   1/1     Running   0          2m1s
kube-system          kube-proxy-tpqt6                                   1/1     Running   0          8m34s
kube-system          kube-scheduler-kubeone-control-plane-1             1/1     Running   0          8m34s
kube-system          kured-8nq58                                        1/1     Running   0          60s
kube-system          kured-s2n24                                        1/1     Running   0          6m42s
kube-system          machine-controller-c5c6495f6-ff72s                 1/1     Running   0          7m14s
kube-system          machine-controller-webhook-76f8484cd-qflbg         1/1     Running   0          7m14s
kube-system          metrics-server-6c8f44fb59-g7747                    1/1     Running   0          7m14s
kube-system          node-local-dns-5phzl                               1/1     Running   0          2m1s
kube-system          node-local-dns-dnvrq                               1/1     Running   0          7m15s
kube-system          operating-system-manager-6cb7c45b4b-kjkz2          1/1     Running   0          7m14s
kube-system          operating-system-manager-webhook-7cccc556f-q5xhf   1/1     Running   0          7m14s
kube-system          unattended-upgrades-install-7znbr                  1/1     Running   0          60s
kube-system          unattended-upgrades-install-tn7t6                  1/1     Running   0          6m42s
reboot-coordinator   flatcar-linux-update-operator-755886c688-7lhsn     0/1     Pending   0          6m56s

Upgrade to "1.24.2"

kubeone apply --credentials credentials.yaml --manifest kubeone.yaml --tfjson tf.json --upgrade-machine-deployments
INFO[21:15:23 CET] Determine hostname...                        
INFO[21:15:24 CET] Determine operating system...                
INFO[21:15:26 CET] Running host probes...                       
INFO[21:15:33 CET] Electing cluster leader...                   
INFO[21:15:33 CET] Elected leader "kubeone-control-plane-1"...  
INFO[21:15:37 CET] Building Kubernetes clientset...             
INFO[21:15:38 CET] Running cluster probes...                    
The following actions will be taken: 
Run with --verbose flag for more information.

	~ upgrade control plane node "kubeone-control-plane-1" (192.168.0.3): 1.24.1 -> 1.24.2
	~ ensure credential
	~ ensure embedded addons
	~ ensure CNI
	~ ensure custom addons
	~ ensure external CCM
	~ upgrade MachineDeployments

Do you want to proceed (yes/no): 

Errors

ERRO[21:17:03 CET] error when evicting pods/"coredns-56fc8c7fff-zdqjk" -n "kube-system" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.  node=167.235.148.156
ERRO[21:17:03 CET] error when evicting pods/"coredns-56fc8c7fff-sv76f" -n "kube-system" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.  node=167.235.148.156

kubectl get nodes -o wide
NAME                             STATUS   ROLES           AGE   VERSION   INTERNAL-IP   EXTERNAL-IP       OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
kubeone-control-plane-1          Ready    control-plane   19m   v1.24.2   192.168.0.3   167.235.148.156   Ubuntu 22.04.1 LTS   5.15.0-46-generic   containerd://1.6.10
kubeone-pool1-786ddf594f-fx5cr   Ready    <none>          12m   v1.24.1   192.168.0.4   5.75.141.58       Ubuntu 22.04.1 LTS   5.15.0-46-generic   containerd://1.6.10

kubectl get pods --all-namespaces
NAMESPACE            NAME                                               READY   STATUS    RESTARTS        AGE
kube-system          calico-kube-controllers-77559f8ddc-7wdkc           1/1     Running   0               8m21s
kube-system          canal-q5mz8                                        2/2     Running   0               13m
kube-system          canal-wbjfm                                        2/2     Running   0               18m
kube-system          cluster-autoscaler-c9b79c6dd-68vjq                 1/1     Running   0               8m22s
kube-system          coredns-56fc8c7fff-fdxrq                           1/1     Running   0               105s
kube-system          coredns-56fc8c7fff-gvnnc                           1/1     Running   0               105s
kube-system          coredns-56fc8c7fff-p59c4                           1/1     Running   0               104s
kube-system          etcd-kubeone-control-plane-1                       1/1     Running   0               19m
kube-system          hcloud-cloud-controller-manager-8585b85967-gmgng   1/1     Running   0               8m22s
kube-system          hcloud-csi-controller-0                            5/5     Running   0               8m16s
kube-system          hcloud-csi-node-kqzlx                              3/3     Running   6 (3m27s ago)   18m
kube-system          hcloud-csi-node-s9w7v                              3/3     Running   0               13m
kube-system          kube-apiserver-kubeone-control-plane-1             1/1     Running   0               4m55s
kube-system          kube-controller-manager-kubeone-control-plane-1    1/1     Running   0               2m14s
kube-system          kube-proxy-ftcnh                                   1/1     Running   0               4m3s
kube-system          kube-proxy-wnzsp                                   1/1     Running   0               4m7s
kube-system          kube-scheduler-kubeone-control-plane-1             1/1     Running   0               4m23s
kube-system          kured-8nq58                                        1/1     Running   0               12m
kube-system          kured-s2n24                                        1/1     Running   3 (3m22s ago)   17m
kube-system          machine-controller-c5c6495f6-sd42w                 1/1     Running   0               8m22s
kube-system          machine-controller-webhook-76f8484cd-xkqrl         1/1     Running   0               8m22s
kube-system          metrics-server-6c8f44fb59-96s7v                    1/1     Running   0               8m21s
kube-system          node-local-dns-5phzl                               1/1     Running   0               13m
kube-system          node-local-dns-dnvrq                               1/1     Running   0               18m
kube-system          operating-system-manager-6cb7c45b4b-n9lw8          1/1     Running   0               8m22s
kube-system          operating-system-manager-webhook-7cccc556f-htj48   1/1     Running   0               8m21s
kube-system          unattended-upgrades-install-7znbr                  1/1     Running   0               12m
kube-system          unattended-upgrades-install-tn7t6                  1/1     Running   4 (3m20s ago)   17m
reboot-coordinator   flatcar-linux-update-operator-755886c688-7lhsn     0/1     Pending   0               18m

[embik]

@clickersmudge This looks completely fine. PodDisruptionBudgets ensure that Pods do not get evicted from nodes if that would cause a service outage, that is what you're seeing (a PDB blocking an eviction). Note how it says that it will retry and eventually succeeds evicting the same Pod in the next log lines.

You can reduce this by running several control planes, so that critical Pods like CoreDNS can be distributed across several nodes.

[clickersmudge]

@embik Thank you. We launched three control planes and three worker nodes and the problem disappeared.

Only this information remains:

ERRO[10:16:42 CET] WARNING: ignoring DaemonSet-managed Pods: kube-system/canal-2fxq6, kube-system/hcloud-csi-node-k9rrk, kube-system/kube-proxy-x8cbp, kube-system/kured-cmg8z, kube-system/node-local-dns-shs47, kube-system/unattended-upgrades-install-4g7xh  node=49.12.206.32

[xmudrii]

@clickersmudge This is as expected, it's only a warning that DaemonSet-managed pods are ignored by eviction.