Expired kubeconfig and control plane certs – how to renew in kube-hetzner setup?

[steand94]

Hi everyone!

We are currently facing an issue related to certificate expiration. After approximately one year of stable operation, we have lost access to our Kubernetes cluster due to an expired certificate in the kubeconfig file. Based on our investigation, it appears that the control plane certificates have also expired.

Attempting to use kubectl results in the following error:

E0424 17:48:29.913976   25245 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: the server has asked for the client to provide credentials"
E0424 17:48:30.058427   25245 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: the server has asked for the client to provide credentials"
E0424 17:48:30.206682   25245 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: the server has asked for the client to provide credentials"
E0424 17:48:30.348468   25245 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: the server has asked for the client to provide credentials"
E0424 17:48:30.493202   25245 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: the server has asked for the client to provide credentials"
error: You must be logged in to the server (the server has asked for the client to provide credentials)

We attempted to follow the standard renewal process using kubeadm certs renew all, but found that kubeadm is not available on the control plane node. Before proceeding with any manual interventions, we would like to ask:

1. What is the recommended or supported procedure within the kube-hetzner setup for renewing control plane and kubeconfig certificates?
2. Is kubeadm expected to be installed by default on control plane nodes provisioned with this module?
3. Are there any built-in mechanisms or recommended practices to automate certificate renewal for long-running clusters?

We would greatly appreciate any guidance or documentation on the appropriate way to handle certificate renewal in this context.

Thank you.

[steand94]

For those who faced the same issue and have doubts about how to manage this situation correctly :)

The simple answer

Just run:

terraform apply

After completion, you will get an updated kubeconfig with renewed certificate:

terraform output kubeconfig

The long answer

K3s automatically rotates certificates about 90 days before expiration (see https://docs.k3s.io/cli/certificate). The updated certificates are stored at:

/var/lib/rancher/k3s/server/tls

And a new kubeconfig is generated at:

/etc/rancher/k3s/k3s.yaml

To manually retrieve and use it:

1. SSH into the first control plane node:

ssh root@<server-ip> -i ./path/to/privatekey -o StrictHostKeyChecking=no

2. Open the kubeconfig with renewed certificates:

cat /etc/rancher/k3s/k3s.yaml

3. Copy the client-certificate-data and client-key-data parameters, and replace them in your local kubeconfig file.

[PCatinean]

Thank you for posting the solution, the question itself had me worried a bit since I am currently in the process of moving from rancher to this 😄

[TimHeckel]

Hey this was realy helpful, but the second command in the simple answer should be

terraform output kubeconfig

I think. Thanks!

[steand94]

Yeah, my bad, apologies. Fixed.