K3s Service stuck at activating on nodes

[Anguianolabs]

k3s-node.service does not start and is stuck on activating.

Expected Behavior

k3s-node.server should be running on nodes.

Current Behavior

`TASK [k3s/master : Create crictl symlink] ******************************************************************************changed: [us-ga-cluster-02]
changed: [us-ga-cluster-01]
changed: [us-ga-cluster-03]

PLAY [node] ************************************************************************************************************
TASK [Gathering Facts] *************************************************************************************************ok: [us-ga-worker-01]
ok: [us-ga-worker-02]

TASK [k3s/node : Copy K3s service file] ********************************************************************************changed: [us-ga-worker-01]
changed: [us-ga-worker-02]

TASK [k3s/node : Enable and check K3s service] *************************************************************************`

`ubuntu@us-ga-worker-01:~$ sudo systemctl status k3s-node.service
● k3s-node.service - Lightweight Kubernetes
Loaded: loaded (/etc/systemd/system/k3s-node.service; enabled; vendor preset: enabled)
Active: activating (start) since Fri 2022-05-20 15:49:17 EDT; 53s ago
Docs: https://k3s.io
Process: 2513 ExecStartPre=/sbin/modprobe br_netfilter (code=exited, status=0/SUCCESS)
Process: 2522 ExecStartPre=/sbin/modprobe overlay (code=exited, status=0/SUCCESS)
Main PID: 2527 (k3s-agent)
Tasks: 8
Memory: 31.2M
CGroup: /system.slice/k3s-node.service
└─2527 /usr/local/bin/k3s agent

May 20 15:49:17 us-ga-worker-01 k3s[2527]: time="2022-05-20T15:49:17-04:00" level=info msg="Starting k3s agent v1.23.6+k3s1 (418c3fa8)"
May 20 15:49:17 us-ga-worker-01 k3s[2527]: time="2022-05-20T15:49:17-04:00" level=info msg="Running load balancer k3s-agent-load-balancer 127.0.0.1:6444 -> [10.20.0.46:6443]"
May 20 15:49:23 us-ga-worker-01 k3s[2527]: time="2022-05-20T15:49:23-04:00" level=error msg="failed to get CA certs: Get "https://127.0.0.1:6444/cacerts\": read tcp 127.0.0.1:51760->127.0.0.1:6444: read: connection reset by peer"
May 20 15:49:29 us-ga-worker-01 k3s[2527]: time="2022-05-20T15:49:29-04:00" level=error msg="failed to get CA certs: Get "https://127.0.0.1:6444/cacerts\": read tcp 127.0.0.1:51768->127.0.0.1:6444: read: connection reset by peer"
May 20 15:49:35 us-ga-worker-01 k3s[2527]: time="2022-05-20T15:49:35-04:00" level=error msg="failed to get CA certs: Get "https://127.0.0.1:6444/cacerts\": read tcp 127.0.0.1:51776->127.0.0.1:6444: read: connection reset by peer"
May 20 15:49:42 us-ga-worker-01 k3s[2527]: time="2022-05-20T15:49:42-04:00" level=error msg="failed to get CA certs: Get "https://127.0.0.1:6444/cacerts\": read tcp 127.0.0.1:51784->127.0.0.1:6444: read: connection reset by peer"
May 20 15:49:48 us-ga-worker-01 k3s[2527]: time="2022-05-20T15:49:48-04:00" level=error msg="failed to get CA certs: Get "https://127.0.0.1:6444/cacerts\": read tcp 127.0.0.1:51792->127.0.0.1:6444: read: connection reset by peer"
May 20 15:49:54 us-ga-worker-01 k3s[2527]: time="2022-05-20T15:49:54-04:00" level=error msg="failed to get CA certs: Get "https://127.0.0.1:6444/cacerts\": read tcp 127.0.0.1:51800->127.0.0.1:6444: read: connection reset by peer"
May 20 15:50:00 us-ga-worker-01 k3s[2527]: time="2022-05-20T15:50:00-04:00" level=error msg="failed to get CA certs: Get "https://127.0.0.1:6444/cacerts\": read tcp 127.0.0.1:51808->127.0.0.1:6444: read: connection reset by peer"
May 20 15:50:06 us-ga-worker-01 k3s[2527]: time="2022-05-20T15:50:06-04:00" level=error msg="failed to get CA certs: Get "https://127.0.0.1:6444/cacerts\": read tcp 127.0.0.1:51816->127.0.0.1:6444: read: connection reset by peer"`

Steps to Reproduce

1. ansible-playbook playbooks/site.yml -K

Context (variables)

Operating system:
ubuntu 20.04
Hardware:
Proxmox VM

Variables Used:

all.yml

k3s_version: v1.23.6+k3s1
# this is the user that has ssh access to these machines
ansible_user: ubuntu
systemd_dir: /etc/systemd/system

# Set your timezone
system_timezone: "America/New_York"

# interface which will be used for flannel
flannel_iface: "eth0"

# apiserver_endpoint is virtual ip-address which will be configured on each master
apiserver_endpoint: "10.20.0.46"

# k3s_token is required  masters can talk together securely
# this token should be alpha numeric only
k3s_token: "xxxxxxxxxxxxxxxxx"

# change these to your liking, the only required one is--no-deploy servicelb
extra_server_args: "--no-deploy servicelb --no-deploy traefik"
extra_agent_args: ""

# image tag for kube-vip
kube_vip_tag_version: "v0.4.3"

# image tag for metal lb
metal_lb_speaker_tag_version: "v0.12.1"
metal_lb_controller_tag_version: "v0.12.1"

# metallb ip range for load balancers
metal_lb_ip_range: "10.20.0.100-10.20.0.120"

Hosts

host.ini

[master]
us-ga-cluster-01
us-ga-cluster-02
us-ga-cluster-03

[node]
us-ga-worker-01
us-ga-worker-02

[k3s_cluster:children]
master
node

Possible Solution

[Anguianolabs]

vip is not reachable so i am thinking if that's the issue.

[vickyingle01]

I'm facing the same issue here, what if we only have one master? Do we need to specify the apiserver_endpoint same as the master IP? Kindly clarify.

[timothystewart6]

Hi. This is an issue with k3s/kube-vip. I would use the version in the template. It will always be the latest tested version

v1.23.4+k3s1

[tjkrz]

This fixed the issue I had getting a constant ping (connection) to the vip. Thanks!

[vickyingle01]

Tried v1.23.4+k3s1, same issue. The VIP is not responding.
Stuck on "TASK [k3s/node : Enable and check K3s service]"

Status on node = activating
level=error msg="failed to get CA certs: Get \"https://127.0.0.1:6444/cacerts\": read tcp 127.0.0.1:42274->127.0.0.1:6444: read: connection reset by peer"

running the playbook with -vvv

TASK [k3s/node : Enable and check K3s service] ****************************************************************************************************************************************************************
task path: /home/ubuntu/k3s-ansible/roles/k3s/node/tasks/main.yml:11
Tuesday 24 May 2022  10:35:54 +0930 (0:00:00.672)       0:00:27.029 *********** 
Using module file /usr/local/lib/python3.8/dist-packages/ansible/modules/systemd.py
Pipelining is enabled.
<172.31.23.xx> ESTABLISH SSH CONNECTION FOR USER: ubuntu
<172.31.23.xx> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="ubuntu"' -o ConnectTimeout=10 -o ControlPath=/home/ubuntu/.ansible/cp/50028a4cf5 172.31.23.xx '/bin/sh -c '"'"'sudo -H -S -n  -u root /bin/sh -c '"'"'"'"'"'"'"'"'echo BECOME-SUCCESS-yqsrpeklwobgjpkvekflzdmhxideormf ; /usr/bin/python3'"'"'"'"'"'"'"'"' && sleep 0'"'"''
Escalation succeeded

[timothystewart6]

Please see this #20

[Anguianolabs]

I wonder if the VIP problem is related to the interface type set in flannel_iface: "eth0" in all.yml. In my case my proxmox hosted ubuntu 20 vm's are getting this interface type. 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: ens18: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000 link/ether b2:07:c2:07:8d:03 brd ff:ff:ff:ff:ff:ff inet 10.20.0.44/24 brd 10.20.0.255 scope global ens18 valid_lft forever preferred_lft forever inet6 fe80::b007:c2ff:fe07:8d03/64 scope link valid_lft forever preferred_lft forever

Which leads to to believe that its failing at creating the network interface for flannel. Should i modify the interface in the yml file to flannel_iface: "ens18" instead?

[Turtlez32]

In my case this was my problem. VIP was not responding. Running VM's on Proxmox the interface was ens18. So have to change eth0 to ens18. I had this changed in my all.yml then did a git pull. So was overwritten with the repository version

[Anguianolabs]

My hunch was correct! everything is working now!

[midget214]

I'm having a similar issue where the playbook stops at the same step as the thread title.
I am not able to ping the VIP when it gets to this point. I've checked the interface name as per the other posts here and everything looks correct.

The only difference though is that I am attempting to run this against LXC containers rather than VMs - would this cause any issues?

[Turtlez32]

Wow trying with LXC I am intrigued how this works. Are you running these via physical linux or a hypervisor like proxmox.

some trouble shooting paths I would take:

1. Check IP of LXC (ip add)
2. Check I can ping that machine from others + LXC if you have different hardware
3. Check to see if your bridging for the LXC is working correctly.

Having only worked with proxmox and LXC I have not seen issues with the LAN address.

I am attempting to run this against LXC containers rather than VMs - would this cause any issues?
I would say no the only issue would be setting up the network correctly so you can see the VIP, which is why you are here.

[midget214]

I will try to add as much info as I can.

• I am hosting the LXC containers in Proxmox.
• Each container is created from a template - specifically Ubuntu 21.04 Standard (downloaded from the provided template list in Proxmox)
• Each container's network is setup to use the same bridge within Proxmox
• I can ping my router and each of the containers from within each container
• The network interfaces for each container is setup to use static IP addresses - IP subnet and VLAN tag are set to be identical - each container has their own unique IP though
• Each container is running the same user (well technically not the exact same user - but the username is the same)
• Each container has it's own SSH key - my ansible host can successfully connect to all containers though - so I doubt that is the issue
• Root is disabled on each container. The user I am connecting with can elevate to root though via sudo
• All containers are unprivilaged
• My cluster currently only contains 1 master and 1 agent. My understanding is that it shouldn't matter how many there are?

[midget214]

I forgot to add the ip info for each container.

Master

test@k3sMaster:~$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever

2: eth0@if34: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 46:ef:99:11:6c:53 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 192.168.107.111/24 brd 192.168.107.255 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::44ef:99ff:fe11:6c53/64 scope link
valid_lft forever preferred_lft forever

Agent

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever

2: eth0@if35: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 22:54:da:2d:b3:8d brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 192.168.107.32/24 brd 192.168.107.255 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::2054:daff:fe2d:b38d/64 scope link
valid_lft forever preferred_lft forever

[midget214]

Tried again this weekend - this time with VMs - and it works. I'm still at a loss why containers are not working.
For the time being I will make use of VMs to solve my particular problem at hand - but once I have that up and running I really want this to work on containers. VMs seems a bit overkill for my use case

[sachasmart]

I'm running into this issue too:

systemctl status k3s-node:

Jul 01 00:47:41 delta k3s[2541]: time="2022-07-01T00:47:41-04:00" level=error msg="failed to get CA certs: Get "https://127.0.0.1:6444/cacerts\": contex>`

It looks like the master's are running and I can ping the VIP. Any hints on where to turn next?

The master nodes ip a

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether fa:89:c1:86:61:13 brd ff:ff:ff:ff:ff:ff
    inet 192.168.2.55/24 brd 192.168.2.255 scope global dynamic eth0
       valid_lft 61574sec preferred_lft 61574sec
    inet 192.168.30.222/32 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::f889:c1ff:fe86:6113/64 scope link
       valid_lft forever preferred_lft forever
3: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default
    link/ether 26:c4:94:7f:51:05 brd ff:ff:ff:ff:ff:ff
    inet 10.42.0.0/32 scope global flannel.1
       valid_lft forever preferred_lft forever
    inet6 fe80::24c4:94ff:fe7f:5105/64 scope link
       valid_lft forever preferred_lft forever
4: cni0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default qlen 1000
    link/ether 76:66:40:f4:58:c7 brd ff:ff:ff:ff:ff:ff
    inet 10.42.0.1/24 brd 10.42.0.255 scope global cni0
       valid_lft forever preferred_lft forever
    inet6 fe80::7466:40ff:fef4:58c7/64 scope link
       valid_lft forever preferred_lft forever
5: veth902780c8@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master cni0 state UP group default
    link/ether 8e:fd:12:0c:6b:1e brd ff:ff:ff:ff:ff:ff link-netns cni-c6c5409e-6b5a-96af-b2e7-2496f4fccce3
    inet6 fe80::8cfd:12ff:fe0c:6b1e/64 scope link
       valid_lft forever preferred_lft forever
6: veth30b10623@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master cni0 state UP group default
    link/ether 0a:a0:18:f6:7c:dc brd ff:ff:ff:ff:ff:ff link-netns cni-a1dd27ee-adb7-1c82-8677-bcad900676b1
    inet6 fe80::8a0:18ff:fef6:7cdc/64 scope link

[sachasmart]

resolved mine.... hate to say it but it was (I believe) a DHCP issue and the allocated ip for the kube-vip was outside of the range. Put it back into the range and everything is running.