General Troubleshooting #20

timothystewart6 · 2022-04-05T03:18:15Z

timothystewart6
Apr 5, 2022
Maintainer

Troubleshooting

Generally speaking, here are things to check if this does not work for you.

✅ Minimum ansible version is 2.11
✅ Reset your cluster using the reset task in reset.yml. This cleans up any changes. You must reset your cluster if anything fails.
✅ Installed and verified that ansible requirements are installed with ansible-galaxy install -r ./collections/requirements.yml
✅ Variables match the ones used below, including no additional args.
✅ That all server / master nodes have the same network interface name (flannel_iface this cannot vary)
✅ If they do not match, create one master / server node and add additional servers outside of this playbook
✅ All machines have enough disk space, at least 4 GB (minimum)
✅ Drives are fast enough to run etcd (SSD or faster)
✅ Network / disk latency is low enough to run etcd
✅ Run the reset command, ensure this reboots all machines.
✅ Same time zone on all machines
✅ k3s_token does not contain special characters (alpha numeric is preferred)
✅ ssh keys work on all nodes
✅ ansible_user is the user that has ssh access via ansible
✅ You have the correct interface name for all nodes (default is eth0)

To see additional commands and more documentation see the docs

How I test this in my HomeLab

Machines

(5) 20.04 LTS Ubuntu machines

Variables (example)

Be sure you are using the default variables here

kubectl

(from one of the server / master nodes)

➜  ~ sudo kubectl get nodes

NAME     STATUS   ROLES                       AGE     VERSION
k3s-01   Ready    control-plane,etcd,master   5m40s   v1.23.4+k3s1
k3s-02   Ready    control-plane,etcd,master   5m25s   v1.23.4+k3s1
k3s-03   Ready    control-plane,etcd,master   5m8s    v1.23.4+k3s1
k3s-04   Ready    <none>                      2m42s   v1.23.4+k3s1
k3s-05   Ready    <none>                      2m42s   v1.23.4+k3s1

VIP

Can ping VIP

ping 192.168.30.222
64 bytes from 192.168.30.222: icmp_seq=2552 ttl=62 time=0.702 ms

Logs

Expand to see log output

➜  k3s-ansible git:(master) ✗ ansible-playbook site.yml -i inventory/sample/hosts.ini

PLAY [k3s_cluster] ***********************************************************************************************************************

TASK [Gathering Facts] *******************************************************************************************************************
Monday 04 April 2022  18:27:07 -0500 (0:00:00.098)       0:00:00.098 **********
ok: [192.168.30.38]
ok: [192.168.30.42]
ok: [192.168.30.39]
ok: [192.168.30.40]
ok: [192.168.30.41]

TASK [prereq : Set SELinux to disabled state] ********************************************************************************************
Monday 04 April 2022  18:27:09 -0500 (0:00:02.135)       0:00:02.233 **********
skipping: [192.168.30.38]
skipping: [192.168.30.39]
skipping: [192.168.30.40]
skipping: [192.168.30.41]
skipping: [192.168.30.42]

TASK [prereq : Enable IPv4 forwarding] ***************************************************************************************************
Monday 04 April 2022  18:27:10 -0500 (0:00:00.155)       0:00:02.389 **********
ok: [192.168.30.39]
ok: [192.168.30.38]
ok: [192.168.30.41]
ok: [192.168.30.42]
ok: [192.168.30.40]

TASK [prereq : Enable IPv6 forwarding] ***************************************************************************************************
Monday 04 April 2022  18:27:10 -0500 (0:00:00.725)       0:00:03.114 **********
ok: [192.168.30.38]
ok: [192.168.30.40]
ok: [192.168.30.39]
ok: [192.168.30.41]
ok: [192.168.30.42]

TASK [prereq : Add br_netfilter to /etc/modules-load.d/] *********************************************************************************
Monday 04 April 2022  18:27:11 -0500 (0:00:00.641)       0:00:03.756 **********
skipping: [192.168.30.38]
skipping: [192.168.30.39]
skipping: [192.168.30.40]
skipping: [192.168.30.41]
skipping: [192.168.30.42]

TASK [prereq : Load br_netfilter] ********************************************************************************************************
Monday 04 April 2022  18:27:11 -0500 (0:00:00.208)       0:00:03.965 **********
skipping: [192.168.30.38]
skipping: [192.168.30.39]
skipping: [192.168.30.40]
skipping: [192.168.30.41]
skipping: [192.168.30.42]

TASK [prereq : Set bridge-nf-call-iptables (just to be sure)] ****************************************************************************
Monday 04 April 2022  18:27:11 -0500 (0:00:00.190)       0:00:04.156 **********
skipping: [192.168.30.38] => (item=net.bridge.bridge-nf-call-iptables)
skipping: [192.168.30.38] => (item=net.bridge.bridge-nf-call-ip6tables)
skipping: [192.168.30.39] => (item=net.bridge.bridge-nf-call-iptables)
skipping: [192.168.30.39] => (item=net.bridge.bridge-nf-call-ip6tables)
skipping: [192.168.30.40] => (item=net.bridge.bridge-nf-call-iptables)
skipping: [192.168.30.40] => (item=net.bridge.bridge-nf-call-ip6tables)
skipping: [192.168.30.41] => (item=net.bridge.bridge-nf-call-iptables)
skipping: [192.168.30.41] => (item=net.bridge.bridge-nf-call-ip6tables)
skipping: [192.168.30.42] => (item=net.bridge.bridge-nf-call-iptables)
skipping: [192.168.30.42] => (item=net.bridge.bridge-nf-call-ip6tables)

TASK [prereq : Add /usr/local/bin to sudo secure_path] ***********************************************************************************
Monday 04 April 2022  18:27:12 -0500 (0:00:00.174)       0:00:04.330 **********
skipping: [192.168.30.38]
skipping: [192.168.30.39]
skipping: [192.168.30.40]
skipping: [192.168.30.41]
skipping: [192.168.30.42]

TASK [download : Download k3s binary x64] ************************************************************************************************
Monday 04 April 2022  18:27:12 -0500 (0:00:00.240)       0:00:04.570 **********
changed: [192.168.30.42]
changed: [192.168.30.38]
changed: [192.168.30.41]
changed: [192.168.30.39]
changed: [192.168.30.40]

TASK [download : Download k3s binary arm64] **********************************************************************************************
Monday 04 April 2022  18:27:19 -0500 (0:00:06.887)       0:00:11.458 **********
skipping: [192.168.30.38]
skipping: [192.168.30.39]
skipping: [192.168.30.40]
skipping: [192.168.30.41]
skipping: [192.168.30.42]

TASK [download : Download k3s binary armhf] **********************************************************************************************
Monday 04 April 2022  18:27:19 -0500 (0:00:00.213)       0:00:11.671 **********
skipping: [192.168.30.38]
skipping: [192.168.30.39]
skipping: [192.168.30.40]
skipping: [192.168.30.41]
skipping: [192.168.30.42]

TASK [raspberrypi : Test for raspberry pi /proc/cpuinfo] *********************************************************************************
Monday 04 April 2022  18:27:19 -0500 (0:00:00.210)       0:00:11.882 **********
ok: [192.168.30.42]
ok: [192.168.30.38]
ok: [192.168.30.40]
ok: [192.168.30.41]
ok: [192.168.30.39]

TASK [raspberrypi : Test for raspberry pi /proc/device-tree/model] ***********************************************************************
Monday 04 April 2022  18:27:20 -0500 (0:00:00.778)       0:00:12.660 **********
ok: [192.168.30.38]
ok: [192.168.30.40]
ok: [192.168.30.41]
ok: [192.168.30.42]
ok: [192.168.30.39]

TASK [raspberrypi : Set raspberry_pi fact to true] ***************************************************************************************
Monday 04 April 2022  18:27:20 -0500 (0:00:00.557)       0:00:13.218 **********
skipping: [192.168.30.38]
skipping: [192.168.30.39]
skipping: [192.168.30.40]
skipping: [192.168.30.41]
skipping: [192.168.30.42]

TASK [raspberrypi : Set detected_distribution to Raspbian] *******************************************************************************
Monday 04 April 2022  18:27:21 -0500 (0:00:00.257)       0:00:13.475 **********
skipping: [192.168.30.38]
skipping: [192.168.30.39]
skipping: [192.168.30.40]
skipping: [192.168.30.41]
skipping: [192.168.30.42]

TASK [raspberrypi : Set detected_distribution to Raspbian (ARM64 on Debian Buster)] ******************************************************
Monday 04 April 2022  18:27:21 -0500 (0:00:00.283)       0:00:13.758 **********
skipping: [192.168.30.38]
skipping: [192.168.30.39]
skipping: [192.168.30.40]
skipping: [192.168.30.42]
skipping: [192.168.30.41]

TASK [raspberrypi : Set detected_distribution_major_version] *****************************************************************************
Monday 04 April 2022  18:27:21 -0500 (0:00:00.227)       0:00:13.986 **********
skipping: [192.168.30.38]
skipping: [192.168.30.39]
skipping: [192.168.30.40]
skipping: [192.168.30.41]
skipping: [192.168.30.42]

TASK [raspberrypi : execute OS related tasks on the Raspberry Pi] ************************************************************************
Monday 04 April 2022  18:27:21 -0500 (0:00:00.256)       0:00:14.242 **********
skipping: [192.168.30.38] => (item=/home/user/dev/public/k3s-ansible/roles/raspberrypi/tasks/prereq/Ubuntu.yml)
skipping: [192.168.30.39] => (item=/home/user/dev/public/k3s-ansible/roles/raspberrypi/tasks/prereq/Ubuntu.yml)
skipping: [192.168.30.40] => (item=/home/user/dev/public/k3s-ansible/roles/raspberrypi/tasks/prereq/Ubuntu.yml)
skipping: [192.168.30.41] => (item=/home/user/dev/public/k3s-ansible/roles/raspberrypi/tasks/prereq/Ubuntu.yml)
skipping: [192.168.30.42] => (item=/home/user/dev/public/k3s-ansible/roles/raspberrypi/tasks/prereq/Ubuntu.yml)

PLAY [master] ****************************************************************************************************************************

TASK [Gathering Facts] *******************************************************************************************************************
Monday 04 April 2022  18:27:22 -0500 (0:00:00.295)       0:00:14.538 **********
ok: [192.168.30.39]
ok: [192.168.30.38]
ok: [192.168.30.40]

TASK [k3s/master : Clean previous runs of k3s-init] **************************************************************************************
Monday 04 April 2022  18:27:23 -0500 (0:00:00.991)       0:00:15.529 **********
ok: [192.168.30.40]
ok: [192.168.30.38]
ok: [192.168.30.39]

TASK [k3s/master : Clean previous runs of k3s-init] **************************************************************************************
Monday 04 April 2022  18:27:24 -0500 (0:00:01.103)       0:00:16.633 **********
ok: [192.168.30.38]
ok: [192.168.30.40]
ok: [192.168.30.39]

TASK [k3s/master : Create manifests directory on first master] ***************************************************************************
Monday 04 April 2022  18:27:24 -0500 (0:00:00.529)       0:00:17.162 **********
skipping: [192.168.30.39]
skipping: [192.168.30.40]
changed: [192.168.30.38]

TASK [k3s/master : Copy vip rbac manifest to first master] *******************************************************************************
Monday 04 April 2022  18:27:25 -0500 (0:00:00.871)       0:00:18.033 **********
skipping: [192.168.30.40]
skipping: [192.168.30.39]
changed: [192.168.30.38]

TASK [k3s/master : Copy vip manifest to first master] ************************************************************************************
Monday 04 April 2022  18:27:27 -0500 (0:00:01.343)       0:00:19.377 **********
skipping: [192.168.30.39]
skipping: [192.168.30.40]
changed: [192.168.30.38]

TASK [k3s/master : Copy metallb namespace manifest to first master] **********************************************************************
Monday 04 April 2022  18:27:27 -0500 (0:00:00.628)       0:00:20.005 **********
skipping: [192.168.30.39]
skipping: [192.168.30.40]
changed: [192.168.30.38]

TASK [k3s/master : Copy metallb ConfigMap manifest to first master] **********************************************************************
Monday 04 April 2022  18:27:28 -0500 (0:00:00.656)       0:00:20.662 **********
skipping: [192.168.30.39]
skipping: [192.168.30.40]
changed: [192.168.30.38]

TASK [k3s/master : Copy metallb main manifest to first master] ***************************************************************************
Monday 04 April 2022  18:27:29 -0500 (0:00:00.853)       0:00:21.516 **********
skipping: [192.168.30.39]
skipping: [192.168.30.40]
changed: [192.168.30.38]

TASK [k3s/master : Init cluster inside the transient k3s-init service] *******************************************************************
Monday 04 April 2022  18:27:29 -0500 (0:00:00.710)       0:00:22.227 **********
changed: [192.168.30.39]
changed: [192.168.30.40]
changed: [192.168.30.38]

TASK [k3s/master : Verify that all nodes actually joined (check k3s-init.service if this fails)] *****************************************
Monday 04 April 2022  18:27:30 -0500 (0:00:00.573)       0:00:22.801 **********
FAILED - RETRYING: Verify that all nodes actually joined (check k3s-init.service if this fails) (20 retries left).
FAILED - RETRYING: Verify that all nodes actually joined (check k3s-init.service if this fails) (20 retries left).
FAILED - RETRYING: Verify that all nodes actually joined (check k3s-init.service if this fails) (20 retries left).
FAILED - RETRYING: Verify that all nodes actually joined (check k3s-init.service if this fails) (19 retries left).
FAILED - RETRYING: Verify that all nodes actually joined (check k3s-init.service if this fails) (19 retries left).
FAILED - RETRYING: Verify that all nodes actually joined (check k3s-init.service if this fails) (19 retries left).
FAILED - RETRYING: Verify that all nodes actually joined (check k3s-init.service if this fails) (18 retries left).
FAILED - RETRYING: Verify that all nodes actually joined (check k3s-init.service if this fails) (18 retries left).
FAILED - RETRYING: Verify that all nodes actually joined (check k3s-init.service if this fails) (18 retries left).
FAILED - RETRYING: Verify that all nodes actually joined (check k3s-init.service if this fails) (17 retries left).
FAILED - RETRYING: Verify that all nodes actually joined (check k3s-init.service if this fails) (17 retries left).
FAILED - RETRYING: Verify that all nodes actually joined (check k3s-init.service if this fails) (17 retries left).
FAILED - RETRYING: Verify that all nodes actually joined (check k3s-init.service if this fails) (16 retries left).
FAILED - RETRYING: Verify that all nodes actually joined (check k3s-init.service if this fails) (16 retries left).
FAILED - RETRYING: Verify that all nodes actually joined (check k3s-init.service if this fails) (16 retries left).
ok: [192.168.30.38]
ok: [192.168.30.40]
ok: [192.168.30.39]

TASK [k3s/master : Kill the temporary service used for initialization] *******************************************************************
Monday 04 April 2022  18:28:33 -0500 (0:01:02.458)       0:01:25.259 **********
changed: [192.168.30.39]
changed: [192.168.30.40]
changed: [192.168.30.38]

TASK [k3s/master : Copy K3s service file] ************************************************************************************************
Monday 04 April 2022  18:30:03 -0500 (0:01:30.905)       0:02:56.165 **********
changed: [192.168.30.38]
changed: [192.168.30.39]
changed: [192.168.30.40]

TASK [k3s/master : Enable and check K3s service] *****************************************************************************************
Monday 04 April 2022  18:30:04 -0500 (0:00:00.900)       0:02:57.065 **********
changed: [192.168.30.39]
changed: [192.168.30.40]
changed: [192.168.30.38]

TASK [k3s/master : Wait for node-token] **************************************************************************************************
Monday 04 April 2022  18:30:20 -0500 (0:00:15.987)       0:03:13.053 **********
ok: [192.168.30.38]
ok: [192.168.30.40]
ok: [192.168.30.39]

TASK [k3s/master : Register node-token file access mode] *********************************************************************************
Monday 04 April 2022  18:30:21 -0500 (0:00:00.766)       0:03:13.820 **********
ok: [192.168.30.40]
ok: [192.168.30.38]
ok: [192.168.30.39]

TASK [k3s/master : Change file access node-token] ****************************************************************************************
Monday 04 April 2022  18:30:21 -0500 (0:00:00.365)       0:03:14.185 **********
changed: [192.168.30.39]
changed: [192.168.30.40]
changed: [192.168.30.38]

TASK [k3s/master : Read node-token from master] ******************************************************************************************
Monday 04 April 2022  18:30:22 -0500 (0:00:00.539)       0:03:14.724 **********
ok: [192.168.30.38]
ok: [192.168.30.39]
ok: [192.168.30.40]

TASK [k3s/master : Store Master node-token] **********************************************************************************************
Monday 04 April 2022  18:30:23 -0500 (0:00:00.554)       0:03:15.279 **********
ok: [192.168.30.38]
ok: [192.168.30.39]
ok: [192.168.30.40]

TASK [k3s/master : Restore node-token file access] ***************************************************************************************
Monday 04 April 2022  18:30:23 -0500 (0:00:00.154)       0:03:15.433 **********
changed: [192.168.30.38]
changed: [192.168.30.39]
changed: [192.168.30.40]

TASK [k3s/master : Create directory .kube] ***********************************************************************************************
Monday 04 April 2022  18:30:23 -0500 (0:00:00.571)       0:03:16.004 **********
ok: [192.168.30.38]
ok: [192.168.30.40]
ok: [192.168.30.39]

TASK [k3s/master : Copy config file to user home directory] ******************************************************************************
Monday 04 April 2022  18:30:24 -0500 (0:00:00.559)       0:03:16.564 **********
changed: [192.168.30.38]
changed: [192.168.30.39]
changed: [192.168.30.40]

TASK [k3s/master : Configure kubectl cluster to https://192.168.30.222:6443] *************************************************************
Monday 04 April 2022  18:30:25 -0500 (0:00:00.691)       0:03:17.255 **********
changed: [192.168.30.39]
changed: [192.168.30.40]
changed: [192.168.30.38]

TASK [k3s/master : Create kubectl symlink] ***********************************************************************************************
Monday 04 April 2022  18:30:25 -0500 (0:00:00.788)       0:03:18.043 **********
ok: [192.168.30.38]
ok: [192.168.30.40]
ok: [192.168.30.39]

TASK [k3s/master : Create crictl symlink] ************************************************************************************************
Monday 04 April 2022  18:30:26 -0500 (0:00:00.364)       0:03:18.408 **********
ok: [192.168.30.38]
ok: [192.168.30.39]
ok: [192.168.30.40]

PLAY [node] ******************************************************************************************************************************

TASK [Gathering Facts] *******************************************************************************************************************
Monday 04 April 2022  18:30:26 -0500 (0:00:00.634)       0:03:19.043 **********
ok: [192.168.30.41]
ok: [192.168.30.42]

TASK [k3s/node : Copy K3s service file] **************************************************************************************************
Monday 04 April 2022  18:30:28 -0500 (0:00:01.553)       0:03:20.596 **********
changed: [192.168.30.41]
changed: [192.168.30.42]

TASK [k3s/node : Enable and check K3s service] *******************************************************************************************
Monday 04 April 2022  18:30:29 -0500 (0:00:00.843)       0:03:21.439 **********
changed: [192.168.30.42]
changed: [192.168.30.41]

PLAY RECAP *******************************************************************************************************************************
192.168.30.38              : ok=31   changed=15   unreachable=0    failed=0    skipped=12   rescued=0    ignored=0
192.168.30.39              : ok=25   changed=9    unreachable=0    failed=0    skipped=18   rescued=0    ignored=0
192.168.30.40              : ok=25   changed=9    unreachable=0    failed=0    skipped=18   rescued=0    ignored=0
192.168.30.41              : ok=9    changed=3    unreachable=0    failed=0    skipped=12   rescued=0    ignored=0
192.168.30.42              : ok=9    changed=3    unreachable=0    failed=0    skipped=12   rescued=0    ignored=0

Monday 04 April 2022  18:30:51 -0500 (0:00:22.709)       0:03:44.149 **********
===============================================================================
k3s/master : Kill the temporary service used for initialization ------------------------------------------------------------------ 90.91s
k3s/master : Verify that all nodes actually joined (check k3s-init.service if this fails) ---------------------------------------- 62.46s
k3s/node : Enable and check K3s service ------------------------------------------------------------------------------------------ 22.71s
k3s/master : Enable and check K3s service ---------------------------------------------------------------------------------------- 15.99s
download : Download k3s binary x64 ------------------------------------------------------------------------------------------------ 6.89s
Gathering Facts ------------------------------------------------------------------------------------------------------------------- 2.14s
Gathering Facts ------------------------------------------------------------------------------------------------------------------- 1.55s
k3s/master : Copy vip rbac manifest to first master ------------------------------------------------------------------------------- 1.34s
k3s/master : Clean previous runs of k3s-init -------------------------------------------------------------------------------------- 1.10s
Gathering Facts ------------------------------------------------------------------------------------------------------------------- 0.99s
k3s/master : Copy K3s service file ------------------------------------------------------------------------------------------------ 0.90s
k3s/master : Create manifests directory on first master --------------------------------------------------------------------------- 0.87s
k3s/master : Copy metallb ConfigMap manifest to first master ---------------------------------------------------------------------- 0.85s
k3s/node : Copy K3s service file -------------------------------------------------------------------------------------------------- 0.84s
k3s/master : Configure kubectl cluster to https://192.168.30.222:6443 ------------------------------------------------------------- 0.79s
raspberrypi : Test for raspberry pi /proc/cpuinfo --------------------------------------------------------------------------------- 0.78s
k3s/master : Wait for node-token -------------------------------------------------------------------------------------------------- 0.77s
prereq : Enable IPv4 forwarding --------------------------------------------------------------------------------------------------- 0.73s
k3s/master : Copy metallb main manifest to first master --------------------------------------------------------------------------- 0.71s
k3s/master : Copy config file to user home directory ------------------------------------------------------------------------------ 0.69s
➜  k3s-ansible git:(master) ✗

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

General Troubleshooting #20

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Replies: 0 comments

Select a reply

Uh oh!

Uh oh!

General Troubleshooting #20

Uh oh!

Uh oh!

timothystewart6 Apr 5, 2022 Maintainer

Troubleshooting

How I test this in my HomeLab

Machines

Variables (example)

kubectl

VIP

Logs

Replies: 0 comments

timothystewart6
Apr 5, 2022
Maintainer