SIngle node deployment, where 1 master=worker

[knuurr]

Hi all, I'm currently doing deep dive into Kubernetes, and having experience with Minikube in VM on my laptop I wanted to step up the game and have my own setup on real hardware.

I have repurposed thin client and Installed Ubuntu 22.04.2 LTS on it. It's OS on bare metal, no Proxmox or anything.

Since it's running on metal and it's single device, single 1 master = worker my understanding was that no more than 1 IP address is needed.

I setup my config accordingly:

hosts.ini

[master]
192.168.100.205

# [node]
# 192.168.100.205

# only required if proxmox_lxc_configure: true
# must contain all proxmox instances that have a master or worker node
# [proxmox]
# 192.168.30.43

[k3s_cluster:children]
master
# node

all.yml

---
k3s_version: v1.25.9+k3s1
# this is the user that has ssh access to these machines
ansible_user: ansible
systemd_dir: /etc/systemd/system

# Set your timezone
system_timezone: "Your/Timezone"

# interface which will be used for flannel
flannel_iface: "enp4s0"

# apiserver_endpoint is virtual ip-address which will be configured on each master
apiserver_endpoint: "192.168.100.205"

# k3s_token is required  masters can talk together securely
# this token should be alpha numeric only
k3s_token: "some-SUPER-DEDEUPER-secret-password"

# The IP on which the node is reachable in the cluster.
# Here, a sensible default is provided, you can still override
# it for each of your hosts, though.
k3s_node_ip: '{{ ansible_facts[flannel_iface]["ipv4"]["address"] }}'

# Disable the taint manually by setting: k3s_master_taint = false
# k3s_master_taint: "{{ true if groups['node'] | default([]) | length >= 1 else false }}"
k3s_master_taint: "false"

# these arguments are recommended for servers as well as agents:
#extra_args: >-
#  --flannel-iface={{ flannel_iface }}
#  --node-ip={{ k3s_node_ip }}

extra_args: >-
  --node-ip={{ k3s_node_ip }}

# change these to your liking, the only required are: --disable servicelb, --tls-san {{ apiserver_endpoint }}
extra_server_args: >-
  {{ extra_args }}
  {{ '--node-taint node-role.kubernetes.io/master=true:NoSchedule' if k3s_master_taint else '' }}
  --tls-san {{ apiserver_endpoint }}
  --disable servicelb
  --disable traefik
extra_agent_args: >-
  {{ extra_args }}

# image tag for kube-vip
kube_vip_tag_version: "v0.5.12"

# metallb type frr or native
metal_lb_type: "native"

# metallb mode layer2 or bgp
metal_lb_mode: "layer2"

# bgp options
# metal_lb_bgp_my_asn: "64513"
# metal_lb_bgp_peer_asn: "64512"
# metal_lb_bgp_peer_address: "192.168.30.1"

# image tag for metal lb
metal_lb_speaker_tag_version: "v0.13.9"
metal_lb_controller_tag_version: "v0.13.9"

# metallb ip range for load balancer
metal_lb_ip_range: "192.168.100.205"

# Only enable if your nodes are proxmox LXC nodes, make sure to configure your proxmox nodes
# in your hosts.ini file.
# Please read https://gist.github.com/triangletodd/02f595cd4c0dc9aac5f7763ca2264185 before using this.
# Most notably, your containers must be privileged, and must not have nesting set to true.
# Please note this script disables most of the security of lxc containers, with the trade off being that lxc
# containers are significantly more resource efficent compared to full VMs.
# Mixing and matching VMs and lxc containers is not supported, ymmv if you want to do this.
# I would only really recommend using this if you have partiularly low powered proxmox nodes where the overhead of
# VMs would use a significant portion of your available resources.
proxmox_lxc_configure: false
# the user that you would use to ssh into the host, for example if you run ssh some-user@my-proxmox-host,
# set this value to some-user
proxmox_lxc_ssh_user: root
# the unique proxmox ids for all of the containers in the cluster, both worker and master nodes
proxmox_lxc_ct_ids:
  - 200
  - 201
  - 202
  - 203
  - 204

However playbook crashes on TASK [k3s_server_post : Wait for MetalLB resources]:

TASK [k3s_server_post : Wait for MetalLB resources] ***********************************************
failed: [192.168.100.205] (item=controller) => {"ansible_loop_var": "item", "changed": false, "cmd": ["k3s", "kubectl", "wait", "deployment", "--namespace=metallb-system", "controller", "--for", "condition=Available=True", "--timeout=120s"], "delta": "0:02:00.337885", "end": "2023-07-31 16:40:36.151095", "item": {"condition": "--for condition=Available=True", "description": "controller", "name": "controller", "resource": "deployment"}, "msg": "non-zero return code", "rc": 1, "start": "2023-07-31 16:38:35.813210", "stderr": "error: timed out waiting for the condition on deployments/controller", "stderr_lines": ["error: timed out waiting for the condition on deployments/controller"], "stdout": "", "stdout_lines": []}
failed: [192.168.100.205] (item=webhook service) => {"ansible_loop_var": "item", "changed": false, "cmd": ["k3s", "kubectl", "wait", "pod", "--namespace=metallb-system", "--selector=component=controller", "--for=jsonpath={.status.phase}=Running", "--timeout=120s"], "delta": "0:02:00.312018", "end": "2023-07-31 16:42:37.059840", "item": {"condition": "--for=jsonpath='{.status.phase}'=Running", "description": "webhook service", "resource": "pod", "selector": "component=controller"}, "msg": "non-zero return code", "rc": 1, "start": "2023-07-31 16:40:36.747822", "stderr": "error: timed out waiting for the condition on pods/controller-844979dcdc-h4f2l", "stderr_lines": ["error: timed out waiting for the condition on pods/controller-844979dcdc-h4f2l"], "stdout": "", "stdout_lines": []}
failed: [192.168.100.205] (item=pods in replica sets) => {"ansible_loop_var": "item", "changed": false, "cmd": ["k3s", "kubectl", "wait", "pod", "--namespace=metallb-system", "--selector=component=controller,app=metallb", "--for", "condition=Ready", "--timeout=120s"], "delta": "0:02:00.299969", "end": "2023-07-31 16:44:37.999494", "item": {"condition": "--for condition=Ready", "description": "pods in replica sets", "resource": "pod", "selector": "component=controller,app=metallb"}, "msg": "non-zero return code", "rc": 1, "start": "2023-07-31 16:42:37.699525", "stderr": "error: timed out waiting for the condition on pods/controller-844979dcdc-h4f2l", "stderr_lines": ["error: timed out waiting for the condition on pods/controller-844979dcdc-h4f2l"], "stdout": "", "stdout_lines": []}
failed: [192.168.100.205] (item=ready replicas of controller) => {"ansible_loop_var": "item", "changed": false, "cmd": ["k3s", "kubectl", "wait", "replicaset", "--namespace=metallb-system", "--selector=component=controller,app=metallb", "--for=jsonpath={.status.readyReplicas}=1", "--timeout=120s"], "delta": "0:00:00.361317", "end": "2023-07-31 16:44:39.013817", "item": {"condition": "--for=jsonpath='{.status.readyReplicas}'=1", "description": "ready replicas of controller", "resource": "replicaset", "selector": "component=controller,app=metallb"}, "msg": "non-zero return code", "rc": 1, "start": "2023-07-31 16:44:38.652500", "stderr": "error: readyReplicas is not found", "stderr_lines": ["error: readyReplicas is not found"], "stdout": "", "stdout_lines": []}
ok: [192.168.100.205] => (item=fully labeled replicas of controller)
failed: [192.168.100.205] (item=available replicas of controller) => {"ansible_loop_var": "item", "changed": false, "cmd": ["k3s", "kubectl", "wait", "replicaset", "--namespace=metallb-system", "--selector=component=controller,app=metallb", "--for=jsonpath={.status.availableReplicas}=1", "--timeout=120s"], "delta": "0:00:00.407084", "end": "2023-07-31 16:44:41.005532", "item": {"condition": "--for=jsonpath='{.status.availableReplicas}'=1", "description": "available replicas of controller", "resource": "replicaset", "selector": "component=controller,app=metallb"}, "msg": "non-zero return code", "rc": 1, "start": "2023-07-31 16:44:40.598448", "stderr": "error: availableReplicas is not found", "stderr_lines": ["error: availableReplicas is not found"], "stdout": "", "stdout_lines": []}

To be sure that even with error cluster is not working, I created an nginx pod, but it's constantly stuck on pending:

qbus@america:~$ sudo kubectl get pods --watch
[sudo] password for qbus:
Sorry, try again.
[sudo] password for qbus:
NAME    READY   STATUS    RESTARTS   AGE
nginx   0/1     Pending   0          5h25m

My question is - what do I need to make it work in my configuration? I really want this automated install, sure I could install it manually but that's not a solution.

I also have pretty weak understanding of all these technolgies glued together, so maybe there is some obvious thing I missed.

I checked other Discusson thread but couldn't glue together any solution. I also took some ideas from #290 (reply in thread) as I thought it's simmilar issue but it didn't help.

[mlanner]

@knuurr It looks as if you're using the same IP for your host, in your hosts.ini file, and then again in all.yml:

apiserver_endpoint: "192.168.100.205"

I think that's not right. You need to give the apiserver_endpoint another IP, not the same one as the host. This is a virtual IP the 'cluster' will use.

I could be wrong about that, but the error appears to point to MetalLB not liking your config.

[knuurr]

Thanks for reply. So IP in hosts.ini is obviously supposed to be reachable externally as in from my laptop within my home LAN network.

But is this API endpoint also supposed to be externally reachable? Or is it virtual network such as Docker networks are?

I don't know if my home router wil assign 2 IPs for single machine.

[mlanner]

Well, as far as I understand it, the apiserver_endpoint IP is a virtual IP (VIP) that "floats" on top of the assigned IPs of your control plane machines. (In your case only one.) Basically, it's intended to provide load balancing across the IPs of the server nodes. But, if you assign the same IP to the apiserver_endpoint as one of your nodes, you'll end up with an IP conflict. In your case, you only have one node, so the VIP is kind of pointless. If your single machine goes down, your "cluster" won't be reachable anyway, so it becomes sort of pointless. But, that said, if you just want it there, instead of trying to remove it, then you should use another IP for that.

Again, I might be wrong and perhaps someone else can chime in here to clarify in that case.