How To: Talos on CAPMOX

[tamcore]

Not exactly an idea, but there's no other option to choose from :)

Since Talos is a recurring topic, i figured, although it's not officially supported here, sharing some manifests with the community might be a good idea.

To note: I had issues getting any sort of extensions working, when using the .machine.install section of the talos machine configuration (which is deprecated anyways: https://www.talos.dev/v1.9/talos-guides/configuration/system-extensions/). So the only reliable option to get the qemu-guest-agent up and running was to have it baked into the image right from the beginning (which is preferable anyways IMHO)

As the first step, we have to download the nocloud image with qemu-guest-agent baked in and register it as a VM template in Proxmox.

The url for the image is the Disk Image referenced on https://factory.talos.dev/?arch=amd64&cmdline-set=true&extensions=-&extensions=siderolabs%2Fqemu-guest-agent&platform=nocloud&target=cloud&version=1.9.3

# Update the variables to your liking
VMID=10001
VMNAME=talos-nocloud-qemu-v1.9.3
VMTAGS="capi,talos,v1.9.3,qemu"
STORAGE=tank

# Don't change anything bellow this line
wget https://factory.talos.dev/image/ce4c980550dd2ab1b17bbf2b08801c7eb59418eafe8f279833297925d67c7515/v1.9.3/nocloud-amd64.raw.xz

xz -d nocloud-amd64.raw.xz

qm create $VMID --name $VMNAME --tags "$VMTAGS"
qm importdisk $VMID nocloud-amd64.raw "$STORAGE"
qm set $VMID \
    --scsihw virtio-scsi-single  \
    --scsi0 $STORAGE:vm-$VMID-disk-0,discard=on,iothread=1,ssd=1  \
    --boot order=scsi0  \
    --cpu host  \
    --rng0 source=/dev/urandom  \
    --template 1  \
    --agent 1  \
    --onboot 1

After the VM is registered, we can start creating the cluster. In the manifests bellow, pay extra care to parameters like allowedNodes, controlPlaneEndpoint, dnsServers, ipv4Config as well as vip in the TalosControlPlane, since you have to adjust those to match your setup.

---
apiVersion: infrastructure.cluster.x-k8s.io/v1alpha1
kind: ProxmoxCluster
metadata:
  name: talos
  namespace: cluster-api
spec:
  allowedNodes:
  - pve
  controlPlaneEndpoint:
    host: 192.168.69.80
    port: 6443
  dnsServers:
  - 8.8.8.8
  - 8.8.4.4
  ipv4Config:
    addresses:
    - 192.168.69.81-192.168.69.89
    gateway: 192.168.69.1
    prefix: 24
---
apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
  name: talos
  namespace: cluster-api
spec:
  controlPlaneRef:
    apiVersion: controlplane.cluster.x-k8s.io/v1alpha3
    kind: TalosControlPlane
    name: talos-control-plane
    namespace: cluster-api
  infrastructureRef:
    apiVersion: infrastructure.cluster.x-k8s.io/v1alpha1
    kind: ProxmoxCluster
    name: talos
    namespace: cluster-api
---
apiVersion: controlplane.cluster.x-k8s.io/v1alpha3
kind: TalosControlPlane
metadata:
  name: talos-control-plane
  namespace: cluster-api
spec:
  version: v1.32.1
  replicas: 3
  infrastructureTemplate:
    apiVersion: infrastructure.cluster.x-k8s.io/v1alpha1
    kind: ProxmoxMachineTemplate
    name: talos-control-plane
    namespace: cluster-api
  controlPlaneConfig:
    controlplane:
      generateType: controlplane
      strategicPatches:
        - |
          machine:
            install:
              disk: /dev/sda
              extraKernelArgs:
                - net.ifnames=0
            network:
              interfaces:
                - interface: eth0
                  dhcp: false
                  vip:
                    ip: 192.168.69.80
            kubelet:
              extraArgs:
                cloud-provider: external
            features:
              kubernetesTalosAPIAccess:
                enabled: true
                allowedRoles:
                  - os:reader
                allowedKubernetesNamespaces:
                  - kube-system
          cluster:
            externalCloudProvider:
              enabled: true
              manifests:
                - https://raw.githubusercontent.com/siderolabs/talos-cloud-controller-manager/main/docs/deploy/cloud-controller-manager.yml
---
apiVersion: infrastructure.cluster.x-k8s.io/v1alpha1
kind: ProxmoxMachineTemplate
metadata:
  name: talos-control-plane
  namespace: cluster-api
spec:
  template:
    spec:
      disks:
        bootVolume:
          disk: scsi0
          sizeGb: 20
      format: qcow2
      full: true
      memoryMiB: 4096
      network:
        default:
          bridge: vmbr0
          model: virtio
      numCores: 1
      numSockets: 1
      sourceNode: pve
      templateID: 10001
      checks:
        skipCloudInitStatus: true
      metadataSettings:
        providerIDInjection: true

After deploying the above manifests, it takes about 4-5 minutes to spin up 3 healthy Talos control-plane nodes with Talos CCM working (so spec.providerID is properly set on every node)

Let's add some worker nodes

---
apiVersion: cluster.x-k8s.io/v1beta1
kind: MachineDeployment
metadata:
  name: talos-worker
  namespace: cluster-api
spec:
  clusterName: talos
  replicas: 1
  selector:
    matchLabels: null
  template:
    spec:
      bootstrap:
        configRef:
          apiVersion: bootstrap.cluster.x-k8s.io/v1alpha3
          kind: TalosConfigTemplate
          name: talos-worker
      clusterName: talos
      infrastructureRef:
        apiVersion: infrastructure.cluster.x-k8s.io/v1alpha1
        kind: ProxmoxMachineTemplate
        name: talos-worker
      version: v1.32.1
---
apiVersion: infrastructure.cluster.x-k8s.io/v1alpha1
kind: ProxmoxMachineTemplate
metadata:
  name: talos-worker
  namespace: cluster-api
spec:
  template:
    spec:
      disks:
        bootVolume:
          disk: scsi0
          sizeGb: 20
      format: qcow2
      full: true
      memoryMiB: 2048
      network:
        default:
          bridge: vmbr0
          model: virtio
      numCores: 1
      numSockets: 1
      sourceNode: pve
      templateID: 10001
      checks:
        skipCloudInitStatus: true
      metadataSettings:
        providerIDInjection: true
---
apiVersion: bootstrap.cluster.x-k8s.io/v1alpha3
kind: TalosConfigTemplate
metadata:
  name: talos-worker
  namespace: cluster-api
spec:
  template:
    spec:
      generateType: worker
      strategicPatches:
        - |
          machine:
            install:
              disk: /dev/sda
              extraKernelArgs:
                - net.ifnames=0
            network:
              interfaces:
                - interface: eth0
                  dhcp: false
            kubelet:
              extraArgs:
                cloud-provider: external

[iverberk]

@tamcore Thanks for this and a very good idea to move this to a discussion from #400 . I installed it the same way you did I believe. I will try this playground later as well, to see if that works better. It looks pretty much the same as what I'm doing but there are some minor differences. Hopefully it'll work. You also didn't explicitly disable the node-lifecycle-controller, right?

[tamcore]

In the referenced manifest it's not enabled, so no need to worry about it.

That's where my question about your installation method comes from. Since when using Helm it could be seen as logical to enable it, as it's also mentioned in the CCM's documentation.

[glitchcrab]

This is a useful discussion as I'm currently working through setting this up myself. I'm actually doing everything with Flux from a bootstrap cluster. I intend documenting everything on my blog when I have time to write it up.

[tamcore]

I've used ArgoCD. But manifests are manifests.. :D

I assume you're using the cluster-api-operator as well?

[glitchcrab]

Yep, as much as possible is operator driven. The only thing I haven't automated is the pivot when I promote the first cluster to being the management cluster.

[iverberk]

I'm still testing some things but the Talos CCM definitely seems to mess up my etcd cluster for some reason. Without the TCCM I get a healthy etcd cluster but obviously the providerID is missing from the Kubernetes nodes. With the TCCM two of my etcd members enter a restart loop with the aforementioned message about the member having been permanently removed and that the data directory needs removal.

I needed to make a few adjustments to your commands:

xz -d nocloud-amd64.raw.xz
qm set $VMID --scsihw virtio-scsi-single --scsi0 $STORAGE:$VMID/vm-$VMID-disk-0.raw,discard=on,iothread=1,ssd=1 --boot order=scsi0 --cpu host --rng0 source=/dev/urandom --template 1 --agent 1 --onboot 1

I'm still using the clusterctl commands because I want to get a stable situation before I start using the operators. I need to find out what TCCM is doing to my nodes while bootstrapping the control plane.

[tamcore]

@iverberk how does your Proxmox setup look like? I think I might've encountered similar issues, when I was still using ZFS on my single-node homelab.

[iverberk]

I'm also running a single-node Proxmox on an OVHcloud server. It's relatively cheap server so I'm working with 2×2 TB HDD SATA Soft RAID disks. I have a single 'directory' type storage. This server really is just for experimenting. I haven't seen the cloning error anymore though and everything seems to be working as expected, including workers.

[tamcore]

Could it maybe be a concurrency thing?

[iverberk]

Yeah that could well be, it's not a production ready setup that I'm working with. But things seem stable right now. Next up is automating this with operators instead of clusterctl commands, although I'm not sure how to do the pivot from bootstrap to mgmt cluster yet (clusterctl move).

[glitchcrab]

Yeah that could well be, it's not a production ready setup that I'm working with. But things seem stable right now. Next up is automating this with operators instead of clusterctl commands, although I'm not sure how to do the pivot from bootstrap to mgmt cluster yet (clusterctl move).

I would love to automate this with my gitops setup, however I think it's probably a bad idea to do this because CAPI dynamically creates a bunch of different resources which aren't going to be managed by Flux/Argo. I automate the deployment up to a certain point, pause it, manually pivot and then resume Flux reconcilliation. It's not like you'll be pivoting from bootstrap to MC often once the MC is running.

[dhess]

@iverberk Did you ever figure out what's causing the etcd issues? I'm having exactly the same problem, and I'm also running the Talos CCM, though I'm not yet convinced that's the root cause.

[lieberlois]

@tamcore In the Proxmox UI, do you see the IP of the node? For me, it shows Agent not running with the same config 🤔