kube-proxy Talos Machine Configuration Not Being Passed To kube-proxy Daemon Set & Pods

[zombiemaker]

Software components:

• Talos Linux 1.10.0
• Kubernetes 1.32.3

I am trying to configure a 4-node Talos Linux / Kubernetes cluster to allow Kubernetes NodePort services to be accessible from each node's IP network interface for public traffic (10.0.1.128/25 subnet).

NodePort services are accessible from each node's IP network interface for private traffic (10.0.1.16/28 subnet). However, NodePort services are not accessible from the public network interfaces.

The solution from NodePort Service Only Accessible via one Interface #9577 does not seem to work when using either "0.0.0.0/0" or "10.0.1.128/25" for the parameter cluster.proxy.extraArgs.nodeport-addresses.

The Talos machine configuration parameter cluster.proxy.extraArgs.nodeport-addresses is not being passed to the kube-proxy pods. I am assuming that it is supposed to work that way.

Below are the running configurations:

• Talos Machine Configuration
• Talos Controlplane Manifest
• Kubernetes kube-proxy Daemon Set
• Kubernetes kube-proxy Pods

Any suggestions?

Talos Machine Configuration

In the Talos machine configuration for the controlplane node (node-1 with IPv4 address 10.0.1.18/28), the cluster.proxy.extraArgs.nodeport-addresses parameter is set to 10.0.1.128/25:

...
cluster:
  ...
  proxy:
      ...
      extraArgs:
        nodeport-addresses: "10.0.1.128/25"
...

Talos Controlplane Manifest

The kube-proxy configuration from the Talos machine configuration is passed to the Talos Controlplane manifest:

talosctl -n 10.0.1.18 get manifests
NODE        NAMESPACE      TYPE       ID                               VERSION
10.0.1.18   controlplane   Manifest   00-kubelet-bootstrapping-token   1
10.0.1.18   controlplane   Manifest   01-csr-approver-role-binding     1
10.0.1.18   controlplane   Manifest   01-csr-node-bootstrap            1
10.0.1.18   controlplane   Manifest   01-csr-renewal-role-binding      1
10.0.1.18   controlplane   Manifest   05-flannel                       1
10.0.1.18   controlplane   Manifest   10-kube-proxy                    1
10.0.1.18   controlplane   Manifest   11-core-dns                      1
10.0.1.18   controlplane   Manifest   11-core-dns-svc                  1
10.0.1.18   controlplane   Manifest   11-kube-config-in-cluster        1

talosctl -n 10.0.1.18 get manifests 10-kube-proxy -o yaml
node: 10.0.1.18
metadata:
    namespace: controlplane
    type: Manifests.kubernetes.talos.dev
    id: 10-kube-proxy
    version: 1
    owner: k8s.ManifestController
    phase: running
    created: 2025-05-27T21:14:13Z
    updated: 2025-05-27T21:14:13Z
spec:
    - apiVersion: apps/v1
      kind: DaemonSet
      metadata:
        labels:
            k8s-app: kube-proxy
            tier: node
        name: kube-proxy
        namespace: kube-system
      spec:
        selector:
            matchLabels:
                k8s-app: kube-proxy
                tier: node
        template:
            metadata:
                labels:
                    k8s-app: kube-proxy
                    tier: node
            spec:
                containers:
                    - command:
                        - /usr/local/bin/kube-proxy
                        - --cluster-cidr=10.244.0.0/16
                        - --conntrack-max-per-core=0
                        - --hostname-override=$(NODE_NAME)
                        - --kubeconfig=/etc/kubernetes/kubeconfig
                        - --nodeport-addresses=10.0.1.128/25
                        - --proxy-mode=nftables
                      env:
                        - name: NODE_NAME
                          valueFrom:
                            fieldRef:
                                fieldPath: spec.nodeName
                        - name: POD_IP
                          valueFrom:
                            fieldRef:
                                fieldPath: status.podIP
                      image: distribution-registry-1/kubernetes/kube-proxy:v1.32.3
                      name: kube-proxy
                      securityContext:
                        privileged: true
                      volumeMounts:
                        - mountPath: /lib/modules
                          name: lib-modules
                          readOnly: true
                        - mountPath: /etc/ssl/certs
                          name: ssl-certs-host
                          readOnly: true
                        - mountPath: /etc/kubernetes
                          name: kubeconfig
                          readOnly: true
                hostNetwork: true
                priorityClassName: system-cluster-critical
                serviceAccountName: kube-proxy
                tolerations:
                    - effect: NoSchedule
                      operator: Exists
                    - effect: NoExecute
                      operator: Exists
                volumes:
                    - hostPath:
                        path: /usr/lib/modules
                      name: lib-modules
                    - hostPath:
                        path: /etc/ssl/certs
                      name: ssl-certs-host
                    - configMap:
                        name: kubeconfig-in-cluster
                      name: kubeconfig
        updateStrategy:
            rollingUpdate:
                maxUnavailable: 1
            type: RollingUpdate
    - apiVersion: v1
      kind: ServiceAccount
      metadata:
        name: kube-proxy
        namespace: kube-system
    - apiVersion: rbac.authorization.k8s.io/v1
      kind: ClusterRoleBinding
      metadata:
        name: kube-proxy
      roleRef:
        apiGroup: rbac.authorization.k8s.io
        kind: ClusterRole
        name: system:node-proxier
      subjects:
        - kind: ServiceAccount
          name: kube-proxy
          namespace: kube-system

Kubernetes kube-proxy Daemon Set

However, the kube-proxy daemon set does NOT have the parameter "--nodeport-addresses":

kubectl get daemonset kube-proxy -n kube-system -o yaml

apiVersion: apps/v1
kind: DaemonSet
metadata:
  annotations:
    deprecated.daemonset.template.generation: "2"
  creationTimestamp: "2025-05-21T16:25:01Z"
  generation: 2
  labels:
    k8s-app: kube-proxy
    tier: node
  name: kube-proxy
  namespace: kube-system
  resourceVersion: "265646"
  uid: cc693d4f-76c7-4c76-b69b-e6bd7b397c54
spec:
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      k8s-app: kube-proxy
      tier: node
  template:
    metadata:
      annotations:
        kubectl.kubernetes.io/restartedAt: "2025-05-27T21:26:23.555Z"
      creationTimestamp: null
      labels:
        k8s-app: kube-proxy
        tier: node
    spec:
      containers:
      - command:
        - /usr/local/bin/kube-proxy
        - --cluster-cidr=10.244.0.0/16
        - --conntrack-max-per-core=0
        - --hostname-override=$(NODE_NAME)
        - --kubeconfig=/etc/kubernetes/kubeconfig
        - --proxy-mode=nftables
        env:
        - name: NODE_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: spec.nodeName
        - name: POD_IP
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: status.podIP
        image: distribution-registry-1/kubernetes/kube-proxy:v1.32.3
        imagePullPolicy: IfNotPresent
        name: kube-proxy
        resources: {}
        securityContext:
          privileged: true
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /lib/modules
          name: lib-modules
          readOnly: true
        - mountPath: /etc/ssl/certs
          name: ssl-certs-host
          readOnly: true
        - mountPath: /etc/kubernetes
          name: kubeconfig
          readOnly: true
      dnsPolicy: ClusterFirst
      hostNetwork: true
      priorityClassName: system-cluster-critical
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      serviceAccount: kube-proxy
      serviceAccountName: kube-proxy
      terminationGracePeriodSeconds: 30
      tolerations:
      - effect: NoSchedule
        operator: Exists
      - effect: NoExecute
        operator: Exists
      volumes:
      - hostPath:
          path: /lib/modules
          type: ""
        name: lib-modules
      - hostPath:
          path: /etc/ssl/certs
          type: ""
        name: ssl-certs-host
      - configMap:
          defaultMode: 420
          name: kubeconfig-in-cluster
        name: kubeconfig
  updateStrategy:
    rollingUpdate:
      maxSurge: 0
      maxUnavailable: 1
    type: RollingUpdate
status:
  currentNumberScheduled: 4
  desiredNumberScheduled: 4
  numberAvailable: 4
  numberMisscheduled: 0
  numberReady: 4
  observedGeneration: 2
  updatedNumberScheduled: 4

Kubernetes kube-proxy Pods

Subsequently, the kube-proxy pods do not have the "--nodeport-addresses" parameter:

kubectl get pods -n kube-system
NAME                                   READY   STATUS    RESTARTS       AGE
coredns-578d4f8ffc-fqtkt               1/1     Running   0              12h
coredns-578d4f8ffc-znpkc               1/1     Running   0              12h
kube-apiserver-supermicro-1            1/1     Running   0              3h3m
kube-controller-manager-supermicro-1   1/1     Running   1 (3h4m ago)   3h3m
kube-flannel-44qxb                     1/1     Running   0              12h
kube-flannel-8grn8                     1/1     Running   15 (12h ago)   7d
kube-flannel-c8lcz                     1/1     Running   0              12h
kube-flannel-dnkw7                     1/1     Running   0              12h
kube-proxy-6hcdn                       1/1     Running   0              12h
kube-proxy-f44fd                       1/1     Running   3 (12h ago)    19h
kube-proxy-nl8rs                       1/1     Running   0              12h
kube-proxy-qf69r                       1/1     Running   0              12h
kube-scheduler-supermicro-1            1/1     Running   1 (3h4m ago)   3h3m

kubectl get pods kube-proxy-6hcdn -n kube-system -o yaml
apiVersion: v1
kind: Pod
metadata:
  annotations:
    kubectl.kubernetes.io/restartedAt: "2025-05-27T21:26:23.555Z"
  creationTimestamp: "2025-05-28T04:55:00Z"
  generateName: kube-proxy-
  labels:
    controller-revision-hash: 579c955546
    k8s-app: kube-proxy
    pod-template-generation: "2"
    tier: node
  name: kube-proxy-6hcdn
  namespace: kube-system
  ownerReferences:
  - apiVersion: apps/v1
    blockOwnerDeletion: true
    controller: true
    kind: DaemonSet
    name: kube-proxy
    uid: cc693d4f-76c7-4c76-b69b-e6bd7b397c54
  resourceVersion: "265514"
  uid: 8200d260-2d24-425e-ad77-6fa8e70f32d3
spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchFields:
          - key: metadata.name
            operator: In
            values:
            - supermicro-4
  containers:
  - command:
    - /usr/local/bin/kube-proxy
    - --cluster-cidr=10.244.0.0/16
    - --conntrack-max-per-core=0
    - --hostname-override=$(NODE_NAME)
    - --kubeconfig=/etc/kubernetes/kubeconfig
    - --proxy-mode=nftables
    env:
    - name: NODE_NAME
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: spec.nodeName
    - name: POD_IP
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: status.podIP
    image: distribution-registry-1/kubernetes/kube-proxy:v1.32.3
    imagePullPolicy: IfNotPresent
    name: kube-proxy
    resources: {}
    securityContext:
      privileged: true
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /lib/modules
      name: lib-modules
      readOnly: true
    - mountPath: /etc/ssl/certs
      name: ssl-certs-host
      readOnly: true
    - mountPath: /etc/kubernetes
      name: kubeconfig
      readOnly: true
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-82b4p
      readOnly: true
  dnsPolicy: ClusterFirst
  enableServiceLinks: true
  hostNetwork: true
  nodeName: supermicro-4
  preemptionPolicy: PreemptLowerPriority
  priority: 2000000000
  priorityClassName: system-cluster-critical
  restartPolicy: Always
  schedulerName: default-scheduler
  securityContext: {}
  serviceAccount: kube-proxy
  serviceAccountName: kube-proxy
  terminationGracePeriodSeconds: 30
  tolerations:
  - effect: NoSchedule
    operator: Exists
  - effect: NoExecute
    operator: Exists
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
  - effect: NoSchedule
    key: node.kubernetes.io/disk-pressure
    operator: Exists
  - effect: NoSchedule
    key: node.kubernetes.io/memory-pressure
    operator: Exists
  - effect: NoSchedule
    key: node.kubernetes.io/pid-pressure
    operator: Exists
  - effect: NoSchedule
    key: node.kubernetes.io/unschedulable
    operator: Exists
  - effect: NoSchedule
    key: node.kubernetes.io/network-unavailable
    operator: Exists
  volumes:
  - hostPath:
      path: /lib/modules
      type: ""
    name: lib-modules
  - hostPath:
      path: /etc/ssl/certs
      type: ""
    name: ssl-certs-host
  - configMap:
      defaultMode: 420
      name: kubeconfig-in-cluster
    name: kubeconfig
  - name: kube-api-access-82b4p
    projected:
      defaultMode: 420
      sources:
      - serviceAccountToken:
          expirationSeconds: 3607
          path: token
      - configMap:
          items:
          - key: ca.crt
            path: ca.crt
          name: kube-root-ca.crt
      - downwardAPI:
          items:
          - fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
            path: namespace
status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2025-05-28T14:07:30Z"
    status: "True"
    type: PodReadyToStartContainers
  - lastProbeTime: null
    lastTransitionTime: "2025-05-28T14:07:29Z"
    status: "True"
    type: Initialized
  - lastProbeTime: null
    lastTransitionTime: "2025-05-28T14:07:30Z"
    status: "True"
    type: Ready
  - lastProbeTime: null
    lastTransitionTime: "2025-05-28T14:07:30Z"
    status: "True"
    type: ContainersReady
  - lastProbeTime: null
    lastTransitionTime: "2025-05-28T04:55:00Z"
    status: "True"
    type: PodScheduled
  containerStatuses:
  - containerID: containerd://4b231c26554355ccdc1c46d5a9e413f823afbcd5ca8f53ddcc094edcf4a06027
    image: distribution-registry-1/kubernetes/kube-proxy:v1.32.3
    imageID: distribution-registry-1/kubernetes/kube-proxy@sha256:39486fd6cf7a25d57ee740c44b6db7d735294e626f1cc6fa7c1c1e292ed13d8a
    lastState: {}
    name: kube-proxy
    ready: true
    restartCount: 0
    started: true
    state:
      running:
        startedAt: "2025-05-28T14:07:30Z"
    volumeMounts:
    - mountPath: /lib/modules
      name: lib-modules
      readOnly: true
      recursiveReadOnly: Disabled
    - mountPath: /etc/ssl/certs
      name: ssl-certs-host
      readOnly: true
      recursiveReadOnly: Disabled
    - mountPath: /etc/kubernetes
      name: kubeconfig
      readOnly: true
      recursiveReadOnly: Disabled
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-82b4p
      readOnly: true
      recursiveReadOnly: Disabled
  hostIP: 10.0.1.21
  hostIPs:
  - ip: 10.0.1.21
  phase: Running
  podIP: 10.0.1.21
  podIPs:
  - ip: 10.0.1.21
  qosClass: BestEffort
  startTime: "2025-05-28T14:07:29Z"

[smira]

This might be not so good UX, but it works as designed here.

Talos machines never automatically modify or delete Kubernetes manifests (as it's not safe to do so given that there are multiple control plane machines which might disagree on the expected state).

These bootstrap manifests are automatically applied when it results in Kubernetes resource being created, so if you made your changes before cluster is bootstrapped, they would appear as expected.

In the current state, you can do a no-op Kubernetes upgrade via talosctl upgrade-k8s --to=1.32.3 to get these manifests synced.

[zombiemaker]

@smira: Thanks for responding to my question!

So after bootstrapping a Talos cluster, changes in a Talos machine config that affect any components that are configured by Talos bootstrap manifests require a "talosctl upgrade-k8s" in order to be applied?

If that is true, would extracting the values from the .spec key from the bootstrap manifests and using "kubectl apply" have the same results?

[smira]

So after bootstrapping a Talos cluster, changes in a Talos machine config that affect any components that are configured by Talos bootstrap manifests require a "talosctl upgrade-k8s" in order to be applied?

Please see above, it depends on the change - new manifests are applied immediately, updated manifests require talosctl upgrade-k8s

If that is true, would extracting the values from the .spec key from the bootstrap manifests and using "kubectl apply" have the same results?

Yes, it's in the docs