Talos cannot read some Kubernetes APIs after K8s v1.32.0

[buroa]

Bug Report

Talos cannot read some Kubernetes APIs after K8s v1.32.0.

Logs

[talos] kubernetes registry node watch error {"component": "controller-runtime", "controller": "cluster.KubernetesPullController", "error": "failed to list *v1.Node: nodes is forbidden: User \"system:node:m0\" cannot list resource \"nodes\" in API group \"\" at the cluster scope: node 'm0' cannot read all nodes, only its own Node object"}

Environment

• Talos version: v1.9.0
• Kubernetes version: v1.32.0
• Platform: metal

[smira]

I guess default Kubernetes RBAC changed (and the changes makes sense in general), while Talos re-uses kubelet's client configuration to access API.

So probably:

• don't use Kubernetes discovery, use Discovery Service which is better in every way (except for air-gapped environments)
• there might be a documented workaround to adjust RBAC to restore functionality.

[smira]

So this issue is caused by Kubernetes 1.32.0 enabling by default a feature gate AuthorizeNodeWithSelectors which is implemented in kubernetes/kubernetes#125571, specifically https://github.com/kubernetes/enhancements/tree/master/keps/sig-auth/4601-authorize-with-selectors#node-authorizer-changes.

Due to the nature of the Kubernetes discovery in Talos, it can't rely on other mechanisms to bootstrap itself, so it picks up the kubelet's client kubeconfig and tries to use it to watch the Node resource. This was allowed by the system:node ClusterRole previously, but with AuthorizeNodeWithSelectors NodeRestriction plugin implements more tight permission to check to allow the system:node:<foo> to only see its own Node <foo> (which makes perfect sense from a security point of view).

The workaround for now is to apply the following config patch (Kubernetes 1.32+ only) to all controlplane nodes:

cluster:
  apiServer:
    extraArgs:
      feature-gates: AuthorizeNodeWithSelectors=false

But this change is not recommended, as this disables other checks/security enhancements as well.

So all in all, the workaround will be documented, but the Kubernetes discovery is likely to be going away in Talos.

[jfroy]

Will SideroLabs allow non-commercial deployments of discovery-service to replace Kubernetes discovery for people/organizations that do not want an external dependency or are airgapped?

[steverfrancis]

Discovery service is under BUSL, so non-commercial, non production deployment is explicitly allowed.

[maulerjan]

Hi, I was previously using Kubernetes discovery. After facing the issues described here, I switched to Discovery service by updating the machineconfigs of all nodes in "auto" mode. Changes were applied without reboot. But I still see these logs on every node in the cluster:

10.8.66.3: user: warning: [2025-01-18T11:42:09.71881442Z]: [talos] kubernetes registry node watch error {"component": "controller-runtime", "controller": "cluster.KubernetesPullController", "error": "failed to list *v1.Node: nodes is forbidden: User \"system:node:kvbt-prod-k8s-main-worker-generic-3\" cannot list resource \"nodes\" in API group \"\" at the cluster scope: node 'kvbt-prod-k8s-main-worker-generic-3' cannot read all nodes, only its own Node object"}

I know for a fact that config changes were applied, because now I can see all the members correctly again when running talosctl -n ... get members. But for some reason, these logs keeps appearing (every 40 seconds or so, on every node).

Also, I was joining a new node to the cluster today. While I can see the node in Talos members list, the node is not joining the kubernetes cluster (it's not listed in kubectl get nodes).

Logs from the new member differ from those of existing members:

10.8.68.3: user: warning: [2025-01-18T11:50:45.88177631Z]: [talos] controller failed {"component": "controller-runtime", "controller": "k8s.NodeApplyController", "error": "1 error(s) occurred:\n\terror getting node: nodes \"kvbt-prod-k8s-main-worker-ethereum-2\" not found"}

Is there something else required to make Talos fully switch to Discovery service? Or maybe these 2 issues are unrelated?

EDIT: For the newly added node, machinestatus says "status is not available yet". Also, the node has been installed with Talos 1.9.2, while the rest of the cluster still runs on 1.9.0. I was planning to upgrade the rest of the nodes over the weekend.

[onedr0p]

Changes were applied without reboot

I had the same problem and discovered a reboot was indeed required to change this option.

[maulerjan]

I had the same problem and discovered a reboot was required to change this option.

Just found out myself like 10 minutes ago! 😅 Seems like the "auto" mode made a wrong call, but that's totally understandable.

Problem solved. The second issue persists, but now I'm pretty sure it's not related, so I won't pollute this thread with it.

Thanks for confirming that a reboot was necessary.

[aossama]

Just upgraded my cluster couple of hours ago from v1.30 to v1.32 and came across this issue.

I created a custom ClusterRoleBinding for my nodes to bind to ClusterRole/system:node

kubectl create clusterrolebinding manual-system-node-crb \
  --user system:node:k8s-worker-01 \
  --user system:node:k8s-worker-02 \
  --user system:node:k8s-worker-03 \
  --clusterrole system:node

[smira]

I created a custom ClusterRoleBinding for my nodes to bind to ClusterRole/system:node

We (Sidero Labs) would not recommend to break Kubernetes security constraints this way.

[fenio]

So what exactly needs to be done to make Talos happy?
I removed any references to discovery options in my talconfig and I'm getting errors about discovery.talos.dev.
The only way to use Talos reliably and not drop security constrains is to make discovery via Talos service?

[jfroy]

So what exactly needs to be done to make Talos happy? I removed any references to discovery options in my talconfig and I'm getting errors about discovery.talos.dev. The only way to use Talos reliably and not drop security constrains is to make discovery via Talos service?

I've deployed the discovery service locally in my network and it's been fine. Very easy to do.

[onedr0p]

I removed any references to discovery options in my talconfig and I'm getting errors about discovery.talos.dev.

So you applied the config and rebooted the nodes as discussed earlier in this issue?

[fenio]

@onedr0p yep
I removed:

    # Enable cluster discovery
    - |-
      cluster:
        discovery:
          registries:
            kubernetes:
              disabled: false
            service:
              disabled: true

Regenerated talconfig, applied it, rebooted all nodes and I ended up with this:

[smira]

So what exactly needs to be done to make Talos happy? I removed any references to discovery options in my talconfig and I'm getting errors about discovery.talos.dev. The only way to use Talos reliably and not drop security constrains is to make discovery via Talos service?

So not sure what kind of security constraints you mean here.

If the discovery service doesn't work for you, check if you can simply reach https://discovery.talos.dev/.

[fenio]

I can reach it flawlessly and I'm still getting errors wrt handshake.

[andrenth]

Does this mean usage of Talos in production in air gapped environments is only possible via a commercial license? This makes Talos unsuitable for us, and we have to start looking for open alternatives.

[adampetrovic]

If you're using Talos in a commercial production environment, then yes you will need a commercial license to run your own Service Registry (BSL 1.1)

BSL 1.1 shouldn't cover personal / testing uses AFAIK

[steverfrancis]

To pedantically clarify - you don't need a commercial license to run Talos in production - but you do for the Discovery Service.

[adampetrovic]

To pedantically clarify - you don't need a commercial license to run Talos in production - but you do for the Discovery Service.

Thanks. I should have been more specific. Updated :)

[baldugus]

Perhaps this information should be in https://www.talos.dev/v1.9/advanced/air-gapped/ as there is no mention of acquiring the Discovery Service license for air gapped environments.

Do you think this is worth an Issue/PR for the docs?

[frezbo]

Perhaps this information should be in https://www.talos.dev/v1.9/advanced/air-gapped/ as there is no mention of acquiring the Discovery Service license for air gapped environments.

Do you think this is worth an Issue/PR for the docs?

i guess it would be nice, but discovery service is not a requirement to run talos, with all discovery disabled Talos would still work fine

[JJGadgets]

Perhaps this information should be in https://www.talos.dev/v1.9/advanced/air-gapped/ as there is no mention of acquiring the Discovery Service license for air gapped environments.

Do you think this is worth an Issue/PR for the docs?

i guess it would be nice, but discovery service is not a requirement to run talos, with all discovery disabled Talos would still work fine

Discovery is used for KubePrism, KubeSpan, talosctl get members and the number of machines shown on the TTY dashboard, is that right?

[smira]

Discovery is used for KubePrism, KubeSpan, talosctl get members and the number of machines shown on the TTY dashboard, is that right?

yes, it's a hard requirement for KubeSpan, but other components are optional.

Talos works better with Discovery, but it would still work without it.

[outbackdingo]

Discovery is used for KubePrism, KubeSpan, talosctl get members and the number of machines shown on the TTY dashboard, is that right?

yes, it's a hard requirement for KubeSpan, but other components are optional.

Talos works better with Discovery, but it would still work without it.

So basically now I am hearing we wount be able to use Kubespan / Kubeprism without a commercial license

And can I ask what makes talos run "better" with Discovery, because i get the feeling, dropping this feature now screws alot of us, and IMHO we seem to be loosing ground, not gaining it, why not just fix the issue, instead of just dropping it

[steverfrancis]

No, you can still use Kubespan / Kubeprism and all of Talos free, in production, without a license, by using our free public Discovery Service.

The only constraint is if you want to run air-gapped, or for some other reason need to run your own Discovery service - that requires a license.

Running "better" with Discovery means Kubeprism works, and talosctl health works without necessarily providing all node members explicitly, etc.

And we cannot fix the Kubernetes based discovery, as it's a change (for the better) for Kubernetes default security.
As Andrey said, it is probably possible to explicitly weaken your Kubernetes RBAC to still allow the Kubernetes based discovery service to work, but as we are always advocating and implementing more security, that is not something we are going to recommend.

[outbackdingo]

No, you can still use Kubespan / Kubeprism and all of Talos free, in production, without a license, by using our free public Discovery Service.

The only constraint is if you want to run air-gapped, or for some other reason need to run your own Discovery service - that requires a license.

Which basically shoots this whole project in the lab in the foot, policies do not allow using external services or dependencies for cluster / devops / security infrastructure, which now means if i move this POC forward i also need a license and have to deploy discovery locally. Maybe its just time to drop talos in light of this, i had hopes but, not sure this is going to fly, we arent a fortune 500 that can afford huge licensing fees for a simple feature.

I know see 8 months of work prototyping in the lab completely wasted.

[onedr0p]

With the changes to Kubernetes API server, it would be nice if Sidero could reconsider the license on the discovery service since it doesn't seem like there is a workaround (at least a secure one), or in the very least give more documentation around running Talos in an airgapped environment with the nuances discussed here summarized.

I know Sidero has to pay employees and turn profit however I am only suggesting this due to the changes to the API server and the concerns people have brought up in this issue which were not present when the discovery service was first released & licensed.

[steverfrancis]

Use the public discovery service. The data transiting it is all encrypted and unreadable by the service.

…

On Fri, Mar 7, 2025 at 5:24 PM Devin Buhl ***@***.***> wrote: With the changes to Kubernetes API server, it would be nice if the discovery service was re-licensed under AGPL / MPL / MIT / Apache or whatever since it doesn't seem like there is a workaround (at least a secure one). — Reply to this email directly, view it on GitHub <#9980 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AQGWG5MO5PPYJUHSRK4QKOL2TJBE7AVCNFSM6AAAAABTY37YOCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDOMBXHA3DGMJVGQ> . You are receiving this because you commented.Message ID: ***@***.***> [image: onedr0p]*onedr0p* left a comment (siderolabs/talos#9980) <#9980 (comment)> With the changes to Kubernetes API server, it would be nice if the discovery service was re-licensed under AGPL / MPL / MIT / Apache or whatever since it doesn't seem like there is a workaround (at least a secure one). — Reply to this email directly, view it on GitHub <#9980 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AQGWG5MO5PPYJUHSRK4QKOL2TJBE7AVCNFSM6AAAAABTY37YOCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDOMBXHA3DGMJVGQ> . You are receiving this because you commented.Message ID: ***@***.***>

[steverfrancis]

Luckily we don’t charge huge licensing fees. :-) But we do like to pay our engineers who mostly build open source stuff that most people and companies pay nothing for.

…

On Fri, Mar 7, 2025 at 5:05 PM outbackdingo ***@***.***> wrote: No, you can still use Kubespan / Kubeprism and all of Talos free, in production, without a license, by using our free public Discovery Service. The only constraint is if you want to run air-gapped, or for some other reason need to run your own Discovery service - that requires a license. Which basically shoots this whole project in the lab in the foot, policies do not allow using external services or dependencies for cluster / devops / security infrastructure, which now means if i move this POC forward i also need a license and have to deploy discovery locally. Maybe its just time to drop talos in light of this, i had hopes but, not sure this is going to fly, we arent a fortune 500 that can afford huge licensing fees for a simple feature. — Reply to this email directly, view it on GitHub <#9980 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AQGWG5PZCJAUKWUTL4KDLQ32TI64LAVCNFSM6AAAAABTY37YOCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDOMBXHAZDIMRXHE> . You are receiving this because you commented.Message ID: ***@***.***> [image: outbackdingo]*outbackdingo* left a comment (siderolabs/talos#9980) <#9980 (comment)> No, you can still use Kubespan / Kubeprism and all of Talos free, in production, without a license, by using our free public Discovery Service. The only constraint is if you want to run air-gapped, or for some other reason need to run your own Discovery service - that requires a license. Which basically shoots this whole project in the lab in the foot, policies do not allow using external services or dependencies for cluster / devops / security infrastructure, which now means if i move this POC forward i also need a license and have to deploy discovery locally. Maybe its just time to drop talos in light of this, i had hopes but, not sure this is going to fly, we arent a fortune 500 that can afford huge licensing fees for a simple feature. — Reply to this email directly, view it on GitHub <#9980 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AQGWG5PZCJAUKWUTL4KDLQ32TI64LAVCNFSM6AAAAABTY37YOCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDOMBXHAZDIMRXHE> . You are receiving this because you commented.Message ID: ***@***.***>

[onedr0p]

For those using Cilium I posted a way to disable service discovery (and optionally kubeprism) in the following discussion

#11513