Removing fields from template doesn't remove it in the resulting resource

[cwrau]

What steps did you take and what happened?

We have a (kubeadmcontrolplane) template:

apiVersion: controlplane.cluster.x-k8s.io/v1beta1
kind: KubeadmControlPlaneTemplate
spec:
  template:
    initConfiguration:
      nodeRegistration:
        imagePullPolicy: IfNotPresent
        kubeletExtraArgs:
          cloud-provider: external
        name: '{{ local_hostname }}'
      patches:
        directory: /etc/kubernetes/patches
    joinConfiguration:
      nodeRegistration:
        imagePullPolicy: IfNotPresent
        kubeletExtraArgs:
          cloud-provider: external
        name: '{{ local_hostname }}'

before the change it was like this:

apiVersion: controlplane.cluster.x-k8s.io/v1beta1
kind: KubeadmControlPlaneTemplate
spec:
  template:
    initConfiguration:
      localAPIEndpoint: {}
      nodeRegistration:
        imagePullPolicy: IfNotPresent
        kubeletExtraArgs:
          cloud-provider: external
          event-qps: "0"
          feature-gates: SeccompDefault=true
          protect-kernel-defaults: "true"
          seccomp-default: "true"
          tls-cipher-suites: TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256
        name: '{{ local_hostname }}'
      patches:
        directory: /etc/kubernetes/patches
    joinConfiguration:
      discovery: {}
      nodeRegistration:
        imagePullPolicy: IfNotPresent
        kubeletExtraArgs:
          cloud-provider: external
          event-qps: "0"
          feature-gates: SeccompDefault=true
          protect-kernel-defaults: "true"
          seccomp-default: "true"
          tls-cipher-suites: TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256
        name: '{{ local_hostname }}'

as you can see we removed all kubeletExtraArgs aside from cloud-provider.

But the resource, in this case kubeadmcontrolplane, still has these fields set, I assume because of a merge apply instead of completely overriding the resource.

This (might) result in a broken cluster which needs manual intervention if the removed fields are not valid anymore, which is the case for a few of these args.

What did you expect to happen?

That the resource (kubeadmcontrolplane) would be the exact result of the template instead of a merged mess.

Cluster API version

1.8.5

Kubernetes version

1.28.15

Anything else you would like to add?

No response

Label(s) to be applied

/kind bug
/area clusterclass

[chrischdi]

How did you apply it? Can you post it in a way we could reproduce it?

Does the behaviour change if you use kubectl apply --server-side=true (on create and on update)

[sbueringer]

We have a (kubeadmcontrolplane) template:

The first YAML shows a KubeadmControlPlane not a KubeadmControlPlaneTemplate

But the resource, in this case kubeadmcontrolplane, still has these fields set, I assume because of a merge apply instead of completely overriding the resource.

We use SSA in the Cluster topology controller. I would have assumed it works

[cwrau]

How did you apply it? Can you post it in a way we could reproduce it?

I apply the cluster-api templates with helm via a flux HelmRelease. Source is https://github.com/teutonet/teutonet-helm-charts/tree/main/charts/t8s-cluster. This is a bit "tricky" to reproduce, as CAPO needs credentials which aren't managed by the cluster-api ecosystem. We have our own operator that fills this gap.

First values:

bastion:
  enabled: false
cloud: ffm3-prod
controlPlane:
  flavor: standard.4.1905
metadata:
  customerID: 1111
  customerName: teuto-net
  serviceLevelAgreement: 24x7
nodePools:
  s2-default:
    availabilityZone: Zone2
    flavor: standard.2.1905
    replicas: 4
version:
  major: 1
  minor: 28
  patch: 15

second values:

bastion:
  enabled: false
cloud: ffm3-prod
controlPlane:
  flavor: standard.4.1905
metadata:
  customerID: 1111
  customerName: teuto-net
nodePools:
  s2-default:
    availabilityZone: Zone2
    flavor: standard.2.1905
    replicas: 4
version:
  major: 1
  minor: 29
  patch: 13

But in principle creating these resources and providing a cluster-api cluster should result in the same problem.

first cluster.yaml:

apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
  name: mgmt-prod-cluster
spec:
  controlPlaneRef:
    apiVersion: controlplane.cluster.x-k8s.io/v1beta1
    kind: KubeadmControlPlane
    name: mgmt-prod-control-plane
    namespace: cluster-mgmt-prod
  infrastructureRef:
    apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
    kind: OpenStackCluster
    name: mgmt-prod-cluster
    namespace: cluster-mgmt-prod
  topology:
    class: mgmt-prod-cluster
    controlPlane:
      metadata:
        labels:
          t8s.teuto.net/cluster: mgmt-prod
          t8s.teuto.net/role: management
      replicas: 3
    variables:
    - name: dnsNameservers
      value:
      - *replace-me
      - *replace-me
    - name: controlPlaneServerGroupID
      value: *replace-me
    - name: machineDeploymentFlavor
      value: compute-plane-placeholder
    version: v1.28.15
    workers:
      machineDeployments:
      - class: compute-plane
        failureDomain: Zone2
        metadata: {}
        name: s2-default
        replicas: 4
        variables:
          overrides:
          - name: machineDeploymentServerGroupID
            value: *replace-me
          - name: machineDeploymentFlavor
            value: standard.2.1905

second cluster.yaml

apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
  name: mgmt-prod-cluster
spec:
  controlPlaneRef:
    apiVersion: controlplane.cluster.x-k8s.io/v1beta1
    kind: KubeadmControlPlane
    name: mgmt-prod-control-plane
    namespace: cluster-mgmt-prod
  infrastructureRef:
    apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
    kind: OpenStackCluster
    name: mgmt-prod-cluster
    namespace: cluster-mgmt-prod
  topology:
    class: mgmt-prod-cluster
    controlPlane:
      metadata:
        labels:
          t8s.teuto.net/cluster: mgmt-prod
          t8s.teuto.net/role: management
      replicas: 3
    variables:
    - name: dnsNameservers
      value:
      - *replace-me
      - *replace-me
    - name: controlPlaneServerGroupID
      value: *replace-me
    - name: machineDeploymentFlavor
      value: compute-plane-placeholder
    version: v1.29.13
    workers:
      machineDeployments:
      - class: compute-plane
        failureDomain: Zone2
        metadata: {}
        name: s2-default
        replicas: 4
        variables:
          overrides:
          - name: machineDeploymentServerGroupID
            value: *replace-me
          - name: machineDeploymentFlavor
            value: standard.2.1905

Does the behaviour change if you use kubectl apply --server-side=true (on create and on update)

The resulting templates are correct, I don't know what another way of applying (which would result in the same resource on the API server) would change?

We have a (kubeadmcontrolplane) template:

The first YAML shows a KubeadmControlPlane not a KubeadmControlPlaneTemplate

Oh yeah, I fixed the yamls 👍

[sbueringer]

Just to ensure I understand this correctly.

So you are able to make the changes you want on the KubeadmControlPlaneTemplate and the problem is that the changes are not applied the same way on the KubeadmControlPlane of a Cluster?

[cwrau]

Just to ensure I understand this correctly.

So you are able to make the changes you want on the KubeadmControlPlaneTemplate and the problem is that the changes are not applied the same way on the KubeadmControlPlane of a Cluster?

Yes, or rather (because of forbidden changes 🙄) I delete the template, create another and change the template ref in the cluster.

[cwrau]

Just to ensure I understand this correctly.
So you are able to make the changes you want on the KubeadmControlPlaneTemplate and the problem is that the changes are not applied the same way on the KubeadmControlPlane of a Cluster?

Yes, or rather (because of forbidden changes 🙄) I delete the template, create another and change the template ref in the cluster.

I have this problem in general with cluster-api resources. We use OpenStack as our infrastructure provider and the OpenStackClusterTemplate just has .spec.template.spec.bastion.enabled=false but in the existing OpenStackCluster we also have .spec.bastion.image.[...] but CAPI doesn't remove it and instead I get the following error on the CAPI cluster:

error reconciling the Cluster topology: failed to create patch helper
for OpenStackCluster/6446a8a8-87de-410d-9c88-bcb4b25fc9b9-x5xpx: server side
apply dry-run failed for modified object: OpenStackCluster.infrastructure.cluster.x-k8s.io
"6446a8a8-87de-410d-9c88-bcb4b25fc9b9-x5xpx" is invalid: [spec.bastion.spec.image:
Invalid value: 0: spec.bastion.spec.image in body should have at least 1 properties,
spec.bastion.spec: Invalid value: "object": at least one of flavor or flavorID
must be set]

If I manually remove these fields on the OpenStackCluster then CAPI can progress (although it runs into another problem I can't understand right now, but that's either our or CAPOs problem)

[chrischdi]

/triage accepted

/help

We'd like to reproduce this in KCP + CAPD first to get some confirmation on the core CAPI side.

[k8s-ci-robot]

@chrischdi:
This request has been marked as needing help from a contributor.

Guidelines

Please ensure that the issue body includes answers to the following questions:

• Why are we solving this issue?
• To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points?
• Does this issue have zero to low barrier of entry?
• How can the assignee reach out to you for help?

For more details on the requirements of such an issue, please see here and ensure that they are met.

If this request no longer meets these requirements, the label can be removed
by commenting with the /remove-help command.

In response to this:

/triage accepted

/help

We'd like to reproduce this in KCP + CAPD first to get some confirmation on the core CAPI side.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.