Development kick-off

[prometherion]

This is a discussion providing a recap of the meeting hosted on the 2nd of May, 2023

Attendees

• Dario Tranchitella @prometherion
• Lennart Jern @lentzi90
• Aurelio Forese @sn4psh0t
• Huy Mai

Recap

During the meeting, we agreed on starting implementing a Control Plane provider for Cluster API based on Kamaji.
The development playground will be OpenStack, @sn4psh0t from Netsons Cloud offered a dev environment.

Starting from a CAPO example following the proposed draft of the manifests:

Cluster

apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
spec:
  name: poc
spec:
  controlPlaneRef:
    apiVersion: controlplane.cluster.x-k8s.io/v1alpha1
    kind: KamajiControlPlane
    name: poc
    namespace: default
  infrastructureRef:
    apiVersion: infrastructure.cluster.x-k8s.io/v1alpha6
    kind: OpenStackCluster
    name: prow

spec.controlPlaneEndpoint entry will be populated once the target Kamaji instance is announcing its API endpoint

we agreed on defining a new Custom Resource Definition named KamajiControlPlane that will contain the definition of the control-plane.

For the sake of sharing, the proposed OpenStackCluster content is the following.

apiVersion: infrastructure.cluster.x-k8s.io/v1alpha6
kind: OpenStackCluster
metadata:
  name: prow
spec:
  apiServerLoadBalancer:
    enabled: false
  disableAPIServerFloatingIP: true
  apiServerFixedIP: ${MASTER_VIP}
  cloudName: prow
  dnsNameservers:
  - 8.8.8.8
  externalNetworkId: fba95253-5543-4078-b793-e2de58c31378
  identityRef:
    kind: Secret
    name: prow-cloud-config
  managedSecurityGroups: true
  nodeCidr: 10.6.0.0/24
  bastion:
    enabled: true
    instance:
      flavor: 1C-2GB-50GB
      image: "Ubuntu 22.04.1 Jammy Jellyfish 230124"
      sshKeyName: metal3ci-key

The key spec.apiServerLoadBalancer must configure the disabled API Server floating IP since Kamaji will provide it.

Once a Tenant Control Plane will have its IP address, the Kamaji Provider will patch the OpenStackCluster resource (key spec.apiServerFixedIP).

Worker machines

We can focus on the MachineDeployment reference for the compute nodes.

apiVersion: cluster.x-k8s.io/v1beta1
kind: MachineDeployment
metadata:
  name: prow-md-0
spec:
  clusterName: prow
  replicas: 1
  selector:
    matchLabels: null
  template:
    spec:
      bootstrap:
        configRef:
          apiVersion: bootstrap.cluster.x-k8s.io/v1beta1
          kind: KubeadmConfigTemplate
          name: prow-md-0
      clusterName: prow 
      failureDomain: nova
      infrastructureRef:
        apiVersion: infrastructure.cluster.x-k8s.io/v1alpha6
        kind: OpenStackMachineTemplate
        name: prow-md-0
      version: v1.26.3

just for the sake of deployment spec.template.spec.clusterName is referring to the Cluster object.

the infrastructure reference here is out of the scope

Worker nodes will join the cluster thanks to the KubeadmConfigTemplate as follows:

apiVersion: bootstrap.cluster.x-k8s.io/v1beta1
kind: KubeadmConfigTemplate
metadata:
  name: prow-md-0
spec:
  template:
    spec:
      joinConfiguration:
        nodeRegistration:
          kubeletExtraArgs:
            cloud-provider: external
          name: '{{ local_hostname }}'

Along with the OpenStackMachineTemplate:

apiVersion: infrastructure.cluster.x-k8s.io/v1alpha6
kind: OpenStackMachineTemplate
metadata:
  name: prow-md-0
spec:
  template:
    spec:
      cloudName: prow
      flavor: 8C-16GB-100GB
      identityRef:
        kind: Secret
        name: prow-cloud-config
      image: ubuntu-2204-kube-v1.26.3
      sshKeyName: metal3ci-key

All these manifests are transparent for Kamaji, mentioned for the sake of sharing.

KamajiControlPlane proposal

The following manifest is a draft for the KamajiControlPlane instance that will provide all the required information to provision a control plane, and sharing the required information with the underlying worker nodes to join the downstream cluster.

apiVerison: controlplane.cluster.x-k8s.io/v1alpha1
kind: KamajiControlPlane
metadata:
  name: poc
spec:
  controlPlane:
    deployment:
      replicas: 2
      additionalVolumes:
        - name: api-server-cloud-conf
          secret:
            name: api-server-cloud-conf
        - name: controller-manager-cloud-conf
          secret:
            name: controller-manager-cloud-conf
      additionalVolumeMounts:
        apiServer:
          - name: api-server-cloud-conf
            mountPath: "/etc/kubernetes/cloud.conf"
        controllerManager:
          - name: controller-manager-cloud-conf
            mountPath: "/etc/kubernetes/cloud.conf"
     extraArgs:
        apiServer: ["cloud-config=/etc/kubernetes/cloud.conf","cloud-provider=external"]
        controllerManager: ["cloud-config=/etc/kubernetes/cloud.conf","cloud-provider=external"]
        scheduler: []
      additionalMetadata:
        labels:
          cluster.api.k8s.io: poc
    service:
      type: LoadBalancer
      additionalMetadata:
        annotations:
          cluster.api.k8s.io: poc
        labels:
          cluster.api.k8s.io: poc
  kubernetes:
    kubelet:
      cgroupfs: systemd
    admissionControllers:
      - ResourceQuota
      - LimitRanger
  networkProfile:
    certSANs:
    - poc.kamaji.clastix.io
  addons:
    coreDNS: {}
    kubeProxy: {}
    konnectivity:
      server:
        port: ${TENANT_PROXY_PORT}
        resources:
          requests:
            cpu: 100m
            memory: 128Mi
          limits: {}

The idea is to struct embed the Kamaji's TenantControlPlane specification in the KamajiControlPlane resource in order to provide a layer of customization. The following keys are just an example according to the current state and may vary during the implementation.

User story/diagram

• A CAPI Cluster resource is created
• An OpenStack cluster reference is used
  • CAPO will ensure the Machine Deployments along with the required SecurityGroups and infrastructure components for the worker nodes
  • CAPO will mark the cluster as infrastructure ready
• A Kamaji Control Plane reference is used
  • Once a valid IP is available, the Kamaji provider will patch the Cluster definition with it
  • The Kamaji provider will provide the required details for the kubeadm-based bootstrap process such as certificates, tokens, etc.
  • Once accomplished, the worker nodes will be able to join the Control Plane thanks to the kubeadm machinery offered by Cluster API

Roadmap

• Finding a consensus on the flow described here (deadline: 31st of May, 2023)
• Setting up the development environment (deadline: 31st of May, 2023)
• Completing the provider development (deadline: 30th of June, 2023)

This post can be subjected to changes during the consensus flow.
Refer to the changes history.

[jds9090]

Good to hear good news!

As soon as you develop the Kamaji Control Plane, I am going to check if it works fine in my company’s environment(OpenStack rocky) and report the result to you.
Also, If I may find something which will help you or improve the Kamaji Control Plane , I will try to talk to you about that.

Thank you for your hard work :) 👍

Please refer to this information.

For OpenStackCluster, at least one of apiServerLoadBalancer, disableAPIServerFloatingIP, apiServerFixedIP is required.
(https://github.com/kubernetes-sigs/cluster-api-provider-openstack/blob/main/controllers/openstackcluster_controller.go#L517)

If Kamaji takes care of LoadBalancer, The expected template will be as below:

apiVersion: infrastructure.cluster.x-k8s.io/v1alpha6
kind: OpenStackCluster
metadata:
  name: prow
spec:
  apiServerLoadBalancer:
    enabled: false
  disableAPIServerFloatingIP: true
  apiServerFixedIP: ${MASTER_VIP}
  cloudName: prow
  dnsNameservers:
  - 8.8.8.8
  externalNetworkId: fba95253-5543-4078-b793-e2de58c31378
  identityRef:
    kind: Secret
    name: prow-cloud-config
  managedSecurityGroups: true
  nodeCidr: 10.6.0.0/24
  bastion:
    enabled: true
    instance:
      flavor: 1C-2GB-50GB
      image: "Ubuntu 22.04.1 Jammy Jellyfish 230124"
      sshKeyName: metal3ci-key

For KubeadmConfigTemplate,

• the cloud configuration and the cacert can be transferred through spec.template.spec.files.
• the joinConfiguration may include the endpoint for bootstrapToken and kubeletExtraArgs for external CCM

The expected template will be as below:

apiVersion: bootstrap.cluster.x-k8s.io/v1beta1
kind: KubeadmConfigTemplate
metadata:
  name: prow-md-0
spec:
  template:
    spec:
      files:
      - content: ${CLOUD_CONF}
        encoding: base64
        owner: root
        path: /etc/kubernetes/cloud.conf
        permissions: \"0600\"
      - content: ${CA_CERT}
        encoding: base64
        owner: root
        path: /etc/certs/cacert
        permissions: \"0600\"
      joinConfiguration:
        discovery:
          bootstrapToken:
            apiServerEndpoint: ${MASTER_VIP}:6443
        nodeRegistration:
          kubeletExtraArgs:
            cloud-config: /etc/kubernetes/cloud.conf
            cloud-provider: external
          name: '{{ local_hostname }}'

[jds9090]

Ok. I see what you want to talk about and I think it makes sense.
However, Let's look at this diagram and process that The Cluster API document shows us

https://cluster-api.sigs.k8s.io/developer/providers/cluster-infrastructure.html

Based on the document, Cluster API wants Cluster Infrastructure provider to take care about LoadBalancer.
As you can see, three infrastructure providers follow the direction.

AzureCluster : https://github.com/kubernetes-sigs/cluster-api-provider-azure/blob/0ec7aa4f03f0437bf279141b8cbe2f77d9491fef/api/v1beta1/types.go#L68
AWSCluster : https://github.com/kubernetes-sigs/cluster-api-provider-aws/blob/32c475c824fc91ca7f6327521d7689f005494310/api/v1beta2/awscluster_types.go#L35
OpenStackCluster : https://github.com/kubernetes-sigs/cluster-api-provider-openstack/blob/783aeea6a5df443fe7dc875ddfc88e38a3f91339/api/v1alpha7/openstackcluster_types.go#L32

So, If the TenantControlPlane manages LoadBalancer, you need to do the exactly same thing with other providers as you explained.
other providers do not support spec.apiServerFixedIP. It means that there may be tight coupling with OpenStackProvider.

Also, if ControlPlane manages LoadBalancer, the behavior does not follow the document.
However, What you want to do makes sense and sounds reasonable for kamaji which aims for the mixed infrastructures cluster.

[prometherion]

other providers do not support spec.apiServerFixedIP. It means that there may be tight coupling with OpenStackProvider.

The plan for getting Kamaji as a CAPI Control Plane provider, along with the feature group, is to start a discussion with the community to rearchitect this.

Also, if ControlPlane manages LoadBalancer, the behavior does not follow the document.
However, What you want to do makes sense and sounds reasonable for kamaji which aims for the mixed infrastructures cluster.

You're right, let's try to summon @fabriziopandini and CAPI people to get their opinion on this.

[mdbooth]

This story from @tobiasgiese also feels relevant here: kubernetes-sigs/cluster-api#8614 (comment). It references this long-standing feature suggestion for an independent LB provider: kubernetes-sigs/cluster-api#1250

I'm personally in favour of this. There are many potential LB solutions just with vanilla OpenStack and we don't want to be including them all in a single provider-specific controller. It's a significant architecture change, though.

[fabriziopandini]

FYI there is a proposal to make CP that manages a load balancer a first-class citizen in CAPI kubernetes-sigs/cluster-api#8500

[prometherion]

Considering the CAPI proposal for Load Balancers, which looks promising, I would go for a first iteration where the Kamaji provider will take care of patching the OpenStack resource by assigning the Load Balancer IP.

That's wrong, I agree, the plan is to add this topic as a next action item in order to get it solved through a general consensus with the community. Let me know if you disagree with this plan, or not.

[fabriziopandini]

kudos for driving this work @prometherion, and most important for having a clear goal to iteratively experiment/showcase, report back, and improve the entire CAPI ecosystem, because ultimately the success of this is tied with the work of defining a robust and extensible contract with the providers that will take care of machines and the rest of the infrastructure.

Going back on the specific work, based on the discussion at the latest Kubecon in Amsterdam, there is a general interest in a control plane in a pod solution, and most probably there is already something you can leverage for a quick start. I have also played personally around with this idea a couple of times, the last one in https://github.com/fabriziopandini/cluster-api-provider-kubemark/tree/kubemark-controlplane, but this is very experimental 😉

However, what is key is that If there is an expectation to capture the above community sometime in the future, it is crucial to design a clear boundary between what is generic and what is specific to kamaji.

note: kcp as a acronym is overrated (CAPI kcp, https://www.kcp.io/), so I suggest using a different acronym, might be something that highlights that the CP runs in a pod

[prometherion]

note: kcp as a acronym is overrated (CAPI kcp, https://www.kcp.io/), so I suggest using a different acronym, might be something that highlights that the CP runs in a pod

You're the second one, first was @bsctl, which suggested using a different short name for Kamaji Control Planes: tl;dr; so much seniority here!

However, what is key is that If there is an expectation to capture the above community sometime in the future, it is crucial to design a clear boundary between what is generic and what is specific to kamaji.

Thanks for jumping in the discussion: I couldn't agree more! 👏🏻

[prometherion]

I'd like to propose a different name for the proposed instance KamajiControlPlane.

What do you think about:

1. KamajiTemplate
2. TenantControlPlaneTemplate
3. KamajiControlPlaneTemplate

I wouldn't use a short name to avoid collisions if we're unaware of other providers since we're sharing the same API group.

/cc @bsctl @fabriziopandini

[lentzi90]

Having template in the name can become quite strange if using ClusterClass I think. Check the docs here: https://cluster-api.sigs.k8s.io/tasks/experimental-features/cluster-class/write-clusterclass.html
I guess we would end up with a KamajiControlPlaneTemplateTemplate or similarly ridiculous kind 😬

[bsctl]

Few options coming in my mind:

• PodControlPlane
• VirtualControlPlane
• FloatingControlPlane
• ControlPlane
• TenantControlPlane

The last one has the same name of the TenantControlPlane Custom Resource in Kamaji but different APIs group.

Names matter: I failed my software engineering examination because I used wrong variable names in the code.

[prometherion]

I know @bsctl won't like this, but KamajiControlPlane seems the most reasonable solution to me.

It is easy to understand that's a control plane in the cluster API boundaries, and that's a Kamaji kind one rather than a generic Pod-based one.

[alexander-demicev]

The proposed design makes sense to me. Ideally, the approach taken here should be easily adaptable to other infrastructure configurations. For example, it should be possible to use microvm provider for control planes and bare metal for worker nodes, here is an article on this topic https://www.weave.works/blog/multi-cluster-kubernetes-on-microvms-for-bare-metal. From the perspective of the CAPI API, it should be agnostic to the combination of infrastructure providers being used. It's important for us to avoid ending in a situation where each similar use case, using different providers, takes a different approach.

[jds9090]

I want to make sure some points as below

• [Manifest] The Kamaji Provider should patch the KubeadmConfigTemplate resource(key joinConfiguration.discovery.bootstrapToken.apiServerEndpoint)
• [User story/diagram] CAPO will mark the OpenStackCluster as ready and CAPI will mark the Cluster as infrastructure ready.
• [KamajiControlPlane proposal] Since the PR(Allow mangling the TCP Deployment kamaji#297) has been closed, the proposal may be something like this.

apiVerison: controlplane.cluster.x-k8s.io/v1alpha1
kind: KamajiControlPlane
metadata:
  name: poc
spec:
  controlPlane:
    deployment:
      replicas: 2
      additionalVolumes:
        - name: api-server-cloud-conf
          secret:
            name: api-server-cloud-conf
        - name: controller-manager-cloud-conf
          secret:
            name: controller-manager-cloud-conf
      additionalVolumeMounts:
        apiServer:
          - name: api-server-cloud-conf
            mountPath: "/etc/kubernetes/cloud.conf"
        controllerManager:
          - name: controller-manager-cloud-conf
            mountPath: "/etc/kubernetes/cloud.conf"
     extraArgs:
        apiServer: ["cloud-config=/etc/kubernetes/cloud.conf","cloud-provider=external"]
        controllerManager: ["cloud-config=/etc/kubernetes/cloud.conf","cloud-provider=external"]
        scheduler: []
      additionalMetadata:
        labels:
          cluster.api.k8s.io: poc
    service:
      additionalMetadata:
        annotations:
          cluster.api.k8s.io: poc
        labels:
          cluster.api.k8s.io: poc
  kubernetes:
    kubelet:
      cgroupfs: systemd
    admissionControllers:
      - ResourceQuota
      - LimitRanger
  networkProfile:
    certSANs:
    - poc.kamaji.clastix.io
  addons:
    coreDNS: {}
    kubeProxy: {}
    konnectivity:
      server:
        port: ${TENANT_PROXY_PORT}
        resources:
          requests:
            cpu: 100m
            memory: 128Mi
          limits: {}

One question here! @prometherion

Is there any expected problem about the kubeadm-based bootstrap process by both Kamaji provider and CAPI(kubeadm bootstrap) since you are going to use KubeadmConfigTemplate for worker nodes?
Can you give us a brief description about how to solve them if you expect some problems?

[prometherion]

Thanks for jumping on the discussion, @jds9090, I really appreciate this!

I confirm all the said points, such as:

• Manifest
• User Story
• KamajiControlPlane proposal

For the last one it could differ a bit according to the needs, I'll keep all the stakeholders here informed with the development progress.

About the following question:

Is there any expected problem about the kubeadm-based bootstrap process by both Kamaji provider and CAPI(kubeadm bootstrap) since you are going to use KubeadmConfigTemplate for worker nodes?
Can you give us a brief description about how to solve them if you expect some problems?

This is a good point. Kamaji is overlapping with several areas such as the infrastructure (e.g.: providing an API address, thus a LoadBalancer) and the bootstrap process.

The development team decided to go opinionated with the kubeadm bootstrap phase by generating all the required components such as cluster-info, kubelet-config, and all the other logical blocks to let any worker node based on kubeadm as a bootstrap strategy to join the Tenant Control Plane.

I could be wrong here, and happy for being corrected, some kubeadm required information such as certificates, configuration, etc. are stored in the management cluster to let the MachineProvider consume these according to their logic. I roughly remember thanks to my first tests using the Docker CAPI provider with Kamaji and some resources needed to be translated from the Kamaji naming to the kubeadm CAPI-based one (e.g.: the admin kubeconfig).

tl;dr; the Kamaji CAPI Control Plane provider will take care of providing seamless integration with the kubeadm-based bootstrap process by generating (sic) all the required resources.

I know this is a hard take, but at the same time I want to be pragmatic given the current state of Kamaji: we can discuss in future how to decouple Kamaji from the kubeadm-based bootstrap strategy using its GitHub repository Discussion section.

[jds9090]

sounds like The Kamaji operator will deploy kubeadmConfigTemplate(of CABPK) itself. If it is right, you may consider that The Kamaji provider should give users options to set KubeletConfiguration or something that kubeadm supports.
For example, we set clusterDNS of KubeletConfiguration to use nodeLocal DNSCache(https://kubernetes.io/docs/tasks/administer-cluster/nodelocaldns/)

apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
clusterDNS:
- 169.254.20.10

I think this approach is reasonable for kamaji at this moment. Let's keep discussing about how to decouple :

the Kamaji CAPI Control Plane provider will take care of providing seamless integration with the kubeadm-based bootstrap process by generating (sic) all the required resources.

[prometherion]

@jds9090 I think this is already supported by Kamaji:

https://github.com/clastix/kamaji/blob/c81d1907199ecb8acbabb90c202f79cf1759347c/api/v1alpha1/tenantcontrolplane_types.go#L32-L33

apiVersion: kamaji.clastix.io/v1alpha1
kind: TenantControlPlane
metadata:
  finalizers:
  - finalizer.kamaji.clastix.io
  - finalizer.kamaji.clastix.io/soot
  name: k8s-126
  namespace: default
spec:
  addons:
    coreDNS: {}
    konnectivity:
      agent:
        image: registry.k8s.io/kas-network-proxy/proxy-agent
        version: v0.0.32
      server:
        image: registry.k8s.io/kas-network-proxy/proxy-server
        port: 8132
        version: v0.0.32
    kubeProxy: {}
  controlPlane:
    deployment:
      additionalMetadata: {}
      registrySettings:
        apiServerImage: kube-apiserver
        controllerManagerImage: kube-controller-manager
        registry: registry.k8s.io
        schedulerImage: kube-scheduler
      replicas: 2
      strategy:
        rollingUpdate:
          maxSurge: 100%
          maxUnavailable: 0
        type: RollingUpdate
    service:
      additionalMetadata: {}
      serviceType: LoadBalancer
  dataStore: default
  kubernetes:
    admissionControllers:
    - CertificateApproval
    - CertificateSigning
    - CertificateSubjectRestriction
    - DefaultIngressClass
    - DefaultStorageClass
    - DefaultTolerationSeconds
    - LimitRanger
    - MutatingAdmissionWebhook
    - NamespaceLifecycle
    - PersistentVolumeClaimResize
    - Priority
    - ResourceQuota
    - RuntimeClass
    - ServiceAccount
    - StorageObjectInUseProtection
    - TaintNodesByCondition
    - ValidatingAdmissionWebhook
    kubelet:
      cgroupfs: systemd
      preferredAddressTypes:
      - Hostname
      - InternalIP
      - ExternalIP
    version: v1.26.0
  networkProfile:
    dnsServiceIPs:
    - 169.254.20.10
    podCidr: 10.244.0.0/16
    port: 6443
    serviceCidr: 10.96.0.0/16

This is the resulting KubeletConfiguration content in the Tenant Control Plane:

    apiVersion: kubelet.config.k8s.io/v1beta1
    authentication:
      anonymous:
        enabled: false
      webhook:
        cacheTTL: 0s
        enabled: true
      x509:
        clientCAFile: /etc/kubernetes/pki/ca.crt
    authorization:
      mode: Webhook
      webhook:
        cacheAuthorizedTTL: 0s
        cacheUnauthorizedTTL: 0s
    cgroupDriver: systemd
    clusterDNS:
    - 169.254.20.10
    clusterDomain: cluster.local
    cpuManagerReconcilePeriod: 0s
    evictionHard:
      imagefs.available: 0%
      nodefs.available: 0%
      nodefs.inodesFree: 0%
    evictionPressureTransitionPeriod: 0s
    fileCheckFrequency: 0s
    healthzBindAddress: 127.0.0.1
    healthzPort: 10248
    httpCheckFrequency: 0s
    imageGCHighThresholdPercent: 100
    imageMinimumGCAge: 0s
    kind: KubeletConfiguration
    logging:
      flushFrequency: 0
      options:
        json:
          infoBufferSize: "0"
      verbosity: 0
    memorySwap: {}
    nodeStatusReportFrequency: 0s
    nodeStatusUpdateFrequency: 0s
    rotateCertificates: true
    runtimeRequestTimeout: 0s
    shutdownGracePeriod: 0s
    shutdownGracePeriodCriticalPods: 0s
    staticPodPath: /etc/kubernetes/manifests
    streamingConnectionIdleTimeout: 0s
    syncFrequency: 0s
    volumeStatsAggPeriod: 0s

[prometherion]

Some good news!

Thanks to @sn4psh0t we're testing the provider in Netsons' infrastructure, many kudos to them for invoicing a nil bill for the resource consumption! <3

I was successfully able to create a Kamaji Control Plane in the CAPI cluster, and the workers nodes in the OpenStack dev env.

apiVersion: bootstrap.cluster.x-k8s.io/v1beta1
kind: KubeadmConfigTemplate
metadata:
  name: capi-quickstart-md-0
  namespace: default
spec:
  template:
    spec:
      joinConfiguration:
        nodeRegistration:
          name: '{{ local_hostname }}'
---
apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
  name: capi-quickstart
  namespace: default
spec:
  clusterNetwork:
    pods:
      cidrBlocks:
      - 192.168.0.0/16
    serviceDomain: cluster.local
  controlPlaneRef:
    apiVersion: controlplane.cluster.x-k8s.io/v1alpha1
    kind: KamajiControlPlane
    name: capo-cp-test
  infrastructureRef:
    apiVersion: infrastructure.cluster.x-k8s.io/v1alpha6
    kind: OpenStackCluster
    name: capi-quickstart
---
apiVersion: infrastructure.cluster.x-k8s.io/v1alpha6
kind: OpenStackCluster
metadata:
  name: capi-quickstart
  namespace: default
spec:
  apiServerLoadBalancer:
    enabled: false
  disableAPIServerFloatingIP: true
  apiServerFixedIP: ""
  cloudName: openstack
  dnsNameservers:
  - 1.1.1.1
  externalNetworkId: <REDACTED>
  identityRef:
    kind: Secret
    name: capi-quickstart-cloud-config
  managedSecurityGroups: true
  nodeCidr: 10.6.0.0/24
---
apiVersion: controlplane.cluster.x-k8s.io/v1alpha1
kind: KamajiControlPlane
metadata:
  name: capo-cp-test
spec:
  replicas: 2
  version: v1.24.0
---
apiVersion: cluster.x-k8s.io/v1beta1
kind: MachineDeployment
metadata:
  name: capi-quickstart-md-0
  namespace: default
spec:
  clusterName: capi-quickstart
  replicas: 3
  selector:
    matchLabels: null
  template:
    spec:
      bootstrap:
        configRef:
          apiVersion: bootstrap.cluster.x-k8s.io/v1beta1
          kind: KubeadmConfigTemplate
          name: capi-quickstart-md-0
      clusterName: capi-quickstart
      failureDomain: <REDACTED>
      infrastructureRef:
        apiVersion: infrastructure.cluster.x-k8s.io/v1alpha6
        kind: OpenStackMachineTemplate
        name: capi-quickstart-md-0
      version: v1.24.0
---
apiVersion: infrastructure.cluster.x-k8s.io/v1alpha6
kind: OpenStackMachineTemplate
metadata:
  name: capi-quickstart-md-0
  namespace: default
spec:
  template:
    spec:
      cloudName: <REDACTED>
      flavor: <REDACTED>
      identityRef:
        kind: Secret
        name: capi-quickstart-cloud-config
      image: <REDACTED>
      sshKeyName: <REDACTED>

However, we're getting some issues with the OpenstackCluster resource.

@jds9090 proposed here to patch the spec.apiServerFixedIP key once the TenantControlPlane address was provided by Kamaji, but unfortunately this is not possible, by encountering this error upon the patch.

2023-06-13T11:38:56+02:00       ERROR   cannot patch capiv1beta1.Cluster        {"controller": "kamajicontrolplane", "controllerGroup": "controlplane.cluster.x-k8s.io", "controllerKind": "KamajiControlPlane", "KamajiControlPlane": {"name":"capo-cp-test","namespace":"default"}, "namespace": "default", "name": "capo-cp-test", "reconcileID": "3c67cf0b-ab77-428f-a454-d1c1267b1747", "error": "admission webhook \"validation.openstackcluster.infrastructure.cluster.x-k8s.io\" denied the request: OpenStackCluster.infrastructure.cluster.x-k8s.io \"capi-quickstart\" is invalid: spec: Forbidden: cannot be modified"}

Digging the CAPO code-base, I ended up here: the OpenStack provider is blocking any update operation, except for:

• Spec.IdentityRef.Name
• Spec.IdentityRef (if unset)
• Spec.ControlPlaneEndpoint (if unset)
• Spec.Bastion
• Spec.APIServerLoadBalancer.AllowedCIDRs

All other changes are not allowed, and this blocks us since the workflow from our side is the following:

1. User creates all the CAPI manifests
2. The Kamaji provider creates the TenantControlPlane
3. The Kamaji provider waits for the exposed TenantControlPlane address
4. Once the address is available, it patches the OpenStackCluster resource, as suggested by @jds9090

However, step nr. 4 is not possible, thus to test the whole lifecycle I had to hack a bit of the code and create a well-known address in advance, just to be sure everything was working as expected.

My plan is to engage with the CAPO community in order to unblock ourselves from this: I'd say we could ask to ignore changes of the spec.apiServerFixedIP key, if and only if we're sure there are no side effects I'm not aware.

[jds9090]

@prometherion Good to hear your progress :) !!

Why don't the Kamaji provider create the OpenStackCluster resource once the address is available like the Kamaji provider takes care of providing seamless integration with the kubeadm-based bootstrap process.

For this approach,
the Kamaji provider needs common information for any infrastructure provider such as OpenStackCluster, AWSCluster, and so one.

I think it can be helpful for the Kamaji provider to collaborate with many infrastructure provider since the Kamaji provider takes care of any infrastructure provider's work such as creating LoadBalancer in terms of CAPI.

[prometherion]

For this approach,
the Kamaji provider needs common information for any infrastructure provider such as OpenStackCluster, AWSCluster, and so one.

There are so many specific options for the provider cluster resource that we cannot abstract.

I think it can be helpful for the Kamaji provider to collaborate with many infrastructure provider since the Kamaji provider takes care of any infrastructure provider's work such as creating LoadBalancer in terms of CAPI.

There's already an ongoing discussion about abstracting the load balancer management in CAPI. We have to follow the community and collaborate with it.

[prometherion]

I need quick feedback about the API specification for the KamajiControlPlane resource.

The following sample is the combination between the resources managed by Kamaji itself combined with the one offered by CAPI objects.

apiVersion: controlplane.cluster.x-k8s.io/v1alpha1
kind: KamajiControlPlane
metadata:
  labels:
    app.kubernetes.io/name: kamajicontrolplane
    app.kubernetes.io/instance: kamajicontrolplane-sample
    app.kubernetes.io/part-of: cluster-api-control-plane-provider-kamaji
    app.kubernetes.io/managed-by: kustomize
    app.kubernetes.io/created-by: cluster-api-control-plane-provider-kamaji
  name: kamajicontrolplane-sample
spec:
  dataStoreName: default
  addons:
    coreDNS: { }
    konnectivity: { }
    kubeProxy: { }
  admissionControllers:
    - AlwaysAdmit
    - AlwaysDeny
    - AlwaysPullImages
    - CertificateApproval
    - CertificateSigning
    - CertificateSubjectRestriction
    - DefaultIngressClass
    - DefaultStorageClass
    - DefaultTolerationSeconds
    - DenyEscalatingExec
    - DenyExecOnPrivileged
    - DenyServiceExternalIPs
    - EventRateLimit
    - ExtendedResourceToleration
    - ImagePolicyWebhook
    - LimitPodHardAntiAffinityTopology
    - LimitRanger
    - MutatingAdmissionWebhook
    - NamespaceAutoProvision
    - NamespaceExists
    - NamespaceLifecycle
    - NodeRestriction
    - OwnerReferencesPermissionEnforcement
    - PersistentVolumeClaimResize
    - PersistentVolumeLabel
    - PodNodeSelector
    - PodSecurity
    - PodSecurityPolicy
    - PodTolerationRestriction
    - Priority
    - ResourceQuota
    - RuntimeClass
    - SecurityContextDeny
    - ServiceAccount
    - StorageObjectInUseProtection
    - TaintNodesByCondition
    - ValidatingAdmissionWebhook
  registry: registry.k8s.io
  controllerManager:
    extraVolumeMounts: [ ]
    extraArgs:
      - --cloud-provider=external
    resources:
      limits:
        cpu: 500m
        memory: 128Mi
    containerImageName: kube-controller-manager
  apiServer:
    extraVolumeMounts: [ ]
    extraArgs:
      - --cloud-provider=external
    resources:
      requests:
        cpu: 750m
        memory: 256Mi
    containerImageName: kube-apiserver
  scheduler:
    extraVolumeMounts: [ ]
    extraArgs: [ ]
    resources:
      limits:
        cpu: 500m
        memory: 128Mi
    containerImageName: kube-scheduler
  kubelet:
    preferredAddressTypes:
      - Hostname
      - InternalIP
      - ExternalIP
    cgroupfs: systemd
  network:
    serviceType: LoadBalancer
    serviceLabels:
      kamaji.clastix.io/service: external
    serviceAnnotations:
      service.beta.kubernetes.io/azure-load-balancer-internal: "false"
    certSANs:
      - kamajicontrolplane-sample.eu-west-01.cloudapp.azure.com
  deployment:
    nodeSelector:
      kubernetes.io/os: linux
    runtimeClassName: runc
    strategy:
      type: RollingUpdate
      rollingUpdate:
        maxUnavailable: 0
        maxSurge: 100%
    affinity:
      podAntiAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
                - key: kamaji.clastix.io/name
                  operator: In
                  values:
                    - kamajicontrolplane-sample
            topologyKey: kubernetes.io/hostname
    tolerations:
      - effect: NoExecute
        operator: Equal
        value: workload
    topologySpreadConstraints:
      - maxSkew: 1
        topologyKey: kubernetes.io/hostname
        whenUnsatisfiable: DoNotSchedule
        labelSelector:
          matchLabels:
            kamaji.clastix.io/name: kamajicontrolplane-sample
        matchLabelKeys:
          - pod-template-hash
    extraInitContainers: [ ]
    extraContainers: [ ]
    extraVolumes: [ ]
  replicas: 2
  version: 1.27.0

You can notice we're missing some values:

• the Kubernetes API port, backed by cluster.Spec.ClusterNetwork.APIServerPort
• the Kubernetes Services CIDR, backed by cluster.Spec.ClusterNetwork.Services
• the Kubernetes Pods CIDR, backed by cluster.Spec.ClusterNetwork.Pods

Please, let me know if you find something odd, or any missing features I lost in the drafting.

Most of the fields are in omitempty mode, the mandatory ones are:

• spec.version
• spec.dataStoreName

[jds9090]

@prometherion I am so excited about your progress! :)

I have two questions for you.

Firstly, Is there any way to set the kubeProxy's mode with the API specification?

apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
mode: ipvs

Secondly, I want to double check how to set cloud configuration based on the Secret resource for controller-manager and api-server since the example uses cloud-provider as external.

Here is what I expect for that. Please focus on extraVolumeMounts, extraArgs and extraVolumes. Is it correct or possible?

apiVersion: controlplane.cluster.x-k8s.io/v1alpha1
kind: KamajiControlPlane
metadata:
  labels:
    app.kubernetes.io/name: kamajicontrolplane
    app.kubernetes.io/instance: kamajicontrolplane-sample
    app.kubernetes.io/part-of: cluster-api-control-plane-provider-kamaji
    app.kubernetes.io/managed-by: kustomize
    app.kubernetes.io/created-by: cluster-api-control-plane-provider-kamaji
  name: kamajicontrolplane-sample
spec:
  dataStoreName: default
  addons:
    coreDNS: { }
    konnectivity: { }
    kubeProxy: { }
  admissionControllers:
    - AlwaysAdmit
    - AlwaysDeny
    - AlwaysPullImages
    - CertificateApproval
    - CertificateSigning
    - CertificateSubjectRestriction
    - DefaultIngressClass
    - DefaultStorageClass
    - DefaultTolerationSeconds
    - DenyEscalatingExec
    - DenyExecOnPrivileged
    - DenyServiceExternalIPs
    - EventRateLimit
    - ExtendedResourceToleration
    - ImagePolicyWebhook
    - LimitPodHardAntiAffinityTopology
    - LimitRanger
    - MutatingAdmissionWebhook
    - NamespaceAutoProvision
    - NamespaceExists
    - NamespaceLifecycle
    - NodeRestriction
    - OwnerReferencesPermissionEnforcement
    - PersistentVolumeClaimResize
    - PersistentVolumeLabel
    - PodNodeSelector
    - PodSecurity
    - PodSecurityPolicy
    - PodTolerationRestriction
    - Priority
    - ResourceQuota
    - RuntimeClass
    - SecurityContextDeny
    - ServiceAccount
    - StorageObjectInUseProtection
    - TaintNodesByCondition
    - ValidatingAdmissionWebhook
  registry: registry.k8s.io
  controllerManager:
    extraVolumeMounts:
    - name: controller-manager-cloud-conf
      mountPath: "/etc/kubernetes/cloud.conf"
    extraArgs:
      - --cloud-provider=external
      - --cloud-config=/etc/kubernetes/cloud.conf
    resources:
      limits:
        cpu: 500m
        memory: 128Mi
    containerImageName: kube-controller-manager
  apiServer:
    extraVolumeMounts:
    - name: api-server-cloud-conf
      mountPath: "/etc/kubernetes/cloud.conf"
    extraArgs:
      - --cloud-provider=external
      - --cloud-config=/etc/kubernetes/cloud.conf
    resources:
      requests:
        cpu: 750m
        memory: 256Mi
    containerImageName: kube-apiserver
  scheduler:
    extraVolumeMounts: [ ]
    extraArgs: [ ]
    resources:
      limits:
        cpu: 500m
        memory: 128Mi
    containerImageName: kube-scheduler
  kubelet:
    preferredAddressTypes:
      - Hostname
      - InternalIP
      - ExternalIP
    cgroupfs: systemd
  network:
    serviceType: LoadBalancer
    serviceLabels:
      kamaji.clastix.io/service: external
    serviceAnnotations:
      service.beta.kubernetes.io/azure-load-balancer-internal: "false"
    certSANs:
      - kamajicontrolplane-sample.eu-west-01.cloudapp.azure.com
  deployment:
    nodeSelector:
      kubernetes.io/os: linux
    runtimeClassName: runc
    strategy:
      type: RollingUpdate
      rollingUpdate:
        maxUnavailable: 0
        maxSurge: 100%
    affinity:
      podAntiAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
                - key: kamaji.clastix.io/name
                  operator: In
                  values:
                    - kamajicontrolplane-sample
            topologyKey: kubernetes.io/hostname
    tolerations:
      - effect: NoExecute
        operator: Equal
        value: workload
    topologySpreadConstraints:
      - maxSkew: 1
        topologyKey: kubernetes.io/hostname
        whenUnsatisfiable: DoNotSchedule
        labelSelector:
          matchLabels:
            kamaji.clastix.io/name: kamajicontrolplane-sample
        matchLabelKeys:
          - pod-template-hash
    extraInitContainers: [ ]
    extraContainers: [ ]
    extraVolumes:
    - name: api-server-cloud-conf
      secret:
         name: api-server-cloud-conf
    - name: controller-manager-cloud-conf
      secret:
        name: controller-manager-cloud-conf
  replicas: 2
  version: 1.27.0

[prometherion]

Let's try answering the two questions:

1. extra volumes, volume mounts, and args

Kamaji supports it starting from v0.3.0

2. kube-proxy IPVS mode

We're relying on kubeadm addons, and it seems to me the default mode is iptables. With that said, there's no way to customise it unless opening a Feature Request on kubeadm, and then implementing it in Kamaji which relies on.

Otherwise, you could skip the kube-proxy Kamaji reconciliation and manage that in a different way, although losing the immediate reconciliation upon a change.

I'd say we should track this on Kamaji's main repository, please, feel free to open an issue there.

[jds9090]

Always, Thank you for your quick reply and necessary information! :)
So.. You mean Kamaji Provider(about CAPI) supports what Kamaji supports currently. I thought few functions could be not supported by Kamaji Provider.

[prometherion]

I'd like to thank all the people involved in this process, and I just wanted to share that the first version of the KamajiControlPlane CAPI provider is out as v0.1.0!

I'm going to close this discussion and I'm encouraging you in giving it a try and spot bugs: looking forward to those in the Issue sections.

Ad maiora!