Adding loadbalancer prevents Kafka StatefulSet from being created

[smarsh-tim]

Describe the bug
I found that when I add a type: loadbalancer to the listeners block in kind: Kafka custom resource - that the operator fails to create the Kafka StatefulSet.

Removing the loadbalancer listener then allows the Kafka StatefulSet to be created, but I need the loadbalancer listener.

This seems to occur because of an order of operations issue between the LoadBalancer resources and Kafka StatefulSet. When the LoadBalancer service is created, it goes into Pending state until the Endpoints are created. Those endpoints cannot be created until Pods exist that match the Selector configuration of the LoadBalancer. However, no Pods will ever exist since the strimzi-kafka-operator is stuck waiting on the LoadBalancer to show Ready before it will continue - and hasn't yet gotten to the Kafka StatefulSet creation.

To Reproduce
Steps to reproduce the behavior:

1. Attempt to create a Kafka custom resource with a listener added such that the list looks like this

listeners:
  - name: plain
    port: 9092
    type: internal
    tls: false
  - name: tls
    port: 9093
    type: internal
    tls: true
  - name: external
    port: 9094
    type: loadbalancer
    tls: false
    configuration:
     externalTrafficPolicy: Local
     bootstrap:
       loadBalancerIP: 10.100.10.3
     brokers:
       - broker: 0
         loadBalancerIP: 10.100.10.4

2. Observe the cluster creation getting stuck on the LoadBalancer resources waiting to come up - failing to proceed to the StatefulSet for Kafka
3. Delete the Kafka custom resource
4. Create a new Kafka custom resource without the loadbalancer:

listeners:
  - name: plain
    port: 9092
    type: internal
    tls: false
  - name: tls
    port: 9093
    type: internal
    tls: true

5. Observe that the Kafka StatefulSet is created successfully

Expected behavior
I expect the StatefulSet to be created before, or in tandem, with the LoadBalancer resources. Such that the Kafka pods are able to generate the Endpoint references required for the LoadBalancer.

Environment (please complete the following information):

• Strimzi version: [e.g. main, 0.22.1]
• Installation method: Helm chart (latest)

apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
  name: strimzi-kafka-operator
  namespace: strimzi-kafka
spec:
  interval: 15m
  chart:
    spec:
      chart: strimzi-kafka-operator
      sourceRef:
        kind: HelmRepository
        name: strimzi
        namespace: strimzi-kafka
      interval: 15m
  values:
    logLevel: DEBUG
    fullReconciliationIntervalMs: 120000
    operationTimeoutMs: 300000
    nodeSelector:
      kubernetes.io/os: linux
    watchAnyNamespace: true

• Kubernetes cluster: Kubernetes 1.21.2
• Infrastructure: Amazon EKS Anywhere

YAML files and logs

Custom Resource (using loadbalancer)

---
apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: my-app
  namespace: default
spec:
  entityOperator:
    topicOperator: {}
    userOperator: {}
  kafka:
    config:
      default.replication.factor: 1
      inter.broker.protocol.version: "3.1"
      min.insync.replicas: 1
      offsets.topic.replication.factor: 1
      transaction.state.log.min.isr: 1
      transaction.state.log.replication.factor: 1
    listeners:
    - name: plain
      port: 9092
      tls: false
      type: internal
    - name: tls
      port: 9093
      tls: true
      type: internal
    - name: external
      port: 9094
      type: loadbalancer
      tls: false
      configuration:
       externalTrafficPolicy: Local
       bootstrap:
         loadBalancerIP: 10.100.10.3
       brokers:
         - broker: 0
           loadBalancerIP: 10.100.10.4
    replicas: 1
    storage:
      type: jbod
      volumes:
      - class: persistent-standard
        deleteClaim: true
        id: 0
        size: 1Gi
        type: persistent-claim
  zookeeper:
    replicas: 1
    storage:
      class: persistent-standard
      deleteClaim: false
      size: 10Gi
      type: persistent-claim

Logs (using loadbalancer)

% kubectl logs strimzi-cluster-operator-6785cf7b69-gwf9q -n strimzi-kafka --follow | grep "Reconciliation #1(watch)"
...
2022-03-04 01:50:17 DEBUG AbstractResourceOperator:276 - Reconciliation #1(watch) Kafka(default/my-app): Service my-app-kafka-0 in namespace default has been created
2022-03-04 01:50:17 DEBUG KafkaAssemblyOperator:2015 - Reconciliation #1(watch) Kafka(default/my-app): Reconciling existing Ingresses [] against the desired ingresses
2022-03-04 01:50:17 DEBUG KafkaAssemblyOperator:2024 - Reconciliation #1(watch) Kafka(default/my-app): Ingresses [] should be deleted
2022-03-04 01:50:17 DEBUG Util:118 - Reconciliation #1(watch) Kafka(default/my-app): Waiting for Service resource my-app-kafka-external-bootstrap in namespace default to get addressable
2022-03-04 01:50:38 INFO  AbstractOperator:373 - Reconciliation #1(watch) Kafka(default/my-app): Reconciliation is in progress
...
2022-03-04 01:54:38 INFO  AbstractOperator:373 - Reconciliation #1(watch) Kafka(default/my-app): Reconciliation is in progress
2022-03-04 01:55:17 ERROR Util:149 - Reconciliation #1(watch) Kafka(default/my-app): Exceeded timeout of 300000ms while waiting for Service resource my-app-kafka-external-bootstrap in namespace default to be addressable
2022-03-04 01:55:17 ERROR AbstractOperator:247 - Reconciliation #1(watch) Kafka(default/my-app): createOrUpdate failed
2022-03-04 01:55:17 INFO  CrdOperator:113 - Reconciliation #1(watch) Kafka(default/my-app): Status of Kafka my-app in namespace default has been updated
2022-03-04 01:55:17 DEBUG AbstractOperator:323 - Reconciliation #1(watch) Kafka(default/my-app): Completed status update
2022-03-04 01:55:17 DEBUG AbstractOperator:618 - Reconciliation #1(watch) Kafka(default/my-app): Updated metric strimzi.resource.state[tag(kind=Kafka),tag(name=my-app),tag(reason=Exceeded timeout of 300000ms while waiting for Service resource my-app-kafka-external-bootstrap in namespace default to be addressable),tag(resource-namespace=default)] = 0
2022-03-04 01:55:17 WARN  AbstractOperator:532 - Reconciliation #1(watch) Kafka(default/my-app): Failed to reconcile
2022-03-04 01:55:17 DEBUG AbstractOperator:440 - Reconciliation #1(watch) Kafka(default/my-app): Lock lock::default::Kafka::my-app released

• see with_loadbalancer.log for full log

Custom Resource (not using loadbalancer)

---
apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: my-app
  namespace: default
spec:
  entityOperator:
    topicOperator: {}
    userOperator: {}
  kafka:
    config:
      default.replication.factor: 1
      inter.broker.protocol.version: "3.1"
      min.insync.replicas: 1
      offsets.topic.replication.factor: 1
      transaction.state.log.min.isr: 1
      transaction.state.log.replication.factor: 1
    listeners:
    - name: plain
      port: 9092
      tls: false
      type: internal
    - name: tls
      port: 9093
      tls: true
      type: internal
    replicas: 1
    storage:
      type: jbod
      volumes:
      - class: persistent-standard
        deleteClaim: true
        id: 0
        size: 1Gi
        type: persistent-claim
  zookeeper:
    replicas: 1
    storage:
      class: persistent-standard
      deleteClaim: false
      size: 10Gi
      type: persistent-claim

Logs (not using loadbalancer)

• see without_loadbalancer.log for full log

Additional context
I did review my Kubernetes Control Plane metrics and confirmed no latency issues or resource restrictions (CPU,Memory,Disk) issues in the cluster.

The example without the loadbalancer is to demonstrate that things can create successfully - and that this does seem to be an order of operations bug.

[scholzj]

This is not a bug. You platform did not created the loadbalancer and provided its address in its status. So Strimzi cannot proceed and create the Kafka brokers. When you list your services with kubectl get service -o wide you should check the EXTERNAL-IP column. It needs to have an IP address or a DNS address in it. If it doesn't and has something like <pending> then the load balancer is not provisioned by the platform.

[smarsh-tim]

Confirmed. The kube-vip provider for the cluster seems to have entered a faulty state - after refreshing (deleting the pods) the kube-vip DaemonSet, the External IPs assigned successfully.

I will move forward with digging into the kube-vip DaemonSet. Thank you @scholzj !