Kafka pods not deploying with anti affinity configuration

[davehouser1]

Problem:
When trying to deploy pods in a required anti-affinity or preferred anti-affinity configuration, kafka pods dont deploy.

Trouble shooting:

• Tried deploying as a "requiredDuringSchedulingIgnoredDuringExecution" configuration. Results: zookeeper will deploy, so will services, but kafka pods do not deploy. Kubernetes events show the following over and over and over again:

FailedCreate statefulset/my-cluster-kafka create Pod my-cluster-kafka-0 in StatefulSet my-cluster-kafka failed error: Pod "my-cluster-kafka-0" is invalid: spec.affinity.podAntiAffinity.requiredDuringSchedulingIgnoredDuringExecution[0].matchExpressions.matchExpressions[0].operator: Invalid value: "": not a valid selector operator

• Tried deploying as a "preferredDuringSchedulingIgnoredDuringExecution" configuration. Results: Kubernetes does nothing and shows no events (I assume a misconfiguration). Any recommendations for checking scheduling logs some other way?

• Followed instructions outlined here. The instructions give an example for affinity, but not anti-affinity. Also the instructions do not detail how to configure preferred anti affinity, only required anti affinity (they have different syntax) is preferred supported?

• Read this blog post, but as other discussions pointed out this post is 3 years old, so I assume its syntax or instructions are not valid?

• Read through this on syntax. I am not using openshift, but I believe the syntax is the same for vanilla Kubernetes.

Manifests:

• This is for the required anti affinity:

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: my-cluster
  labels:
    app: kafka
spec:
  kafka:
    affinity:
      podAntiAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: app 
              operator: In
              values:
              - kafka
            topologyKey: "kubernetes.io/hostname"   
    replicas: 3
    listeners:
      - name: plain
        port: 9092
        type: internal
        tls: false
      - name: tls 
        port: 9093
        type: internal
        tls: true
        authentication:
          type: tls 
      - name: external
        port: 9094
        type: loadbalancer
        tls: false
    storage:
      type: jbod
      volumes:
      - id: 0
        type: persistent-claim
        size: 10Gi
        class: unity-iscsi
        deleteClaim: false
    config:
      offsets.topic.replication.factor: 1
      transaction.state.log.replication.factor: 1
      transaction.state.log.min.isr: 1
  zookeeper:
    replicas: 3
    storage:
      type: persistent-claim
      size: 10Gi
      class: unity-iscsi
      deleteClaim: false
  entityOperator:
    topicOperator: {}
    userOperator: {}

• This is for the preferred anti affinity:

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: my-cluster
  labels:
    app: kafka
spec:
  kafka:
    affinity:
      podAntiAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 1
        podAffinityTerm:
          labelSelector:
            matchExpressions:
            - key: app 
              operator: In
              values:
              - kafka
          topologyKey: "kubernetes.io/hostname"   
    replicas: 3
    listeners:
      - name: plain
        port: 9092
        type: internal
        tls: false
      - name: tls 
        port: 9093
        type: internal
        tls: true
        authentication:
          type: tls 
      - name: external
        port: 9094
        type: loadbalancer
        tls: false
    storage:
      type: jbod
      volumes:
      - id: 0
        type: persistent-claim
        size: 10Gi
        class: unity-iscsi
        deleteClaim: false
    config:
      offsets.topic.replication.factor: 1
      transaction.state.log.replication.factor: 1
      transaction.state.log.min.isr: 1
  zookeeper:
    replicas: 3
    storage:
      type: persistent-claim
      size: 10Gi
      class: unity-iscsi
      deleteClaim: false
  entityOperator:
    topicOperator: {}
    userOperator: {}

[davehouser1]

Hmmm, after comparing required to preferred I noticed an indentation difference

      podAntiAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:

      podAntiAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:

When adjusting indentation for preferred, this allowed the pods to deploy properly.

[scholzj]

You have the affinity in the wrong place in the Kafka CR. Follow the docs for the right placement - it should be in .spec.kafka.template.... The blog post is 3 years old and not everything is valid anymore.