KafkaClusterTemplate for volume mounting got ignored when KafkaNodePoolTemplate is used

[shk3]

Hi, I'm trying to follow the documentation below to convert our TLS auth listener to a custom listener with out own client CA as truststore. The documentation asks us to mount the truststore as additional volumes via KafkaClusterTemplate, and we did exactly what is done in the documentation.
https://strimzi.io/docs/operators/0.46.0/configuring.html#configuring_customized_tls_client_authentication

---
apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
...
spec:
  kafka:
    template:
      pod:
        volumes:
          - name: mtls-clients-ca
            configMap:
              name: mtls-clients-ca
      kafkaContainer:
        volumeMounts:
          - name: mtls-clients-ca
            readOnly: true
            mountPath: /mnt/mtls-clients-ca

However, it never works, and the broker pods were not mounted with the new volumes at all even though the configs get successfully updated, so the broker pod ended up crashlooping.

We have some affinity and toleration rules that need to be added to pods via KafkaNodePoolTemplate. When I tried to add the volume mounting pod template and kafka containers to KafkaNodePoolTemplate instead, the mounting worked.

---
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaNodePool
metadata:
  name: pool-broker
  labels:
    strimzi.io/cluster: my-cluster
spec:
  replicas: ...
  roles:
    - broker
  resources:
    ...
  storage:
    ...
  template:
    pod:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchExpressions:
                  - key: strimzi.io/pool-name
                    operator: In
                    values:
                      - pool-broker
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                  - key: role
                    operator: In
                    values:
                      - some-group
      tolerations:
        - key: dedicated
          effect: NoSchedule
          operator: Equal
          value: some-group
      volumes:
        - name: mtls-clients-ca
          configMap:
            name: mtls-clients-ca
    kafkaContainer:
      volumeMounts:
        - name: mtls-clients-ca
          readOnly: true
          mountPath: /mnt/mtls-clients-ca

It seems to me that the existence of KafkaNodePoolTemplate has somehow made KafkaClusterTemplate ineffective.
I wonder if this was an intentional design choice. If so, can we add it in the documentation? The current behavior is a bit counter intuitive.

[scholzj]

You should not mix the template fields as there is no merging happening. So if you have the template in KafkaNodePool, the related things from the KAfka template will be ignored. It is described here: https://strimzi.io/docs/operators/latest/full/deploying.html#config-node-pools-str

[shk3]

Ah makes sense. Thanks!