Facing issue while deploying Kafka MirrorMaker2 with TLS passthrough across clusters

[Harshitha-1996]

Hi Team,

We are deploying Kafka MirrorMaker2 using Strimzi to mirror data from an active Kafka cluster to a passive Kafka cluster, running in two separate Kubernetes environments.

We have enabled TLS passthrough at the gateway level, and communication between clusters is over the Kafka internal port 9093.

🧩 Environment
Strimzi version: 0.46.0
Kafka version: 4.0.0

KafkaMirrorMaker2 configuration:

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaMirrorMaker2
metadata:
  name: onap-strimzi-mm2-do6
  namespace: onap
  labels:
    app: strimzi-mirrormaker2
spec:
  version: 4.0.0
  replicas: 1
  connectCluster: target-cluster
  template:
    pod:
      imagePullSecrets:
        - name: onap-docker-registry-key
  clusters:
    - alias: source-cluster
      bootstrapServers: <domin-name>:9010
      tls:
        trustedCertificates:
          - secretName: do4-cluster-ca-cert
            certificate: ca.crt
      authentication:
        type: tls
        certificateAndKey:
          secretName: kafka-mm2-user-tls-do4
          certificate: user.crt
          key: user.key

    - alias: target-cluster
      bootstrapServers: <domin-name>:9010
      tls:
        trustedCertificates:
          - secretName: do6-cluster-ca-cert 
            certificate: ca.crt
      authentication:
        type: tls
        certificateAndKey:
          secretName: kafka-mm2-user-tls-do6
          certificate: user.crt
          key: user.key

  config:
    config.storage.replication.factor: 1
    offset.storage.replication.factor: 1
    status.storage.replication.factor: 1

  mirrors:
    - sourceCluster: source-cluster
      targetCluster: target-cluster
      topicsPattern: "."
      groupsPattern: "."
      sourceConnector:
        tasksMax: 3
        config:
          replication.factor: 1
          offset-syncs.topic.replication.factor: 1
          refresh.topics.interval.seconds: 2
          replication.policy.class: "org.apache.kafka.connect.mirror.IdentityReplicationPolicy"
          sync.topic.acls.enabled: "false"
          groupId: "mirrormaker2-cluster"
      checkpointConnector:
        tasksMax: 3
        config:
          checkpoints.topic.replication.factor: 1
          refresh.groups.interval.seconds: 2
          replication.policy.class: "org.apache.kafka.connect.mirror.IdentityReplicationPolicy"
          sync.group.offsets.enabled: "true"
      heartbeatConnector:
        config:
          refresh.groups.interval.seconds: 2
          replication.policy.class: "org.apache.kafka.connect.mirror.IdentityReplicationPolicy"
          sync.group.offsets.enabled: "true"
          topicsPattern: "."
          groupsPattern: "."

KafkaUser

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaUser
metadata:
  name: kafka-mm2-user-tls-do6
  namespace: onap
  labels:
    strimzi.io/cluster: onap-strimzi
spec:
  authentication:
    type: tls
  authorization:
    type: simple
    acls:
      - resource:
          type: topic
          name: mirrormaker2-cluster-offsets
          patternType: literal
        operations: ["Describe", "Read", "Write"]
      - resource:
          type: topic
          name: ""
          patternType: literal
        operations: ["Read", "Write", "Describe"]
      - resource:
          type: group
          name: ""
          patternType: literal
        operations: ["Read", "Write"]
        

Problem:
MirrorMaker2 pod fails to become ready. We see errors such as:

Pod Events:

  Warning  Unhealthy        4s    kubelet            Readiness probe failed: Get "http://100.90.194.101:15020/app-health/onap-strimzi-mm2-do6-mirrormaker2/readyz": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
  Warning  Unhealthy        0s    kubelet            Liveness probe failed: Get "http://100.90.194.101:15020/app-health/onap-strimzi-mm2-do6-mirrormaker2/livez": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

MirrorMaker2 Logs

2025-07-03 13:30:43 WARN  [DistributedHerder-connect-onap-strimzi-mm2-do6-mirrormaker2-0.onap-strimzi-mm2-do6-mirrormaker2.onap.svc:8083-1] RetryUtil:90 - Attempt 1 to list offsets for topic partitions resulted in RetriableException; retrying automatically. Reason: Timed out while waiting to get end offsets for topic 'mirrormaker2-cluster-offsets' on brokers at kafka-api-do6-dev.tnaplab.telekom.de:9010
org.apache.kafka.common.errors.TimeoutException: Timed out while waiting to get end offsets for topic 'mirrormaker2-cluster-offsets' on brokers at kafka-api-do6-dev.tnaplab.telekom.de:9010
Caused by: org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment. Call: listOffsets(api=LIST_OFFSETS)
2025-07-03 13:30:50 INFO  [qtp401891515-40] RestServer:57 - 127.0.0.6 - - [03/Jul/2025:13:30:40 +0000] "GET /health HTTP/1.1" 503 183 "-" "kube-probe/1.33" 10120
2025-07-03 13:30:53 INFO  [qtp401891515-35] RestServer:57 - 127.0.0.6 - - [03/Jul/2025:13:30:43 +0000] "GET /health HTTP/1.1" 503 183 "-" "kube-probe/1.33" 10004
2025-07-03 13:31:00 INFO  [qtp401891515-47] RestServer:57 - 127.0.0.6 - - [03/Jul/2025:13:30:50 +0000] "GET /health HTTP/1.1" 503 183 "-" "kube-probe/1.33" 10003
2025-07-03 13:31:03 INFO  [qtp401891515-40] RestServer:57 - 127.0.0.6 - - [03/Jul/2025:13:30:53 +0000] "GET /health HTTP/1.1" 503 183 "-" "kube-probe/1.33" 10002
2025-07-03 13:31:05 INFO  [qtp401891515-38] RestServer:57 - 127.0.0.6 - - [03/Jul/2025:13:30:55 +0000] "GET /health HTTP/1.1" 503 183 "-" "kube-probe/1.33" 10002
2025-07-03 13:31:08 INFO  [JettyShutdownThread] Server:650 - Stopped oejs.Server@7fae4d4a{STOPPING}[12.0.15,sto=60000]
2025-07-03 13:31:08 INFO  [connect-shutdown-hook] Connect:87 - Kafka Connect stopping
2025-07-03 13:31:08 INFO  [connect-shutdown-hook] RestServer:363 - Stopping REST server
2025-07-03 13:31:08 INFO  [JettyShutdownThread] Server:141 - Shutdown oejs.Server@7fae4d4a{STOPPING}[12.0.15,sto=60000]
2025-07-03 13:31:08 INFO  [connect-shutdown-hook] ServletContextHandler:1113 - Stopped oeje10s.ServletContextHandler@76c52298{ROOT,/,b=null,a=AVAILABLE,h=oeje10s.SessionHandler@bb9ab64{STOPPED}}
2025-07-03 13:31:13 INFO  [qtp401891515-39] RestServer:57 - 127.0.0.6 - - [03/Jul/2025:13:31:03 +0000] "GET /health HTTP/1.1" 503 183 "-" "kube-probe/1.33" 10004
2025-07-03 13:31:15 INFO  [qtp401891515-36] RestServer:57 - 127.0.0.6 - - [03/Jul/2025:13:31:05 +0000] "GET /health HTTP/1.1" 503 183 "-" "kube-probe/1.33" 10004
2025-07-03 13:31:15 INFO  [JettyShutdownThread] AbstractConnector:381 - Stopped http_8083@89c65d5{HTTP/1.1, (http/1.1)}{0.0.0.0:8083}
2025-07-03 13:31:15 INFO  [connect-shutdown-hook] RestServer:392 - REST server stopped
2025-07-03 13:31:15 INFO  [connect-shutdown-hook] DistributedHerder:854 - [Worker clientId=connect-onap-strimzi-mm2-do6-mirrormaker2-0.onap-strimzi-mm2-do6-mirrormaker2.onap.svc:8083, groupId=mirrormaker2-cluster] Herder stopping

Broker logs are

2025-07-03 13:29:43 INFO  [data-plane-kafka-request-handler-6] logger:307 - Principal = User:CN=kafka-mm2-user-tls-do6 is Denied operation = DESCRIBE_CONFIGS from host = 127.0.0.6 on resource = Topic:LITERAL:mirrormaker2-cluster-offsets for request = DescribeConfigs with resourceRefCount = 1 based on rule DefaultDeny
2025-07-03 13:31:43 INFO  [data-plane-kafka-request-handler-4] logger:307 - Principal = User:CN=kafka-mm2-user-tls-do6 is Denied operation = DESCRIBE_CONFIGS from host = 127.0.0.6 on resource = Topic:LITERAL:mirrormaker2-cluster-offsets for request = DescribeConfigs with resourceRefCount = 1 based on rule DefaultDeny

❓Questions / Request
Could this be due to missing ACLs for connect-cluster-* internal topics that MirrorMaker2 tries to create or describe?

Does MM2 require any additional internal topics ACLs that are not documented in Strimzi examples?

Could the readiness probe failure be due to MM2 not becoming healthy because of the timeout in offset retrieval?

[scholzj]

If you get ACL errors, you should certainly fix them in the first place. From what I remember from our docs and examples, MM2 needs quite a lot of ACL rights. I definitely remember more various rules than what your user has.

[Harshitha-1996]

Thanks for the suggestion let me summaries the scenarios
There are 2 strimzi kafka cluster running on 2 different k8s clusters and our use-case to use as active and backup cluster

strimzi version : 0.46.0
kafka version: 4.0.0
K8s version : 1.33

we are using k8s gatewayApi to connect these two kafka clusters

Scenario 1:

We are using plaintext and kafka listener 9094 is being used but it is exposed as a clusterIP ( connectivity through the gatewayApi )
MM2 is setup on target cluster
Below is the configuration of kafka MM2, kafka clusters and kafkauser for this setup and secrets for kafkauser

MM2 Configuration

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaMirrorMaker2
metadata:
  name: onap-strimzi-mm2-do5
  namespace: onap
  labels:
    app: strimzi-mirrormaker2
spec:
  version: 4.0.0  # <-- your supported newer version here
  replicas: 1
  connectCluster: "target-cluster"
  template:
    pod:
      imagePullSecrets:
        - name: onap-docker-registry-key
  clusters:
  - alias: "source-cluster"
    bootstrapServers: "<domin-name>:9010"
    authentication:
      type: scram-sha-512
      username: kafka-mm2-user-do2
      passwordSecret:
        secretName: kafka-mm2-user-do2
        password: password
    config:
      security.protocol: SASL_PLAINTEXT
      sasl.mechanism: SCRAM-SHA-512
      # value.converter: org.apache.kafka.connect.json.JsonConverter
      # key.converter: org.apache.kafka.connect.json.JsonConverter
     
            
  - alias: "target-cluster"
    bootstrapServers: "<domin-name>:9010"
    authentication:
      type: scram-sha-512
      username: kafka-mm2-user-do5
      passwordSecret:
        secretName: kafka-mm2-user-do5
        password: password
    config:
      security.protocol: SASL_PLAINTEXT
      sasl.mechanism: SCRAM-SHA-512
      # value.converter: org.apache.kafka.connect.json.JsonConverter
      # key.converter: org.apache.kafka.connect.json.JsonConverter
      config.storage.replication.factor: 1
      offset.storage.replication.factor: 1
      status.storage.replication.factor: 1

  mirrors:
  - sourceCluster: "source-cluster"
    targetCluster: "target-cluster"
    sourceConnector:
      tasksMax: 3
      config:
        replication.factor: 1
        offset-syncs.topic.replication.factor: 1
        sync.topic.acls.enabled: "true"
        refresh.topics.interval.seconds: 2
        replication.policy.class: org.apache.kafka.connect.mirror.IdentityReplicationPolicy
    checkpointConnector:
      tasksMax: 3
      config:
        checkpoints.topic.replication.factor: 1
        sync.group.offsets.enabled: "true"
        refresh.groups.interval.seconds: 2
        replication.policy.class: org.apache.kafka.connect.mirror.IdentityReplicationPolicy
        # transforms: dropPrefix
        # transforms.dropPrefix.type: org.apache.kafka.connect.transforms.RegexRouter
        # transforms.dropPrefix.regex: source-cluster\\.(.*)
        # transforms.dropPrefix.replacement: $1
    heartbeatConnector:
      config:
        replication.policy.class: org.apache.kafka.connect.mirror.IdentityReplicationPolicy
        sync.group.offsets.enabled: "true"
        refresh.groups.interval.seconds: 2
    topicsPattern: ".*"
    groupsPattern: ".*"

kafkauser configuration

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaUser
metadata:
  name: kafka-mm2-user-do5
  namespace: onap  # Change this if your namespace is different
  labels:
    strimzi.io/cluster: onap-strimzi  # Ensure this matches your Kafka cluster name
spec:
  authentication:
    type: scram-sha-512
  authorization:
    type: simple
    acls:
      # Allow full access to all topics (for replication)
      - resource:
          type: topic
          name: '*'
          patternType: literal
        operation: All
        host: '*'

      # Allow full access to all consumer groups (for offset syncing)
      - resource:
          type: group
          name: '*'
          patternType: literal
        operation: All
        host: '*'

      # Allow full cluster operations (needed for MM2 internal topics)
      - resource:
          type: cluster
          name: '*'
          patternType: literal
        operation: All
        host: '*'

      # Allow access to transactional IDs (if using transactions)
      - resource:
          type: transactionalId
          name: '*'
          patternType: literal
        operation: All
        host: '*
```'

kafka  configuration

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  annotations:
    argocd.argoproj.io/tracking-id: onap-strimzi-do5:kafka.strimzi.io/Kafka:onap/onap-strimzi
    strimzi.io/kraft: enabled
    strimzi.io/node-pools: enabled
  name: onap-strimzi
  namespace: onap
 
spec:
  cruiseControl:
    metricsConfig:
      type: jmxPrometheusExporter
      valueFrom:
        configMapKeyRef:
          key: cruisecontrol-metrics-config.yml
          name: onap-strimzi
    resources:
      limits:
        cpu: 2
        memory: 2Gi
      requests:
        cpu: 100m
        memory: 1Gi
    template:
      cruiseControlContainer:
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            drop:
            - ALL
            - CAP_NET_RAW
          readOnlyRootFilesystem: true
          runAsGroup: 1001
          runAsNonRoot: true
          runAsUser: 1001
      pod:
        imagePullSecrets:
        - name: onap-docker-registry-key
        securityContext:
          seccompProfile:
            type: RuntimeDefault
  entityOperator:
    template:
      pod:
        imagePullSecrets:
        - name: onap-docker-registry-key
        securityContext:
          seccompProfile:
            type: RuntimeDefault
      topicOperatorContainer:
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            drop:
            - ALL
            - CAP_NET_RAW
          readOnlyRootFilesystem: true
          runAsGroup: 1001
          runAsNonRoot: true
          runAsUser: 1001
      userOperatorContainer:
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            drop:
            - ALL
            - CAP_NET_RAW
          readOnlyRootFilesystem: true
          runAsGroup: 1001
          runAsNonRoot: true
          runAsUser: 1001
    topicOperator:
      resources:
        limits:
          cpu: 2
          memory: 2Gi
        requests:
          cpu: 100m
          memory: 1Gi
    userOperator:
      resources:
        limits:
          cpu: 2
          memory: 2Gi
        requests:
          cpu: 100m
          memory: 1Gi
  kafka:
    authorization:
      superUsers:
      - strimzi-kafka-admin
      type: simple
    config:
      auto.create.topics.enable: true
      default.replication.factor: 3
      inter.broker.protocol.version: 4.0.0
      log.message.format.version: 4.0.0
      min.insync.replicas: 2
      num.partitions: 6
      offsets.topic.replication.factor: 3
      transaction.state.log.min.isr: 2
      transaction.state.log.replication.factor: 3
    listeners:
    - authentication:
        type: scram-sha-512
      name: plain
      port: 9092
      tls: false
      type: internal
    - authentication:
        type: scram-sha-512
      configuration:
        brokers:
        - advertisedHost: kafka-api-do5-dev.example.de
          advertisedPort: 9000
          broker: 0
        - advertisedHost: kafka-api-do5-dev.example.de
          advertisedPort: 9001
          broker: 1
        - advertisedHost: kafka-api-do5-dev example.de
          advertisedPort: 9002
          broker: 2
      name: external
      port: 9094
      tls: false
      type: cluster-ip
    metricsConfig:
      type: jmxPrometheusExporter
      valueFrom:
        configMapKeyRef:
          key: kafka-metrics-config.yml
          name: onap-strimzi
    version: 4.0.0
  kafkaExporter:
    enableSaramaLogging: true
    groupRegex: .*
    livenessProbe:
      initialDelaySeconds: 15
      timeoutSeconds: 5
    logging: debug
    readinessProbe:
      initialDelaySeconds: 15
      timeoutSeconds: 5
    resources:
      limits:
        cpu: 5
        memory: 1.5Gi
      requests:
        cpu: 2
        memory: 600Mi
    template:
      container:
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            drop:
            - ALL
            - CAP_NET_RAW
          readOnlyRootFilesystem: true
          runAsGroup: 1001
          runAsNonRoot: true
          runAsUser: 1001
      pod:
        imagePullSecrets:
        - name: onap-docker-registry-key
        securityContext:
          seccompProfile:
            type: RuntimeDefault
    topicRegex: .*



**kafkauser secret**


apiVersion: v1
data:
  password: password
  sasl.jaas.config: < sasl encrypted data >
kind: Secret
metadata:
  name: kafka-mm2-user-do5
  namespace: onap
type: Opaque

------

apiVersion: v1
data:
  password: password
  sasl.jaas.config: <  sasl encrypted data >
kind: Secret
metadata:
  name: kafka-mm2-user-do2
  namespace: onap
type: Opaque

While using plaintext communication with above configuration its working fine. MM2 is able to mirror topics and messages from source kafka cluster to target kafka cluster

And we didn't face any ACL issue.

Scenario 2:

We are using TLS and for this kafka listener 9093 is configured ( clusterIP )
Connectivity is through gatwayApi
MM2 setup on target cluster
Below is the configuration of kafka MM2, kafka clusters and kafkauser for this setup and secrets for kafkauser

kafka configuration

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  annotations:
    argocd.argoproj.io/tracking-id: onap-strimzi-do6:kafka.strimzi.io/Kafka:onap/onap-strimzi
    strimzi.io/kraft: enabled
    strimzi.io/node-pools: enabled
  generation: 4
  labels:
    argocd.argoproj.io/instance: onap-strimzi-do6
  name: onap-strimzi
  namespace: onap
spec:
  cruiseControl:
    metricsConfig:
      type: jmxPrometheusExporter
      valueFrom:
        configMapKeyRef:
          key: cruisecontrol-metrics-config.yml
          name: onap-strimzi
    resources:
      limits:
        cpu: 2
        memory: 2Gi
      requests:
        cpu: 100m
        memory: 1Gi
    template:
      cruiseControlContainer:
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            drop:
            - ALL
            - CAP_NET_RAW
          readOnlyRootFilesystem: true
          runAsGroup: 1001
          runAsNonRoot: true
          runAsUser: 1001
      pod:
        imagePullSecrets:
        - name: onap-docker-registry-key
        securityContext:
          seccompProfile:
            type: RuntimeDefault
  entityOperator:
    template:
      pod:
        imagePullSecrets:
        - name: onap-docker-registry-key
        securityContext:
          seccompProfile:
            type: RuntimeDefault
      topicOperatorContainer:
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            drop:
            - ALL
            - CAP_NET_RAW
          readOnlyRootFilesystem: true
          runAsGroup: 1001
          runAsNonRoot: true
          runAsUser: 1001
      userOperatorContainer:
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            drop:
            - ALL
            - CAP_NET_RAW
          readOnlyRootFilesystem: true
          runAsGroup: 1001
          runAsNonRoot: true
          runAsUser: 1001
    topicOperator:
      resources:
        limits:
          cpu: 2
          memory: 2Gi
        requests:
          cpu: 100m
          memory: 1Gi
    userOperator:
      resources:
        limits:
          cpu: 2
          memory: 2Gi
        requests:
          cpu: 100m
          memory: 1Gi
  kafka:
    authorization:
      superUsers:
      - strimzi-kafka-admin
      type: simple
    config:
      auto.create.topics.enable: true
      default.replication.factor: 3
      inter.broker.protocol.version: 4.0.0
      log.message.format.version: 4.0.0
      min.insync.replicas: 2
      num.partitions: 6
      offsets.topic.replication.factor: 3
      transaction.state.log.min.isr: 2
      transaction.state.log.replication.factor: 3
    listeners:
    - authentication:
        type: tls
      configuration:
        brokers:
        - advertisedHost: kafka-api-do6-dev.example.de
          advertisedPort: 9000
          broker: 0
        - advertisedHost: kafka-api-do6-dev.example.de
          advertisedPort: 9001
          broker: 1
        - advertisedHost: kafka-api-do6-dev.example.de
          advertisedPort: 9002
          broker: 2
      name: internal
      port: 9093
      tls: true
      type: cluster-ip
    metricsConfig:
      type: jmxPrometheusExporter
      valueFrom:
        configMapKeyRef:
          key: kafka-metrics-config.yml
          name: onap-strimzi
    version: 4.0.0
  kafkaExporter:
    enableSaramaLogging: true
    groupRegex: .*
    livenessProbe:
      initialDelaySeconds: 15
      timeoutSeconds: 5
    logging: debug
    readinessProbe:
      initialDelaySeconds: 15
      timeoutSeconds: 5
    resources:
      limits:
        cpu: 5
        memory: 1.5Gi
      requests:
        cpu: 2
        memory: 600Mi
    template:
      container:
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            drop:
            - ALL
            - CAP_NET_RAW
          readOnlyRootFilesystem: true
          runAsGroup: 1001
          runAsNonRoot: true
          runAsUser: 1001
      pod:
        imagePullSecrets:
        - name: onap-docker-registry-key
        securityContext:
          seccompProfile:
            type: RuntimeDefault
    topicRegex: .*

MM2 Configuration

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaMirrorMaker2
metadata:
  name: onap-strimzi-mm2-do6
  namespace: onap
  labels:
    app: strimzi-mirrormaker2
spec:
  version: 4.0.0
  replicas: 1
  connectCluster: target-cluster
  template:
    pod:
      imagePullSecrets:
        - name: onap-docker-registry-key
      # livenessProbe: null
      # readinessProbe: null
  clusters:
    - alias: source-cluster
      bootstrapServers: kafka-api-do4-dev.example:9010
      tls:
        trustedCertificates:
          - secretName: do4-cluster-ca-cert 
            certificate: ca.crt
      authentication:
        type: tls
        certificateAndKey:
          secretName: kafka-mm2-user-tls-do4
          certificate: user.crt
          key: user.key


    - alias: target-cluster
      bootstrapServers: kafka-api-do6-dev.example:9010
      tls:
        trustedCertificates:
          - secretName: do6-cluster-ca-cert
            certificate: ca.crt
      authentication:
        type: tls
        certificateAndKey:
          secretName: kafka-mm2-user-tls-do6
          certificate: user.crt
          key: user.key
        # value.converter: org.apache.kafka.connect.json.JsonConverter
        # key.converter: org.apache.kafka.connect.json.JsonConverter
      config:
        config.storage.replication.factor: 1
        offset.storage.replication.factor: 1
        status.storage.replication.factor: 1
  mirrors:
    - sourceCluster: source-cluster
      targetCluster: target-cluster
      topicsPattern: ".*"
      groupsPattern: ".*"
      sourceConnector:
        tasksMax: 3
        config:
          replication.factor: 1
          offset-syncs.topic.replication.factor: 1
          refresh.topics.interval.seconds: 2
          replication.policy.class: "org.apache.kafka.connect.mirror.IdentityReplicationPolicy"
          sync.topic.acls.enabled: "false"
          groupId: "mirrormaker2-cluster"
      checkpointConnector:
        tasksMax: 3
        config:
          checkpoints.topic.replication.factor: 1
          refresh.groups.interval.seconds: 2
          replication.policy.class: "org.apache.kafka.connect.mirror.IdentityReplicationPolicy"
          sync.group.offsets.enabled: "true"
      heartbeatConnector:
        config:
          refresh.groups.interval.seconds: 2
          replication.policy.class: "org.apache.kafka.connect.mirror.IdentityReplicationPolicy"
          sync.group.offsets.enabled: "true"
      topicsPattern: ".*"
      groupsPattern: ".*"

Kafakuser configuration


apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaUser
metadata:
  name: kafka-mm2-user-tls-do6
  namespace: onap
  labels:
    strimzi.io/cluster: onap-strimzi
spec:
  authentication:
    type: tls
  authorization:
    type: simple
    acls:
      # Allow reading and describing all topics
      - resource:
          type: topic
          name: mirrormaker2-cluster-offsets
          patternType: literal
        operations:
          - Read
          - Write
          - Describe
          
      - resource:
          type: topic
          name: "*"
          patternType: literal
        operations:
          - Read
          - Describe

      # Allow writing and creating topics (used on target cluster)
      - resource:
          type: topic
          name: "*"
          patternType: literal
        operations:
          - Write
          - Create
          - Describe

      # Allow access to MM2's group (consumer group)
      - resource:
          type: group
          name: "mirrormaker2-cluster"
          patternType: literal
        operations:
          - Read
          - Describe

      # Allow access to MM2 internal topics (heartbeats, checkpoints, offsets)
      - resource:
          type: topic
          name: "mirrormaker2-cluster"
          patternType: prefix
        operations:
          - Read
          - Write
          - Describe

      # Optional: allow access to cluster-level operations
      - resource:
          type: cluster
        operations:
          - Create
          - DescribeConfigs
          - AlterConfigs

logs of broker

INFO  [data-plane-kafka-request-handler-6] logger:307 - Principal = User:CN=kafka-mm2-user-tls-do6 is Denied operation = DESCRIBE_CONFIGS from host = 127.0.0.6 on resource = Topic:LITERAL:mirrormaker2-cluster-offsets for request = DescribeConfigs with resourceRefCount = 1 based on rule DefaultDeny
 INFO  [data-plane-kafka-request-handler-4] logger:307 - Principal = User:CN=kafka-mm2-user-tls-do6 is Denied operation = DESCRIBE_CONFIGS from host = 127.0.0.6 on resource = Topic:LITERAL:mirrormaker2-cluster-offsets for request = DescribeConfigs with resourceRefCount = 1 based on rule DefaultDeny

• But we are getting above ACL error not sure why this error is coming in SCENARIO 2 only ( when using TLS ),
  we didn't observed any TLS error in Brokers logs or in MM2 logs
• while in SCENARIO 1 ( plaintext with SCRAM-SH-512 )it is working fine no ACL error

Please let me know if any configuration update is required and if you can share kafka TLS user example that is to be used?
Would you please look into the issue and extend your support?
Your prompt and timely support would be highly appreciated.
Many thanks in advance.

[scholzj]

It is badly formatted, so the Kafka configurations are not readable at all. But if you get ACL errors, you have ACL issues you have to fix. That is not directly related to TLS or SCRAM. Just check the broker logs to see what right are you missing, make sure it has the right user, and add the rights.

[Harshitha-1996]

Thanks For the suggestion and apologies for bad format ..
Here is the kafka configuration

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  annotations:
    argocd.argoproj.io/tracking-id: onap-strimzi-do6:kafka.strimzi.io/Kafka:onap/onap-strimzi
    strimzi.io/kraft: enabled
    strimzi.io/node-pools: enabled
  generation: 4
  labels:
    argocd.argoproj.io/instance: onap-strimzi-do6
  name: onap-strimzi
  namespace: onap
spec:
  cruiseControl:
    metricsConfig:
      type: jmxPrometheusExporter
      valueFrom:
        configMapKeyRef:
          key: cruisecontrol-metrics-config.yml
          name: onap-strimzi
    resources:
      limits:
        cpu: 2
        memory: 2Gi
      requests:
        cpu: 100m
        memory: 1Gi
    template:
      cruiseControlContainer:
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            drop:
            - ALL
            - CAP_NET_RAW
          readOnlyRootFilesystem: true
          runAsGroup: 1001
          runAsNonRoot: true
          runAsUser: 1001
      pod:
        imagePullSecrets:
        - name: onap-docker-registry-key
        securityContext:
          seccompProfile:
            type: RuntimeDefault
  entityOperator:
    template:
      pod:
        imagePullSecrets:
        - name: onap-docker-registry-key
        securityContext:
          seccompProfile:
            type: RuntimeDefault
      topicOperatorContainer:
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            drop:
            - ALL
            - CAP_NET_RAW
          readOnlyRootFilesystem: true
          runAsGroup: 1001
          runAsNonRoot: true
          runAsUser: 1001
      userOperatorContainer:
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            drop:
            - ALL
            - CAP_NET_RAW
          readOnlyRootFilesystem: true
          runAsGroup: 1001
          runAsNonRoot: true
          runAsUser: 1001
    topicOperator:
      resources:
        limits:
          cpu: 2
          memory: 2Gi
        requests:
          cpu: 100m
          memory: 1Gi
    userOperator:
      resources:
        limits:
          cpu: 2
          memory: 2Gi
        requests:
          cpu: 100m
          memory: 1Gi
  kafka:
    authorization:
      superUsers:
      - strimzi-kafka-admin
      type: simple
    config:
      auto.create.topics.enable: true
      default.replication.factor: 3
      inter.broker.protocol.version: 4.0.0
      log.message.format.version: 4.0.0
      min.insync.replicas: 2
      num.partitions: 6
      offsets.topic.replication.factor: 3
      transaction.state.log.min.isr: 2
      transaction.state.log.replication.factor: 3
    listeners:
    - authentication:
        type: tls
      configuration:
        brokers:
        - advertisedHost: kafka-api-do6-dev.example.de
          advertisedPort: 9000
          broker: 0
        - advertisedHost: kafka-api-do6-dev.example.de
          advertisedPort: 9001
          broker: 1
        - advertisedHost: kafka-api-do6-dev.example.de
          advertisedPort: 9002
          broker: 2
      name: internal
      port: 9093
      tls: true
      type: cluster-ip
    metricsConfig:
      type: jmxPrometheusExporter
      valueFrom:
        configMapKeyRef:
          key: kafka-metrics-config.yml
          name: onap-strimzi
    version: 4.0.0
  kafkaExporter:
    enableSaramaLogging: true
    groupRegex: .*
    livenessProbe:
      initialDelaySeconds: 15
      timeoutSeconds: 5
    logging: debug
    readinessProbe:
      initialDelaySeconds: 15
      timeoutSeconds: 5
    resources:
      limits:
        cpu: 5
        memory: 1.5Gi
      requests:
        cpu: 2
        memory: 600Mi
    template:
      container:
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            drop:
            - ALL
            - CAP_NET_RAW
          readOnlyRootFilesystem: true
          runAsGroup: 1001
          runAsNonRoot: true
          runAsUser: 1001
      pod:
        imagePullSecrets:
        - name: onap-docker-registry-key
        securityContext:
          seccompProfile:
            type: RuntimeDefault
    topicRegex: .*

At User level we are giving ACL rights below is the kafkauser configuration
We are giving rights for all the topics and for all groups
And as per broker logs we are giving specific topic permission as well for the correct user

Note: Below is the similar ACL permissions for NON-TLS kafka user ( authentication: type: scram-sha-512 ) is working but when we use the same acl rules for kafka TLS-USER we are getting ACL'S Error in broker.

Kafakuser configuration

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaUser
metadata:
  name: kafka-mm2-user-tls-do6
  namespace: onap
  labels:
    strimzi.io/cluster: onap-strimzi
spec:
  authentication:
    type: tls
  authorization:
    type: simple
    acls:
      # Allow reading and describing all topics
      - resource:
          type: topic
          name: mirrormaker2-cluster-offsets
          patternType: literal
        operations:
          - Read
          - Write
          - Describe
          
      - resource:
          type: topic
          name: "*"
          patternType: literal
        operations:
          - Read
          - Describe

      # Allow writing and creating topics (used on target cluster)
      - resource:
          type: topic
          name: "*"
          patternType: literal
        operations:
          - Write
          - Create
          - Describe

      # Allow access to MM2's group (consumer group)
      - resource:
          type: group
          name: "mirrormaker2-cluster"
          patternType: literal
        operations:
          - Read
          - Describe

      # Allow access to MM2 internal topics (heartbeats, checkpoints, offsets)
      - resource:
          type: topic
          name: "mirrormaker2-cluster"
          patternType: prefix
        operations:
          - Read
          - Write
          - Describe

      # Optional: allow access to cluster-level operations
      - resource:
          type: cluster
        operations:
          - Create
          - DescribeConfigs
          - AlterConfigs

Can you please look once and check this configuration if this configuration looks ok .Please share your suggestions/insights for this configuration and if we need any other additional configuration while using kafka TLS user.

[scholzj]

Well, as I said, if you get ACL errors, it means you connected and authenticated just fine and you are just missing the ACL rights. So you need to add the ACL rights. Mirror Maker does not care whether you connected with mTLS or with SCRAM. It does the same in both cases. So the ACL rights need to be the same in both cases.

[Harshitha-1996]

[scholzj]

I do not know what this means, sorry. Some networking issues? Misconfiguration of the access to (one of) the Kafka clusters? Not sure, just some ideas.

[Harshitha-1996]

Thank You so much as per your suggestion we successfully resolved ACL issue by giving proper ACL rights to kafkauser.
Thank you