Mirrormaker replicating data 3 times

[smolina74]

hi,

1. I have two kafka clusters running in two different namespaces (source: santiago3 and target: santiago5)
   In every experiment I'm recreating the target kafka cluster and also deleting the "mm2-offset-syncs.target.internal" topic in the source side

2. I run the following kafkamirrormaker2 in santiago5 (target side)

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaMirrorMaker2
metadata:
  name: restore-e728055d-52de-4ca2-bcc4-a409a68aaa82
spec:
  clusters:
  - alias: source
    bootstrapServers: cluster-santiago3-kafka-bootstrap.santiago3:9092
  - alias: target
    bootstrapServers: cluster-santiago5-kafka-bootstrap:9092
  connectCluster: target
  livenessProbe:
    initialDelaySeconds: 25
    timeoutSeconds: 25
  logging:
    loggers:
      connect.root.logger.level: INFO
    type: inline
  mirrors:
  - checkpointConnector:
      config:
        replication.policy.class: io.strimzi.kafka.connect.mirror.IdentityReplicationPolicy
        replication.policy.separator: ''
    groupsPattern: .*
    sourceCluster: source
    sourceConnector:
      config:
        replication.policy.class: io.strimzi.kafka.connect.mirror.IdentityReplicationPolicy
        replication.policy.separator: ''
        sync.topic.acls.enabled: 'false'
    targetCluster: target
    topicsPattern: ^(?!.*backup.api).*
  readinessProbe:
    initialDelaySeconds: 25
    timeoutSeconds: 25
  version: 2.8.1

3. Then when I count the number of messages in the source side, for a particular topic, I get 39. But when I count the number of messages for the same topic in the target side, sometimes I'm getting 39, others 118 (same 39 messages replicated 3 times) and others 156 (same 39 messages replicated 4 times).

Any idea?

thanks,
Santiago

[scholzj]

I'm afraid it is not entirely clear what the full setup is and what exactly are you doing. Can you provide more step-by-step guide to reproduce it? I think that without knowing the full setup and all the steps, it is hard to understand what exactly are you doing.

[scholzj]

Do you also delete the PVCs? Because they would be recovered and they have the data.

[smolina74]

yes, I'm deleting PVCs also. Basically I'm running the following script for cleaning that environment:

oc delete kafkamirrormaker2 -n santiago5 --all
oc delete kafka -n santiago5 --all
oc delete kafkatopics -n santiago5 --all
oc delete pvc data-0-cluster-santiago5-kafka-0 data-0-cluster-santiago5-kafka-1 data-0-cluster-santiago5-kafka-2 data-cluster-santiago5-zookeeper-0 data-cluster-santiago5-zookeeper-1 data-cluster-santiago5-zookeeper-2 -n santiago5

[smolina74]

I found what could be the problem. It turns out that source kafka cluster (santiago3) had be used before also as a target, receiving stuff from another kafka cluster. Then, when the replication to santiago5 happened also mirror maker related topics, like this one "mirrormaker2-cluster-offsets", were copied and that what was causing the multiple replication issue. In my new version I've excluded those mirror maker related topics from the replication and the replication is working fine. Question, do you know if there is a way for prefixing all of those mirrormaker/kafka connect related topics so they can be easily identified?

[scholzj]

I don't think you can prefix them somehow easily. But you can probably configure the names for some of these topics individually. For example by adding something like this:

    offset.storage.topic: connect-cluster-offsets
    config.storage.topic: connect-cluster-configs
    status.storage.topic: connect-cluster-status

To the config section of the target cluster you might be able to configure the names used by the underlying Connect topic. There might be some other options in other places for some other topics as well (I do not have an exhaustive list I'm afraid).

[smolina74]

thanks