Mirror Maker 2 Negative Offsets

[urton]

We recently started a migration from a traditional self-managed Apache Kafka cluster to Strimzi. We enabled group offset sync and something I'm struggling to fully understand is why the mirrored group has a current-offset matching (or close to matching) the source offset, but then the log-end-offset essentially starts over resulting in a negative lag. Is this just aesthetic? The lag is slowly moving closer to 0 but some of our negative lags are very "large". Several hundred million. I verified all the data has moved over, but I'm struggling on this lag issue.

Any help clearing this up is much appreciated!

[scholzj]

I'm not sure I follow what do you mean with negative lag here. Maybe you can provide more details or some examples?

[urton]

Thanks for the quick reply @scholzj. Example below.

[urton]

$ kafka-consumer-groups.sh --bootstrap-server my.strimzi.cluster:443 --command-config ${SSL_CLIENT_FILE} --describe --group my-group

Consumer group 'my-group' has no active members.

GROUP		TOPIC		PARTITION	CURRENT-OFFSET	LOG-END-OFFSET  LAG		CONSUMER-ID	HOST	CLIENT-ID
my-groug	my-topic	4		249358195	6481035		-242877160	-		-	-
my-group	my-topic	5		249479276	6628555		-242850721	-		-	-
my-group	my-topic	6		249307720	6382889		-242924831	-		-	-
my-group	my-topic	7		249333351	6447255		-242886096	-		-	-
my-group	my-topic	0		249328507	6304497		-243024010	-		-	-
my-group	my-topic	1		249340201	6286979		-243053222	-		-	-
my-group	my-topic	2		249339598	6384640		-242954958	-		-	-
my-group	my-topic	3		249336895	6344119		-242992776	-		-	-
my-group	my-topic	8		249368568	6390421		-242978147	-		-	-
my-group	my-topic	9		249319569	6379316		-242940253	-		-	-
my-group	my-topic	10		249339115	6388443		-242950672	-		-	-
my-group	my-topic	11		249478879	6341630		-243137249	-		-	-

$ kafka-consumer-groups.sh --bootstrap-server my.legacy.cluster:9092 --describe --group my-group

GROUP		TOPIC		PARTITION	CURRENT-OFFSET	LOG-END-OFFSET	LAG		CONSUMER-ID			HOST		CLIENT-ID
my-group	my-topic	9		252072134	252072134	0		consumer-1-xxxxx-8a985875be7a	/x.x.x.x	consumer-1
my-group	my-topic	10		252092284	252092284	0		consumer-1-xxxxx-8a985875be7a	/x.x.x.x	consumer-1
my-group	my-topic	11		252097489	252097489	0		consumer-1-xxxxx-8a985875be7a	/x.x.x.x	consumer-1
my-group	my-topic	0		252080774	252080774	0		consumer-1-xxxxx-c03960eb87e0	/x.x.x.x	consumer-1
my-group	my-topic	1		252092610	252092610	0		consumer-1-xxxxx-c03960eb87e0	/x.x.x.x	consumer-1
my-group	my-topic	2		252094190	252094190	0		consumer-1-xxxxx-c03960eb87e0	/x.x.x.x	consumer-1
my-group	my-topic	6		252060021	252060021	0		consumer-1-xxxxx-fb7b93c4241d	/x.x.x.x	consumer-1
my-group	my-topic	7		252088387	252088387	0		consumer-1-xxxxx-fb7b93c4241d	/x.x.x.x	consumer-1
my-group	my-topic	8		252124449	252124449	0		consumer-1-xxxxx-fb7b93c4241d	/x.x.x.x	consumer-1
my-group	my-topic	3		252090940	252090940	0		consumer-1-xxxxx-5232473666a7	/x.x.x.x	consumer-1
my-group	my-topic	4		252109115	252109115	0		consumer-1-xxxxx-5232473666a7	/x.x.x.x	consumer-1
my-group	my-topic	5		252095913	252095913	0		consumer-1-xxxxx-5232473666a7	/x.x.x.x	consumer-1

[urton]

source-legacy to target-strimzi is the mirror we are discussing.

[scholzj]

Hmm, that looks good to me. Not sure why does it not sync them properly. Might be some bug directly in Mirror Maker I guess.

[urton]

Dang I was hoping you'd have some magic config for me @scholzj . For reference, I opened an issue with Apache:
https://issues.apache.org/jira/browse/KAFKA-13562

[urton]

For anyone who encounters this, deleting the group, in the destination cluster with Kafka cli, and allowing the offset sync to recreate it fixed the negative lag we saw.

For "stale" groups (e.g. empty topic), I manually reset all partition offsets to-latest and this also fixed the issue.

[scholzj]

Great, thanks for sharing the solution 👍

[fvaleri]

@urton negative lag on target topic only happens when the partition is empty, and we start MM2 (KAFKA-12635). The theory is that the target CG offset is set to the same numerical value of the source partition (X) instead of 0. This is not updated until the actual target CG offset is greater than X.

[urton]

Hey @fvaleri! Funny you bring that up. So we have a theory, but I haven't been able to fully bake it. Like you mentioned, if the topic (or even just a partition in the topic) is empty, the negative lag will be there. But we mirrored several hundred non-empty topics as well that had the negative lag initially. My theory is if the offset sync is not enabled initially, but rather after the topics have started replicating with data, then this wouldn't happen (except on empty topics). I will update if/when I can test that. We did see another weird issue where certain groups don't get created in the target cluster, but that's only if the topics are empty or the offsets aren't getting set. Almost like MM2 knows it is a "stale" group.

[fvaleri]

@urton I just want to let you know that there is a candidate fix here: apache/kafka#11748.

[urton]

Thanks for the heads up @fvaleri, we are just now beginning our 2.8 to 3.1 migration, so I'll be sure to check this out when we can!