Under replicated partitions issues for 1 partition even though we have a healthy kafka cluster #7297

dbaarorapr · 2022-09-09T03:19:56Z

dbaarorapr
Sep 9, 2022

we have a situation where our kafka cluster is healthy and none of the brokers have any issues

But then one of our partitions in kafka strimzi which was fine is suddenly not able to do replication and as shown below ,it has been stuck at this state for the past 12 hours and need help

version of kafka is

[kafka@site-kafka-kafka-0 kafka]$ bin/kafka-dump-log.sh --version
2.8.0 (Commit:ebb1d6e21cc92130)
[kafka@site-kafka-kafka-0 kafka]$

version of strimzi is 0.26.1

[kafka@site-kafka-zookeeper-0 kafka]$ bin/kafka-topics.sh --describe --zookeeper localhost:12181 --under-replicated-partitions
Topic: acs2.expired Partition: 56 Leader: 3 Replicas: 3,4,5 Isr: 3
[kafka@site-kafka-zookeeper-0 kafka]$

Below is the describe on that Topic

[kafka@site-kafka-zookeeper-0 kafka]$ bin/kafka-topics.sh --describe  --zookeeper localhost:12181 --topic acs2.expired
Topic: acs2.expired	TopicId: wGKgqs5vRR2LgwKS8HTovg	PartitionCount: 72	ReplicationFactor: 3	Configs: retention.ms=259200000,segment.bytes=1073741824
	Topic: acs2.expired	Partition: 0	Leader: 4	Replicas: 4,6,7	Isr: 7,4,6
	Topic: acs2.expired	Partition: 1	Leader: 5	Replicas: 5,7,2	Isr: 7,2,5
	Topic: acs2.expired	Partition: 2	Leader: 3	Replicas: 3,2,1	Isr: 3,1,2
	Topic: acs2.expired	Partition: 3	Leader: 8	Replicas: 8,1,0	Isr: 0,1,8
	Topic: acs2.expired	Partition: 4	Leader: 6	Replicas: 6,0,4	Isr: 0,4,6
	Topic: acs2.expired	Partition: 5	Leader: 7	Replicas: 7,4,5	Isr: 7,4,5
	Topic: acs2.expired	Partition: 6	Leader: 2	Replicas: 2,5,3	Isr: 3,2,5
	Topic: acs2.expired	Partition: 7	Leader: 1	Replicas: 1,3,8	Isr: 3,1,8
	Topic: acs2.expired	Partition: 8	Leader: 0	Replicas: 0,8,6	Isr: 0,8,6
	Topic: acs2.expired	Partition: 9	Leader: 4	Replicas: 4,1,0	Isr: 0,1,4
	Topic: acs2.expired	Partition: 10	Leader: 5	Replicas: 5,0,4	Isr: 0,4,5
	Topic: acs2.expired	Partition: 11	Leader: 3	Replicas: 3,4,5	Isr: 3,4,5
	Topic: acs2.expired	Partition: 12	Leader: 8	Replicas: 8,5,3	Isr: 3,8,5
	Topic: acs2.expired	Partition: 13	Leader: 6	Replicas: 6,3,8	Isr: 3,8,6
	Topic: acs2.expired	Partition: 14	Leader: 7	Replicas: 7,8,6	Isr: 7,8,6
	Topic: acs2.expired	Partition: 15	Leader: 2	Replicas: 2,6,7	Isr: 7,2,6
	Topic: acs2.expired	Partition: 16	Leader: 1	Replicas: 1,7,2	Isr: 7,1,2
	Topic: acs2.expired	Partition: 17	Leader: 0	Replicas: 0,2,1	Isr: 0,1,2
	Topic: acs2.expired	Partition: 18	Leader: 4	Replicas: 4,3,6	Isr: 3,4,6
	Topic: acs2.expired	Partition: 19	Leader: 5	Replicas: 5,8,7	Isr: 7,8,5
	Topic: acs2.expired	Partition: 20	Leader: 3	Replicas: 3,6,2	Isr: 3,2,6
	Topic: acs2.expired	Partition: 21	Leader: 8	Replicas: 8,7,1	Isr: 7,1,8
	Topic: acs2.expired	Partition: 22	Leader: 6	Replicas: 6,2,0	Isr: 0,2,6
	Topic: acs2.expired	Partition: 23	Leader: 7	Replicas: 7,1,4	Isr: 7,1,4
	Topic: acs2.expired	Partition: 24	Leader: 2	Replicas: 2,0,5	Isr: 0,2,5
	Topic: acs2.expired	Partition: 25	Leader: 1	Replicas: 1,4,3	Isr: 3,1,4
	Topic: acs2.expired	Partition: 26	Leader: 0	Replicas: 0,5,8	Isr: 0,8,5
	Topic: acs2.expired	Partition: 27	Leader: 4	Replicas: 4,7,1	Isr: 7,1,4
	Topic: acs2.expired	Partition: 28	Leader: 5	Replicas: 5,2,0	Isr: 0,2,5
	Topic: acs2.expired	Partition: 29	Leader: 3	Replicas: 3,1,4	Isr: 3,1,4
	Topic: acs2.expired	Partition: 30	Leader: 8	Replicas: 8,0,5	Isr: 0,8,5
	Topic: acs2.expired	Partition: 31	Leader: 6	Replicas: 6,4,3	Isr: 3,4,6
	Topic: acs2.expired	Partition: 32	Leader: 7	Replicas: 7,5,8	Isr: 7,8,5
	Topic: acs2.expired	Partition: 33	Leader: 2	Replicas: 2,3,6	Isr: 3,2,6
	Topic: acs2.expired	Partition: 34	Leader: 1	Replicas: 1,8,7	Isr: 7,1,8
	Topic: acs2.expired	Partition: 35	Leader: 0	Replicas: 0,6,2	Isr: 0,2,6
	Topic: acs2.expired	Partition: 36	Leader: 4	Replicas: 4,0,5	Isr: 0,4,5
	Topic: acs2.expired	Partition: 37	Leader: 5	Replicas: 5,4,3	Isr: 3,4,5
	Topic: acs2.expired	Partition: 38	Leader: 3	Replicas: 3,5,8	Isr: 3,8,5
	Topic: acs2.expired	Partition: 39	Leader: 8	Replicas: 8,3,6	Isr: 3,8,6
	Topic: acs2.expired	Partition: 40	Leader: 6	Replicas: 6,8,7	Isr: 7,8,6
	Topic: acs2.expired	Partition: 41	Leader: 7	Replicas: 7,6,2	Isr: 7,2,6
	Topic: acs2.expired	Partition: 42	Leader: 2	Replicas: 2,7,1	Isr: 7,1,2
	Topic: acs2.expired	Partition: 43	Leader: 1	Replicas: 1,2,0	Isr: 0,1,2
	Topic: acs2.expired	Partition: 44	Leader: 0	Replicas: 0,1,4	Isr: 0,1,4
	Topic: acs2.expired	Partition: 45	Leader: 4	Replicas: 4,6,7	Isr: 7,4,6
	Topic: acs2.expired	Partition: 46	Leader: 5	Replicas: 5,7,2	Isr: 7,2,5
	Topic: acs2.expired	Partition: 47	Leader: 3	Replicas: 3,2,1	Isr: 3,1,2
	Topic: acs2.expired	Partition: 48	Leader: 8	Replicas: 8,1,0	Isr: 0,1,8
	Topic: acs2.expired	Partition: 49	Leader: 6	Replicas: 6,0,4	Isr: 0,4,6
	Topic: acs2.expired	Partition: 50	Leader: 7	Replicas: 7,4,5	Isr: 7,4,5
	Topic: acs2.expired	Partition: 51	Leader: 2	Replicas: 2,5,3	Isr: 3,2,5
	Topic: acs2.expired	Partition: 52	Leader: 1	Replicas: 1,3,8	Isr: 3,1,8
	Topic: acs2.expired	Partition: 53	Leader: 0	Replicas: 0,8,6	Isr: 0,8,6
	Topic: acs2.expired	Partition: 54	Leader: 4	Replicas: 4,1,0	Isr: 0,1,4
	Topic: acs2.expired	Partition: 55	Leader: 5	Replicas: 5,0,4	Isr: 0,4,5
	Topic: acs2.expired	Partition: 56	Leader: 3	Replicas: 3,4,5	Isr: 3
	Topic: acs2.expired	Partition: 57	Leader: 8	Replicas: 8,5,3	Isr: 3,8,5
	Topic: acs2.expired	Partition: 58	Leader: 6	Replicas: 6,3,8	Isr: 3,8,6
	Topic: acs2.expired	Partition: 59	Leader: 7	Replicas: 7,8,6	Isr: 7,8,6
	Topic: acs2.expired	Partition: 60	Leader: 2	Replicas: 2,6,7	Isr: 7,2,6
	Topic: acs2.expired	Partition: 61	Leader: 1	Replicas: 1,7,2	Isr: 7,1,2
	Topic: acs2.expired	Partition: 62	Leader: 0	Replicas: 0,2,1	Isr: 0,1,2
	Topic: acs2.expired	Partition: 63	Leader: 4	Replicas: 4,5,3	Isr: 3,4,5
	Topic: acs2.expired	Partition: 64	Leader: 5	Replicas: 5,3,8	Isr: 3,8,5
	Topic: acs2.expired	Partition: 65	Leader: 3	Replicas: 3,8,6	Isr: 3,8,6
	Topic: acs2.expired	Partition: 66	Leader: 8	Replicas: 8,6,7	Isr: 7,8,6
	Topic: acs2.expired	Partition: 67	Leader: 6	Replicas: 6,7,2	Isr: 7,2,6
	Topic: acs2.expired	Partition: 68	Leader: 7	Replicas: 7,2,1	Isr: 7,1,2
	Topic: acs2.expired	Partition: 69	Leader: 2	Replicas: 2,1,0	Isr: 0,1,2
	Topic: acs2.expired	Partition: 70	Leader: 1	Replicas: 1,0,4	Isr: 0,1,4
	Topic: acs2.expired	Partition: 71	Leader: 0	Replicas: 0,4,5	Isr: 0,4,5
[kafka@site-kafka-zookeeper-0 kafka]$

and kafka cluster is perfectly healthy

site-kafka-dr-kafka-0                                          1/1     Running   0          28d
site-kafka-dr-kafka-1                                          1/1     Running   0          28d
site-kafka-dr-kafka-2                                          1/1     Running   0          2d10h
site-kafka-dr-kafka-3                                          1/1     Running   0          169m
site-kafka-dr-kafka-4                                          1/1     Running   0          28d
site-kafka-dr-kafka-5                                          1/1     Running   0          2d
site-kafka-dr-kafka-6                                          1/1     Running   0          28d
site-kafka-dr-kafka-7                                          1/1     Running   0          28d
site-kafka-dr-kafka-8                                          1/1     Running   0          28d
site-kafka-dr-kafka-exporter-849d56b588-rrj4w                  1/1     Running   0          28d
site-kafka-dr-zookeeper-0                                      1/1     Running   0          28d
site-kafka-dr-zookeeper-1                                      1/1     Running   0          169m
site-kafka-dr-zookeeper-2                                      1/1     Running   0          28d
site-kafka-entity-operator-f6b88b976-r9zbp                     3/3     Running   0          28d
site-kafka-kafka-0                                             1/1     Running   0          28d
site-kafka-kafka-1                                             1/1     Running   0          28d
site-kafka-kafka-2                                             1/1     Running   0          28d
site-kafka-kafka-3                                             1/1     Running   0          28d
site-kafka-kafka-4                                             1/1     Running   0          10h
site-kafka-kafka-5                                             1/1     Running   0          10h
site-kafka-kafka-6                                             1/1     Running   0          10m
site-kafka-kafka-7                                             1/1     Running   0          28d
site-kafka-kafka-8                                             1/1     Running   0          2d
site-kafka-kafka-exporter-74c5fbd684-5lrb8                     1/1     Running   0          28d
site-kafka-mm2-dr-us-ashburn-1-mirrormaker2-6c99786fbd-lrmfx   1/1     Running   6          2d1h
site-kafka-mm2-dr-us-ashburn-1-mirrormaker2-6c99786fbd-tt4pw   1/1     Running   0          2d10h
site-kafka-zookeeper-0                                         1/1     Running   0          28d
site-kafka-zookeeper-1                                         1/1     Running   0          28d
site-kafka-zookeeper-2                                         1/1     Running   0          2d10h


NAME            DESIRED KAFKA REPLICAS   DESIRED ZK REPLICAS   READY   WARNINGS
site-kafka      9                        3                     True    
site-kafka-dr   9                        3                     True

i see below ib broker logs

2022-09-06 17:06:46,167 WARN [Partition acs2.expired-56 broker=3] Failed to update ISR to PendingExpandIsr(isr=Set(3), newInSyncReplicaId=5) due to unexpected UNKNOWN_SERVER_ERROR. Retrying. (kafka.cluster.Partition) [broker-3-to-controller-send-thread]
2022-09-06 17:06:46,168 ERROR [broker-3-to-controller] Uncaught error in request completion: (org.apache.kafka.clients.NetworkClient) [broker-3-to-controller-send-thread]
java.lang.IllegalStateException: Failed to enqueue ISR change state LeaderAndIsr(leader=3, leaderEpoch=113, isr=List(3, 5), zkVersion=203) for partition acs2.expired-56
	at kafka.cluster.Partition.sendAlterIsrRequest(Partition.scala:1374)
	at kafka.cluster.Partition.$anonfun$handleAlterIsrResponse$1(Partition.scala:1408)
	at kafka.cluster.Partition.handleAlterIsrResponse(Partition.scala:1387)
	at kafka.cluster.Partition.$anonfun$sendAlterIsrRequest$1(Partition.scala:1365)
	at kafka.cluster.Partition.$anonfun$sendAlterIsrRequest$1$adapted(Partition.scala:1365)
	at kafka.server.DefaultAlterIsrManager.$anonfun$handleAlterIsrResponse$8(AlterIsrManager.scala:262)
	at kafka.server.DefaultAlterIsrManager.$anonfun$handleAlterIsrResponse$8$adapted(AlterIsrManager.scala:259)
	at scala.collection.immutable.List.foreach(List.scala:333)
	at kafka.server.DefaultAlterIsrManager.handleAlterIsrResponse(AlterIsrManager.scala:259)
	at kafka.server.DefaultAlterIsrManager$$anon$1.onComplete(AlterIsrManager.scala:179)

dbaarorapr · 2022-09-09T13:58:05Z

dbaarorapr
Sep 9, 2022
Author

restart of the leader broker helped solve this

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Strimzi

Under replicated partitions issues for 1 partition even though we have a healthy kafka cluster #7297

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Replies: 1 comment

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

Strimzi

Under replicated partitions issues for 1 partition even though we have a healthy kafka cluster #7297

Uh oh!

Uh oh!

dbaarorapr Sep 9, 2022

Replies: 1 comment

Uh oh!

dbaarorapr Sep 9, 2022 Author

dbaarorapr
Sep 9, 2022

dbaarorapr
Sep 9, 2022
Author