[Bug]: Strimzi deleted Kafka CRD resource

[Staniel]

Bug Description

We run 0.27.1 version strimzi on our Kafka cluster and we recently run into an issue with strimzi deleted the cluster unexpectedly. We hit this issue in 3 clusters already and I wonder what could be the trigger.
Before the deletion happens, we saw some test CRD created in this cluster, it's unclear what trigger these tests.
This is the log we saw:

2023-03-15 22:12:15 INFO  ClusterOperator:121 - Triggering periodic reconciliation for namespace *
2023-03-15 22:12:15 INFO  AbstractOperator:226 - Reconciliation #48042(timer) Kafka(my-kafka-cluster-name/my-kafka-cluster-name): Kafka my-kafka-cluster-name will be checked for creation or modification
2023-03-15 22:12:17 INFO  AbstractOperator:517 - Reconciliation #48042(timer) Kafka(my-kafka-cluster-name/my-kafka-cluster-name): reconciled
2023-03-15 22:14:15 INFO  ClusterOperator:121 - Triggering periodic reconciliation for namespace *
2023-03-15 22:14:15 INFO  AbstractOperator:226 - Reconciliation #48043(timer) Kafka(my-kafka-cluster-name/my-kafka-cluster-name): Kafka my-kafka-cluster-name will be checked for creation or modification
2023-03-15 22:14:19 INFO  AbstractOperator:517 - Reconciliation #48043(timer) Kafka(my-kafka-cluster-name/my-kafka-cluster-name): reconciled
2023-03-15 22:15:37 INFO  OperatorWatcher:38 - Reconciliation #48044(watch) KafkaMirrorMaker2(kafkamirrormaker2-crd-it/test-kafka-mirror-maker-2): KafkaMirrorMaker2 test-kafka-mirror-maker-2 in namespace kafkamirrormaker2-crd-it was ADDED
2023-03-15 22:15:37 INFO  AbstractOperator:226 - Reconciliation #48044(watch) KafkaMirrorMaker2(kafkamirrormaker2-crd-it/test-kafka-mirror-maker-2): KafkaMirrorMaker2 test-kafka-mirror-maker-2 will be checked for creation or modification
2023-03-15 22:15:38 INFO  OperatorWatcher:38 - Reconciliation #48045(watch) KafkaMirrorMaker2(kafkamirrormaker2-crd-it/test-kafka-mirror-maker-2): KafkaMirrorMaker2 test-kafka-mirror-maker-2 in namespace kafkamirrormaker2-crd-it was DELETED
2023-03-15 22:15:41 INFO  OperatorWatcher:38 - Reconciliation #48046(watch) KafkaMirrorMaker2(kafkamirrormaker2-crd-it/test-kafka-mirror-maker-2): KafkaMirrorMaker2 test-kafka-mirror-maker-2 in namespace kafkamirrormaker2-crd-it was ADDED
2023-03-15 22:15:42 INFO  OperatorWatcher:38 - Reconciliation #48047(watch) KafkaMirrorMaker2(kafkamirrormaker2-crd-it/test-kafka-mirror-maker-2): KafkaMirrorMaker2 test-kafka-mirror-maker-2 in namespace kafkamirrormaker2-crd-it was DELETED
2023-03-15 22:15:45 INFO  OperatorWatcher:38 - Reconciliation #48048(watch) KafkaMirrorMaker2(kafkamirrormaker2-crd-it/test-kafka-mirror-maker-2): KafkaMirrorMaker2 test-kafka-mirror-maker-2 in namespace kafkamirrormaker2-crd-it was ADDED
2023-03-15 22:15:46 INFO  OperatorWatcher:38 - Reconciliation #48049(watch) KafkaMirrorMaker2(kafkamirrormaker2-crd-it/test-kafka-mirror-maker-2): KafkaMirrorMaker2 test-kafka-mirror-maker-2 in namespace kafkamirrormaker2-crd-it was DELETED
2023-03-15 22:15:49 INFO  OperatorWatcher:38 - Reconciliation #48050(watch) KafkaMirrorMaker2(kafkamirrormaker2-crd-it/test-kafka-mirror-maker-2): KafkaMirrorMaker2 test-kafka-mirror-maker-2 in namespace kafkamirrormaker2-crd-it was ADDED
2023-03-15 22:15:51 INFO  OperatorWatcher:38 - Reconciliation #48051(watch) KafkaMirrorMaker2(kafkamirrormaker2-crd-it/test-kafka-mirror-maker-2): KafkaMirrorMaker2 test-kafka-mirror-maker-2 in namespace kafkamirrormaker2-crd-it was MODIFIED
2023-03-15 22:15:53 INFO  OperatorWatcher:38 - Reconciliation #48052(watch) KafkaMirrorMaker2(kafkamirrormaker2-crd-it/test-kafka-mirror-maker-2): KafkaMirrorMaker2 test-kafka-mirror-maker-2 in namespace kafkamirrormaker2-crd-it was DELETED
2023-03-15 22:15:58 INFO  OperatorWatcher:38 - Reconciliation #48053(watch) KafkaMirrorMaker2(kafkamirrormaker2-crd-it/test-kafka-mirror-maker-2): KafkaMirrorMaker2 test-kafka-mirror-maker-2 in namespace kafkamirrormaker2-crd-it was ADDED
2023-03-15 22:16:00 INFO  OperatorWatcher:38 - Reconciliation #48054(watch) KafkaMirrorMaker2(kafkamirrormaker2-crd-it/test-kafka-mirror-maker-2): KafkaMirrorMaker2 test-kafka-mirror-maker-2 in namespace kafkamirrormaker2-crd-it was DELETED
2023-03-15 22:16:05 INFO  OperatorWatcher:38 - Reconciliation #48055(watch) KafkaMirrorMaker2(kafkamirrormaker2-crd-it/test-kafka-mirror-maker-2): KafkaMirrorMaker2 test-kafka-mirror-maker-2 in namespace kafkamirrormaker2-crd-it was ADDED
2023-03-15 22:16:06 INFO  OperatorWatcher:38 - Reconciliation #48056(watch) KafkaMirrorMaker2(kafkamirrormaker2-crd-it/test-kafka-mirror-maker-2): KafkaMirrorMaker2 test-kafka-mirror-maker-2 in namespace kafkamirrormaker2-crd-it was DELETED
2023-03-15 22:16:12 INFO  OperatorWatcher:38 - Reconciliation #48057(watch) KafkaMirrorMaker2(kafkamirrormaker2-crd-it/test-kafka-mirror-maker-2): KafkaMirrorMaker2 test-kafka-mirror-maker-2 in namespace kafkamirrormaker2-crd-it was ADDED
2023-03-15 22:16:13 INFO  OperatorWatcher:38 - Reconciliation #48058(watch) KafkaMirrorMaker2(kafkamirrormaker2-crd-it/test-kafka-mirror-maker-2): KafkaMirrorMaker2 test-kafka-mirror-maker-2 in namespace kafkamirrormaker2-crd-it was DELETED
2023-03-15 22:16:15 INFO  ClusterOperator:121 - Triggering periodic reconciliation for namespace *
2023-03-15 22:16:15 INFO  AbstractOperator:226 - Reconciliation #48059(timer) Kafka(my-kafka-cluster-name/my-kafka-cluster-name): Kafka my-kafka-cluster-name will be checked for creation or modification
2023-03-15 22:16:17 INFO  AbstractOperator:517 - Reconciliation #48059(timer) Kafka(my-kafka-cluster-name/my-kafka-cluster-name): reconciled
2023-03-15 22:16:20 WARN  WatcherWebSocketListener:73 - Exec Failure: HTTP 404, Status: 404 - 404 page not found

2023-03-15 22:16:22 WARN  WatcherWebSocketListener:73 - Exec Failure: HTTP 404, Status: 404 - 404 page not found

2023-03-15 22:16:26 WARN  WatcherWebSocketListener:73 - Exec Failure: HTTP 404, Status: 404 - 404 page not found

2023-03-15 22:16:34 WARN  WatcherWebSocketListener:73 - Exec Failure: HTTP 404, Status: 404 - 404 page not found

2023-03-15 22:16:37 INFO  AbstractOperator:373 - Reconciliation #48044(watch) KafkaMirrorMaker2(kafkamirrormaker2-crd-it/test-kafka-mirror-maker-2): Reconciliation is in progress
2023-03-15 22:16:38 WARN  StrimziPodSetController:210 - Invalid event received: StrimziPodSet was without the required strimzi.io/cluster label
2023-03-15 22:16:43 WARN  WatcherWebSocketListener:73 - Exec Failure: HTTP 404, Status: 404 - 404 page not found

2023-03-15 22:16:45 WARN  WatcherWebSocketListener:73 - Exec Failure: HTTP 404, Status: 404 - 404 page not found

2023-03-15 22:16:49 WARN  WatcherWebSocketListener:73 - Exec Failure: HTTP 404, Status: 404 - 404 page not found

2023-03-15 22:16:50 WARN  WatcherWebSocketListener:73 - Exec Failure: HTTP 404, Status: 404 - 404 page not found

2023-03-15 22:16:57 WARN  WatcherWebSocketListener:73 - Exec Failure: HTTP 404, Status: 404 - 404 page not found

2023-03-15 22:17:13 WARN  WatcherWebSocketListener:73 - Exec Failure: HTTP 404, Status: 404 - 404 page not found

2023-03-15 22:17:22 WARN  WatcherWebSocketListener:73 - Exec Failure: HTTP 404, Status: 404 - 404 page not found

2023-03-15 22:17:37 INFO  AbstractOperator:373 - Reconciliation #48044(watch) KafkaMirrorMaker2(kafkamirrormaker2-crd-it/test-kafka-mirror-maker-2): Reconciliation is in progress
2023-03-15 22:17:37 INFO  AbstractConnectOperator:199 - Reconciliation #48060(connector-watch) KafkaConnect(kafkaconnector-crd-it/my-connect-cluster): KafkaConnector test-source in namespace kafkaconnector-crd-it was ADDED, but Connect cluster my-connect-cluster does not exist
2023-03-15 22:17:37 WARN  VersionUsageUtils:60 - The client is using resource type 'kafkaconnectors' with unstable version 'v1beta2'
2023-03-15 22:17:37 INFO  CrdOperator:113 - Reconciliation #48060(connector-watch) KafkaConnect(kafkaconnector-crd-it/my-connect-cluster): Status of KafkaConnector test-source in namespace kafkaconnector-crd-it has been updated
2023-03-15 22:17:37 INFO  AbstractConnectOperator:199 - Reconciliation #48061(connector-watch) KafkaConnect(kafkaconnector-crd-it/my-connect-cluster): KafkaConnector test-source in namespace kafkaconnector-crd-it was MODIFIED, but Connect cluster my-connect-cluster does not exist
2023-03-15 22:17:39 INFO  AbstractConnectOperator:199 - Reconciliation #48062(connector-watch) KafkaConnect(kafkaconnector-crd-it/my-connect-cluster): KafkaConnector test-source in namespace kafkaconnector-crd-it was DELETED, but Connect cluster my-connect-cluster does not exist
2023-03-15 22:17:41 INFO  AbstractConnectOperator:199 - Reconciliation #48063(connector-watch) KafkaConnect(kafkaconnector-crd-it/my-connect-cluster): KafkaConnector test-source in namespace kafkaconnector-crd-it was ADDED, but Connect cluster my-connect-cluster does not exist
2023-03-15 22:17:41 INFO  CrdOperator:113 - Reconciliation #48063(connector-watch) KafkaConnect(kafkaconnector-crd-it/my-connect-cluster): Status of KafkaConnector test-source in namespace kafkaconnector-crd-it has been updated
2023-03-15 22:17:41 INFO  AbstractConnectOperator:199 - Reconciliation #48064(connector-watch) KafkaConnect(kafkaconnector-crd-it/my-connect-cluster): KafkaConnector test-source in namespace kafkaconnector-crd-it was MODIFIED, but Connect cluster my-connect-cluster does not exist
2023-03-15 22:17:43 INFO  AbstractConnectOperator:199 - Reconciliation #48065(connector-watch) KafkaConnect(kafkaconnector-crd-it/my-connect-cluster): KafkaConnector test-source in namespace kafkaconnector-crd-it was MODIFIED, but Connect cluster my-connect-cluster does not exist
2023-03-15 22:17:43 INFO  CrdOperator:113 - Reconciliation #48065(connector-watch) KafkaConnect(kafkaconnector-crd-it/my-connect-cluster): Status of KafkaConnector test-source in namespace kafkaconnector-crd-it has been updated
2023-03-15 22:17:43 INFO  AbstractConnectOperator:199 - Reconciliation #48066(connector-watch) KafkaConnect(kafkaconnector-crd-it/my-connect-cluster): KafkaConnector test-source in namespace kafkaconnector-crd-it was MODIFIED, but Connect cluster my-connect-cluster does not exist
2023-03-15 22:17:44 INFO  AbstractConnectOperator:199 - Reconciliation #48067(connector-watch) KafkaConnect(kafkaconnector-crd-it/my-connect-cluster): KafkaConnector test-source in namespace kafkaconnector-crd-it was DELETED, but Connect cluster my-connect-cluster does not exist
2023-03-15 22:17:45 WARN  WatcherWebSocketListener:73 - Exec Failure: HTTP 404, Status: 404 - 404 page not found

2023-03-15 22:17:48 WARN  WatcherWebSocketListener:73 - Exec Failure: HTTP 404, Status: 404 - 404 page not found

2023-03-15 22:17:50 WARN  WatcherWebSocketListener:73 - Exec Failure: HTTP 404, Status: 404 - 404 page not found

2023-03-15 22:17:54 WARN  WatcherWebSocketListener:73 - Exec Failure: HTTP 404, Status: 404 - 404 page not found

2023-03-15 22:17:54 WARN  WatcherWebSocketListener:73 - Exec Failure: HTTP 404, Status: 404 - 404 page not found

2023-03-15 22:18:02 WARN  WatcherWebSocketListener:73 - Exec Failure: HTTP 404, Status: 404 - 404 page not found

2023-03-15 22:18:12 WARN  WatcherWebSocketListener:73 - Exec Failure: HTTP 404, Status: 404 - 404 page not found

2023-03-15 22:18:14 WARN  WatcherWebSocketListener:73 - Exec Failure: HTTP 404, Status: 404 - 404 page not found

2023-03-15 22:18:15 INFO  ClusterOperator:121 - Triggering periodic reconciliation for namespace *
2023-03-15 22:18:15 INFO  AbstractOperator:226 - Reconciliation #48068(timer) Kafka(my-kafka-cluster-name/my-kafka-cluster-name): Kafka my-kafka-cluster-name will be checked for creation or modification
2023-03-15 22:18:17 INFO  AbstractOperator:517 - Reconciliation #48068(timer) Kafka(my-kafka-cluster-name/my-kafka-cluster-name): reconciled
2023-03-15 22:18:17 WARN  WatcherWebSocketListener:73 - Exec Failure: HTTP 404, Status: 404 - 404 page not found

2023-03-15 22:18:18 WARN  WatcherWebSocketListener:73 - Exec Failure: HTTP 404, Status: 404 - 404 page not found

2023-03-15 22:18:18 INFO  OperatorWatcher:38 - Reconciliation #48069(watch) Kafka(my-kafka-cluster-name/my-kafka-cluster-name): Kafka my-kafka-cluster-name in namespace my-kafka-cluster-name was DELETED
2023-03-15 22:18:18 INFO  AbstractOperator:261 - Reconciliation #48069(watch) Kafka(my-kafka-cluster-name/my-kafka-cluster-name): Kafka my-kafka-cluster-name should be deleted
2023-03-15 22:18:19 INFO  AbstractOperator:266 - Reconciliation #48069(watch) Kafka(my-kafka-cluster-name/my-kafka-cluster-name): Assembly my-kafka-cluster-name or some parts of it will be deleted by garbage collection
2023-03-15 22:18:19 WARN  WatcherWebSocketListener:73 - Exec Failure: HTTP 404, Status: 404 - 404 page not found

Steps to reproduce

We don't have a way to reproduce it but it does happen 3 times.

Expected behavior

The Kafka CRD should not be deleted.

Strimzi version

0.27.1

Kubernetes version

1.22

Installation method

Helm chart

Infrastructure

Amazon EKS

Configuration files and logs

We can not paste full config here due to legal concern, but let us know what specific sections of config can help here. We are running Kafka version 2.8.1.

Additional context

My guess is something trigger the Strimzi to run its test suites, and during the test, it create/delete some kafka-mirrormaker-2 and kafkaconnect resources, after the test finished, it accidentally delete the Kafka resource as well, probably because the tests are failing.

It's unclear to me 1) what could trigger the test running 2) why the clusters are deleted after the tests output.

[scholzj]

Strimzi does not delete CRDs. It does not have any functionality for it - it even does not have the RBAC rights to do something like that. The logs indeed suggest that the CRDs were deleted - but it would have to be caused by something else. Maybe you can try to set finalizers on the CRDs or something like that to prevent it and see where the errors show up?

[Staniel]

I see. But Strimzi does create/delete CRs under kafka-mirrormaker-2 and kafka-connect. Do you have any idea under what conditions will strimzi trigger all these test resources creation?

[Staniel]

Could strimzi-drain-cleaner be related?

[scholzj]

The only custom resources which Strimzi creates, modifies or deletes are the KafkaTopic resources in the Topic Operator. But even there it would be only CRs and not CRDs. Everything else is created or deleted by the user. Strimzi only updates their status. Drain Cleaner actually does not touch the CRs or CRDs at all. It only operates with Pods.

The Strimzi tests do work with all CRs and CRDs as part of the tests. But the tests are not part of the Strimzi container images which we ship - and even if it would be, it does not have the RBAC to manage CRDs. If someone decides to run the Strimzi test suite (for example as part of building Strimzi locally) and runs it against the same Kubernetes cluster, then it might indeed delete and install CRDs. But that is not really Strimzi's fault but of the person running them.

[Staniel]

Thanks for the hint! We figure out some other tasks are running tests and accidentally connect to the EKS cluster, causing the problem.