Strimzi Entity Operator (Topic Operator) Failing Probes Due to Kafka Connection Issues with KRaft Cluster

[periyasamy003]

kafka-crds.zip
[amqstreams.v2.9.0-3] - Operator in Openshift
netpolicy-kafka.zip

Problem Description:
The Strimzi Entity Operator's Topic Operator container was consistently failing its startup and readiness probes, exhibiting two distinct symptoms:

1. Startup probe failed: ... connect: connection refused on port 8080 (indicating the application either crashed immediately or wasn't listening).
2. Readiness probe failed: HTTP probe failed with statuscode: 500 (indicating the application was listening, but its internal readiness check was failing, almost always due to an inability to connect to Kafka).

Troubleshooting Steps & Findings:

1. Network Policy Review:

   • Initial suspicion was network policies blocking communication.
   • Provided NetworkPolicy files were reviewed: kafka-bt-allow-kafka-egress, kafka-bt-allow-kafka-ingress, kafka-bt-allow-entity-operator-egress, kafka-bt-allow-entity-operator-ingress-self.
   • Suggestions were made to enhance these policies to explicitly allow:
     • Kafka broker egress for standard DNS (port 53/UDP/TCP).
     • Kafka broker ingress from OpenShift Ingress Controller (for external Route traffic on 9094).
     • Entity Operator egress for Kubernetes API server (port 6443/TCP).
   • Crucially, a provided generic kafka-bt-allow-dns-netpolicy was found to be incorrectly configured for port 5353/UDP instead of standard DNS port 53/UDP/TCP. This was corrected as it would affect all pods.

2. Environment Variable Check (Root Cause Identification):

   • Despite network policy adjustments, the issue persisted. Attention shifted to the Entity Operator's Kafka bootstrap configuration.
   • The STRIMZI_KAFKA_BOOTSTRAP_SERVERS environment variable inside the running kafka-bt-cluster-entity-operator pod was checked via oc exec command.
   • Key Finding: The variable was persistently set to kafka-bt-cluster-kafka-bootstrap:9091 instead of the correct kafka-bt-cluster-kafka-bootstrap:9093 (the internal listener for KRaft).
   • Examination of the Kafka Custom Resource (oc get kafka kafka-bt-cluster -n kafka-bt -o yaml) revealed the underlying problem: The spec.entityOperator.topicOperator and spec.entityOperator.userOperator sections were missing the template.container.env blocks required to inject this override. Without this, the Entity Operator was defaulting to an incorrect listener port for connection.

Solution Applied/Proposed:

1. Correct Network Policies: Ensure all necessary NetworkPolicy resources are correctly applied, specifically:

   • The kafka-bt-deny-all-netpolicy (if applicable).
   • The corrected kafka-bt-allow-dns-netpolicy (allowing port 53/UDP/TCP).
   • The updated Entity Operator egress policy (kafka-bt-allow-entity-operator-egress) including Kubernetes API server access.
   • Other specific Kafka ingress/egress policies as discussed.

2. Critical Kafka CR Fix:

   • The Kafka Custom Resource (kafka-bt-cluster) was updated to include the template.container.env override for both topicOperator and userOperator within the entityOperator section. This explicitly sets STRIMZI_KAFKA_BOOTSTRAP_SERVERS to kafka-bt-cluster-kafka-bootstrap:9093.
   • This can be done either by oc applying a corrected YAML file or by oc editing the resource directly.

Expected Outcome:
With the STRIMZI_KAFKA_BOOTSTRAP_SERVERS environment variable correctly pointing to 9093 and necessary network policies in place, the Entity Operator should now be able to establish a successful connection to the Kafka cluster, resolve the 500 Internal Server Error on its readiness probes, and start functioning correctly.

[im-konge]

Hey, you mentioned AMQ Streams, which is a product from RedHat. You should message their support if you have issue with the product.

[scholzj]

As Lukas suggested, please contact Red Hat support next time. But in short, kafka-bt-cluster-kafka-bootstrap:9091 is the correct address, so that finding is not valid.