Failed to reconcile kafka connect strimzi, wont move to ready status

[alejandrade]

I've installed strimzi 0.22.1 we are only using the kafka connect functionality.

The following exception keeps spamming my logs.

We are using aws EKS 1.19. No other special configurations.

io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: PUT at: https://172.20.0.1/apis/kafka.strimzi.io/v1alpha1/namespaces/kafka-connect/kafkaconnectors/kafka-es-bl-entry-connector/status. Message: kafkaconnectors.kafka.strimzi.io "kafka-es-bl-entry-connector" not found. Received status: Status(apiVersion=v1, code=404, details=StatusDetails(causes=[], group=kafka.strimzi.io, kind=kafkaconnectors, name=kafka-es-bl-entry-connector, retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status, message=kafkaconnectors.kafka.strimzi.io "kafka-es-bl-entry-connector" not found, metadata=ListMeta(_continue=null, remainingItemCount=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=NotFound, status=Failure, additionalProperties={}). at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:570) ~[io.fabric8.kubernetes-client-5.0.2.jar:?] at io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:509) ~[io.fabric8.kubernetes-client-5.0.2.jar:?] at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:474) ~[io.fabric8.kubernetes-client-5.0.2.jar:?] at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:435) ~[io.fabric8.kubernetes-client-5.0.2.jar:?] at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:418) ~[io.fabric8.kubernetes-client-5.0.2.jar:?] at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleStatusUpdate(OperationSupport.java:292) ~[io.fabric8.kubernetes-client-5.0.2.jar:?] at io.fabric8.kubernetes.client.dsl.base.BaseOperation.updateStatus(BaseOperation.java:746) ~[io.fabric8.kubernetes-client-5.0.2.jar:?] at io.fabric8.kubernetes.client.dsl.base.BaseOperation.updateStatus(BaseOperation.java:85) ~[io.fabric8.kubernetes-client-5.0.2.jar:?] at io.strimzi.operator.common.operator.resource.CrdOperator.lambda$updateStatusAsync$2(CrdOperator.java:111) ~[io.strimzi.operator-common-0.22.1.jar:0.22.1] at io.vertx.core.impl.ContextImpl.lambda$executeBlocking$2(ContextImpl.java:313) ~[io.vertx.vertx-core-3.9.1.jar:3.9.1] at io.vertx.core.impl.TaskQueue.run(TaskQueue.java:76) ~[io.vertx.vertx-core-3.9.1.jar:3.9.1] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?] at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty.netty-common-4.1.60.Final.jar:4.1.60.Final] at java.lang.Thread.run(Thread.java:834) [?:?]

[alejandrade]

It seems status it never crosses to ready

[alejandrade]

the manifests where the same. I also tried helm chart of latest version. Same issue, this is truly strange. I posted more findings bellow.

[scholzj]

WDYM with I also tried helm chart of latest version.? The latest version would be Strimzi 0.24. That doesn't use the v1beta1 API at all. So did you in that case got the errors with different paths?

[alejandrade]

yes, it did the other path. I took the opportunity to update strimzi. I actually have multiple kubes environments with the same issue.

io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: PUT at: https://172.20.0.1/apis/kafka.strimzi.io/v1beta2/namespaces/kafka-connect/kafkaconnectors/m2perf-kafka-es-bl-entry-connector/status. Message: kafkaconnectors.kafka.strimzi.io "m2perf-kafka-es-bl-entry-connector" not found. Received status: Status(apiVersion=v1, code=404, details=StatusDetails(causes=[], group=kafka.strimzi.io, kind=kafkaconnectors, name=m2perf-kafka-es-bl-entry-connector, retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status, message=kafkaconnectors.kafka.strimzi.io "m2perf-kafka-es-bl-entry-connector" not found, metadata=ListMeta(_continue=null, remainingItemCount=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=NotFound, status=Failure, additionalProperties={}).
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:639) ~[io.fabric8.kubernetes-client-5.4.1.jar:?]
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:578) ~[io.fabric8.kubernetes-client-5.4.1.jar:?]
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:543) ~[io.fabric8.kubernetes-client-5.4.1.jar:?]
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:504) ~[io.fabric8.kubernetes-client-5.4.1.jar:?]
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleUpdate(OperationSupport.java:330) ~[io.fabric8.kubernetes-client-5.4.1.jar:?]
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleUpdate(OperationSupport.java:310) ~[io.fabric8.kubernetes-client-5.4.1.jar:?]
at io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleUpdate(BaseOperation.java:898) ~[io.fabric8.kubernetes-client-5.4.1.jar:?]
at io.fabric8.kubernetes.client.dsl.base.BaseOperation.updateStatus(BaseOperation.java:753) ~[io.fabric8.kubernetes-client-5.4.1.jar:?]
at io.fabric8.kubernetes.client.dsl.base.BaseOperation.updateStatus(BaseOperation.java:86) ~[io.fabric8.kubernetes-client-5.4.1.jar:?]
at io.strimzi.operator.common.operator.resource.CrdOperator.lambda$updateStatusAsync$2(CrdOperator.java:112) ~[io.strimzi.operator-common-0.24.0.jar:0.24.0]
at io.vertx.core.impl.ContextImpl.lambda$null$0(ContextImpl.java:160) ~[io.vertx.vertx-core-4.1.0.jar:4.1.0]
at io.vertx.core.impl.AbstractContext.dispatch(AbstractContext.java:96) ~[io.vertx.vertx-core-4.1.0.jar:4.1.0]
at io.vertx.core.impl.ContextImpl.lambda$executeBlocking$1(ContextImpl.java:158) ~[io.vertx.vertx-core-4.1.0.jar:4.1.0]
at io.vertx.core.impl.TaskQueue.run(TaskQueue.java:76) ~[io.vertx.vertx-core-4.1.0.jar:4.1.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty.netty-common-4.1.65.Final.jar:4.1.65.Final]
at java.lang.Thread.run(Thread.java:829) [?:?]

[alejandrade]

I noticed there is a status property in the yaml. Am I supposed to configure this url somehow?

[scholzj]

No, the URI seems to work everywhere else. I have no idea what could be the issue :-(. I'm even not aware how to disable the status in the API server or something like that. Maybe some admission webhook can be capturing this and screwing this up?

[alejandrade]

the following curl returns a manifest
curl --location --request GET 'https://172.20.0.1/apis/kafka.strimzi.io/v1beta1/namespaces/kafka-connect/kafkaconnects/kafka-connect-debezium'

but the following returns 404
curl --location --request GET 'https://172.20.0.1/apis/kafka.strimzi.io/v1beta1/namespaces/kafka-connect/kafkaconnects/kafka-connect-debezium/status'

{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {
    
  },
  "status": "Failure",
  "message": "kafkaconnects.kafka.strimzi.io \"kafka-connect-debezium\" not found",
  "reason": "NotFound",
  "details": {
    "name": "kafka-connect-debezium",
    "group": "kafka.strimzi.io",
    "kind": "kafkaconnects"
  },