Strimzi
KafkaConnector error: ConnectRestException #10875
Unanswered
benedictmulongo
asked this question in
Q&A
Replies: 4 comments 8 replies
-
|
I guess the start will be to share the full logs from the operator and from Connect. |
Beta Was this translation helpful? Give feedback.
5 replies
-
|
The only commands I have used to get some sort of response is: kubectl -n kafka describe KafkaConnector kafka-connect-s3-connectorwhich returns Spec:
Class: io.confluent.connect.s3.S3SinkConnector
Config:
behavior.on.null.values: ignore
flush.size: 1000
format.class: io.confluent.connect.s3.format.parquet.ParquetFormat
partition.duration.ms: 3600000
partitioner.class: io.confluent.connect.storage.partitioner.TimeBasedPartitioner
path.format: 'year'=YYYY/'month'=MM/'day'=dd/'hour'=HH
s3.bucket.name: delta-lake-test
s3.part.size: 1073741824
s3.region: us-east-1
storage.class: io.confluent.connect.s3.storage.S3Storage
store.url: https://s3.storage.url.net
Topics: test-topic
Tasks Max: 2
Status:
Conditions:
Last Transition Time: 2024-11-25T09:08:27.686902118Z
Message: PUT /connectors/kafka-connect-s3-connector/config returned 500 (Server Error): Unknown error message
Reason: ConnectRestException
Status: True
Type: NotReady
Observed Generation: 1
Tasks Max: 2
Events: <none>kubectl -n kafka get KafkaConnectorreturns error, because the connector is not even scheduled as a pod that is failing. kubectl -n kafka logs kafka-connect-s3-connector
|
Beta Was this translation helpful? Give feedback.
-
|
The connector will never have an Pod. That is a misunderstanding how Kafka Connect work. It will run inside the Connect Pod(s). So as I said, you have to share the operator and Connect logs. |
Beta Was this translation helpful? Give feedback.
-
|
It seems to be errors with plugins missing or similiar: Caused by: java.lang.ExceptionInInitializerError: Exception java.lang.NoSuchMethodError: 'java.lang.String org.apache.kafka.common.utils.Utils.join(java.util.Collection, java.lang.String)' [in thread "pool-2-thread-1"]
...
javax.servlet.ServletException: org.glassfish.jersey.server.ContainerException: java.lang.NoClassDefFoundError: Could not initialize class io.confluent.connect.s3.S3SinkConnectorConfig$CannedAclValidator
...But I have explicity added those in my kafkaconnect: ...
build:
output:
type: docker
image: platform.sunet.se/benedith/das-kafka-connect-cluster:latest
pushSecret: docker-platform-creds
plugins:
- name: kafka-connect-s3
artifacts:
- type: zip
url: .../jars/confluentinc-kafka-connect-s3-10.5.17.zip
sha512sum: 743b3..
- name: avro-connector
artifacts:
- type: zip
url: .../jars/confluentinc-kafka-connect-avro-converter-7.7.1.zip
sha512sum: aab7ef2d8732af3f4
template:
pod:
imagePullSecrets:
- name: docker-platform-creds
... |
Beta Was this translation helpful? Give feedback.
-
|
Well, it is really hard to help when you are ignoring what you are told and not sharing the logs. But in general, you should be mindful of the various classpaths. You should likely:
|
Beta Was this translation helpful? Give feedback.
-
|
Sorry, excuse me if you felt like I was ignoring your guides it just that I was avoiding to post long logs that may be unintelligible and afraid of posting online info about servers. That why a post a summary of the logs errors: Caused by: java.lang.ExceptionInInitializerError: Exception java.lang.NoSuchMethodError: 'java.lang.String org.apache.kafka.common.utils.Utils.join(java.util.Collection, java.lang.String)' [in thread "pool-2-thread-1"]
...
javax.servlet.ServletException: org.glassfish.jersey.server.ContainerException: java.lang.NoClassDefFoundError: Could not initialize class io.confluent.connect.s3.S3SinkConnectorConfig$CannedAclValidator
...The logs was basically returning the two errors in loops, after I guess some internal retries. But your last response seems to have solved my problems. Thanks ! If the S3 connector needs to use the Avro converter, you need to add it as two artifacts of the same plugin in order to be on the same plugin classpath and not as two separate plugins. |
Beta Was this translation helpful? Give feedback.
-
|
Sorry, I would need more help. It seems like the problem what solved by adding everything under the same artifcats as you suggested: plugins:
- name: kafka-connect-s3
artifacts:
- type: zip
url: .../jars/confluentinc-kafka-connect-s3-10.4.2.zip
sha512sum: b5fv4e...
- type: zip
url: .../jars/confluentinc-kafka-connect-avro-converter-7.3.3.zip
sha512sum: 8b9b3...it seems like I was looking at the wrong operators, kafkaconnect instead of kafkaconnectors. kafkaconnectors still seem not ready, after I tried to send a message thourgh the topic, there is a senderror. Here is a log: Thanks for all help |
Beta Was this translation helpful? Give feedback.
0 replies
-
|
I have isolated the root/main problem: Caused by: java.lang.ExceptionInInitializerError: Exception java.lang.NoSuchMethodError: 'java.lang.String org.apache.kafka.common.utils.Utils.join(java.util.Collection, java.lang.String)' [in thread "pool-2-thread-1"]
at io.confluent.connect.s3.S3SinkConnectorConfig$CannedAclValidator.<clinit>(S3SinkConnectorConfig.java:1222)The image created seems to be using kafka with version |
Beta Was this translation helpful? Give feedback.
3 replies
-
|
The second number means the Kafka version - 3.8.0. We ship the binaries from Apache Kafka |
Beta Was this translation helpful? Give feedback.
-
|
It seems indeed like there was a problem with version mismatch, by building the image myself with all the dependencies everything seems to work so far and the connector is at least registered. It is possible to automatically build & push with strimzi and kaniko like the following: apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnect
metadata:
name: kafka-connect-cluster
namespace: kafka
annotations:
strimzi.io/use-connector-resources: "true"
spec:
...
build:
output:
type: docker
image: platform.sunet.se/benedith/das-kafka-connect-cluster:latest
pushSecret: docker-platform-creds
plugins:
- name: kafka-connect-s3
artifacts:
- type: zip
url: .../jars/confluentinc-kafka-connect-s3-10.5.17.zip
sha512sum: 17b4ba9
- type: zip
url: .../jars/confluentinc-kafka-connect-avro-converter-7.7.1.zip
sha512sum: ce3f3f4
template:
pod:
imagePullSecrets:
- name: docker-platform-credsand specifiyng the underlying base image e.g. |
Beta Was this translation helpful? Give feedback.
-
|
I'm not sure what exactly you did ... but please keep in mind:
|
Beta Was this translation helpful? Give feedback.
-
|
Seems like upstream issue with newer kafka. Fix merged yesterday. Not yet released |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
I have the following config file for both kafkaconnect and kafkaconnector respectively:
The pod for kafka connect is running without any problem and the image created is successfully pushed to the repository. However kafka connector is not running and I am getting errors:
gives:
Status: Conditions: Last Transition Time: 2024-11-25T09:08:27.686902118Z Message: PUT /connectors/kafka-connect-s3-connector/config returned 500 (Server Error): Unknown error message Reason: ConnectRestException Status: True Type: NotReady Observed Generation: 1 Tasks Max: 2Yes, I have read the discussion at https://github.com/orgs/strimzi/discussions/6501 but that does not seem to solve my particular problem
Beta Was this translation helpful? Give feedback.
All reactions