Kafka Connect custom image build questions

[davehouser1]

I am trying to deploy kafka connect with a custom plugin.
I read though this link here , and section 4.2.3 here

I also found documentation on the parameters for the build process here

Few things:

• It is not clear to me if a repo needs to be created a head of time on our private registry before trying to create the build. The name "output" implies that after Kafka Connect custom build is created it will be placed into the repo defined in "image". Must the repo specified in spec.build.output.image be created ahead of time? or will Kafka Connect build create the image and deposit it in the specified location?
• I am trying to use a private repo with this syntax https://10.10.10.10:443/v2/my-connect-cluster:latest, I think this is supported but the examples in the last document I shared do not include 'https://' will this syntax work?
• The last document states that pushSecret is required. Our private docker repo has no password. Is there some work around where I can make this manifest run without a password?

[scholzj]

It is not clear to me if a repo needs to be created a head of time on our private registry before trying to create the build. The name "output" implies that after Kafka Connect custom build is created it will be placed into the repo defined in "image". Must the repo specified in spec.build.output.image be created ahead of time? or will Kafka Connect build create the image and deposit it in the specified location?

I do not think there is any standard for creating container repositories. So it all depends on your container registry. Some would auto-create the repository when you try to push to it. Some would not. So it depends. In general, creating it ahead should not cause any harm.

I am trying to use a private repo with this syntax https://10.10.10.10:443/v2/my-connect-cluster:latest, I think this is supported but the examples in the last document I shared do not include 'https://' will this syntax work?

I do not think the https:// should be included. I never used it when referencing Docker images. But not sure what happens if you use it, I do not think I ever tested it.

The last document states that pushSecret is required. Our private docker repo has no password. Is there some work around where I can make this manifest run without a password?

That would be a bug if it says it is required. The push secret is needed only if your repository requires authentication. If it doesn't as you mentioned it should not be needed.

[scholzj]

If you enabled the Connector Operator, you can do kubectl get kafkaconnect -o yaml and check the status section of the YAML. But it looks like you do not use the connector operator ... in which case I think we do not fill this field in. So you can do something like curl localhost:8083/connector-plugins from the Connect pod for example.

[davehouser1]

Actually kubectl get kafkaconnect -o yaml worked and gave the output I needed to check class names.
When deploying my KafkaConnector for setting Splunk Connect parameters I see lots of java errors dealing with serialization:

2021-06-17 16:05:26,063 ERROR WorkerSinkTask{id=my-sink-connector-2} Error converting message value in topic 'zeek' partition 2 at offset 0 and timestamp 1623093595907: Converting byte[] to Kafka Connect data failed due to serialization error:  (org.apache.kafka.connect.runtime.WorkerSinkTask) [task-thread-my-sink-connector-2]
org.apache.kafka.connect.errors.DataException: Converting byte[] to Kafka Connect data failed due to serialization error:

The connector seems to deploy but kafka connect cannot seem to parse my data.
Here is what the data looks like

{"ts":1623945189.032393,"uid":"CunKml3CUMWzScKpo4","id.orig_h":"192.168.104.100","id.orig_p":50460,"id.resp_h":"192.168.105.100","id.resp_p":53,"proto":"udp","trans_id":29636,"query":"nilhost","qclass":1,"qclass_name":"C_INTERNET","qtype":1,"qtype_name":"A","AA":false,"TC":false,"RD":true,"RA":false,"Z":0,"rejected":false}

KafkaConnector file:

apiVersion: kafka.strimzi.io/v1beta2 
kind: KafkaConnector 
metadata: 
  name: my-sink-connector 
  labels: 
    strimzi.io/cluster: my-connect-cluster 
spec: 
  class: com.splunk.kafka.connect.SplunkSinkConnector 
  tasksMax: 3 
  config: 
    topics: zeek 
    splunk.indexes: splunk-kafka 
    splunk.hec.uri: http://inputs.10.111.11.11:8088/services/collector 
    splunk.hec.token: 9f1acd9a-34fb-4d31-af97-542e80d6b404 
    splunk.hec.raw: true 
    splunk.hec.ack.enabled: false 
    splunk.hec.track.data: true 

• Followed this guide on "Splunk Connect for Kafka tasks fail due to serialization error", problem persists. I am using the JSON parser.
• Made sure topic exists, also there is data populated in the topic.
• Tried using the StringParser, which caused more errors. I think I need to use JSON.

I understand if this is out of scope for Strimzi, just curious if anyone ran into this issue before?

[scholzj]

I'm afraid I do not know much about this and nothing about Splunk. I know how to configure the convertors but that is it. So I'm probably not going to be able to help.

I'm normally using the StringConverter:

  config:
    key.converter: org.apache.kafka.connect.storage.StringConverter
    value.converter: org.apache.kafka.connect.storage.StringConverter
    key.converter.schemas.enable: false
    value.converter.schemas.enable: false

But I think there is a ByteArrayConverter or something like that as well. Also, you should try to switch of the schema if your JSOn doesn't have it.

[davehouser1]

Got it working. Now data is coming to Splunk. Perfect!

• Problem 1: The topic had non json data in it (Just strings), as well as JSON data that I wanted. Therefore no single parser would work. Deleted the topic and created a fresh one. Only put JSON data in it. This worked.
• Problem 2: The SPLUNK URI was invalid, needed to be "https://10.111.11.11:8088"
• Problem 3: Don't validate certs with "splunk.hec.ssl.validate.certs: false"

Last question I have for now:
It looks like I need to use KakaConnector manifests to make changes to the API. Is it possible to expose the service/my-connect-cluster-connect-api with the load balancer, and be able to use REST API with the KafkaConnector manifests? I believe I read somewhere that when KafkaConnect is enabled, any REST API changes will be reverted. No way to be able to use KafakConnector API changes with REST API?

[scholzj]

The way the operator works is that it sees the KafkaConnector resources as the single source of truth. So you cannot go and change something over the REST API because the operator will revert it. You can use it only to read information from it.

If you wanna do that, you will need to:

• Create your own network policy to enable the access tot he port 8083 in the Connect pods
• If needed, create your own load balancer, Ingress, or whatever else you need

Or you can disable the connector operator and do everything over the REST API without using the KafkaConnector resources.