Plugins for KafkaConnect

[benedictmulongo]

In the following blog article https://strimzi.io/blog/2021/03/29/connector-build/ it is shown how to create KafkaConnect pod that automatically build an image using kaniko in the background:

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnect
metadata:
  name: my-connect-cluster
  annotations:
    strimzi.io/use-connector-resources: "true"
spec:
  ...
  build: (1)
    output: (2)
      type: docker
      image: my-registry.io/my-org/my-connect-cluster:latest
      pushSecret: my-registry-credentials
    plugins: (3) 
      - name: debezium-postgres-connector
        artifacts:
          - type: zip
            url: https://repo1.maven.org/maven2/io/debezium/debezium-connector-postgres/1.4.2.Final/debezium-connector-postgres-1.4.2.Final-plugin.zip
            sha512sum: ef1620547e6ddf5be010271849b6a87a19f6e6beee93b379c80883815b8f37ec5137095b2c99975d7704cbf957e6a33d76c61109582cad57c7cbbfae43adc86c
      - name: camel-timer
        artifacts:
          - type: tgz
            url: https://repo.maven.apache.org/maven2/org/apache/camel/kafkaconnector/camel-timer-kafka-connector/0.8.0/camel-timer-kafka-connector-0.8.0-package.tar.gz
            sha512sum: c0102700ae176b1ef078bdb86d640935a1333865e58b870f6b0ef3d9dad067221122aee6bf52852dad209a7c2051f359e0434fc7fbd783fb50e21707c7577ef9
      - name: echo-connector
        artifacts:
          - type: jar
            url: https://github.com/scholzj/echo-sink/releases/download/1.1.0/echo-sink-1.1.0.jar
            sha512sum: b7da48d5ecd1e4199886d169ced1bf702ffbdfd704d69e0da97e78ff63c1bcece2f59c2c6c751f9c20be73472b8cb6a31b6fd4f75558c1cb9d96daa9e9e603d2

My use case is a bit different because I am using kafka-connect-avro-converter & kafka-connect-s3, the version from maven are not working and the ones from confluent hub https://www.confluent.io/hub/ have many dependencies. It is possible to point the yaml config to a plugins folder where all the plugins will be automatically uploaded into the created image? This is done in the following article but they are creating the image manually https://blog.min.io/stream-data-to-minio-using-kafka-kubernetes/

I want to do something like the following:

FROM confluentinc/cp-kafka-connect:7.0.9 as cp
RUN confluent-hub install --no-prompt confluentinc/kafka-connect-s3:10.4.2
RUN confluent-hub install --no-prompt confluentinc/kafka-connect-avro-converter:7.3.3
FROM quay.io/strimzi/kafka:0.34.0-kafka-3.4.0
USER root:root
# Add S3 dependency
COPY --from=cp /usr/share/confluent-hub-components/confluentinc-kafka-connect-s3/ /opt/kafka/plugins/kafka-connect-s3/

where all the plugins /usr/share/confluent-hub-components/confluentinc-kafka-connect-s3/ are directly uploaded to the image but using Yaml and kaniko as presented in the blog article. Is it possible?

[katheris]

No, as stated in the docs you have two options:

• Configure Kafka Connect to build a new container image with plugins that Strimzi downloads from a given url
• Build a Docker image yourself from the base Kafka Connect image

[benedictmulongo]

Thanks for a quick reply. I solved it by using build.plugins.artifacts.typ: zip, it seems to work. Bu I will have the manual docker build version as a backup