Trouble running clickhouse-kafka-connect as a docker image

[mmarkell]

I am trying to follow this example with docker-compose:

clickhouse-kafka-connect-sink:
        build:
            context: .
            dockerfile: /packages/kafka-clickhouse-connect/Dockerfile
        environment:
            CONNECT_REST_PORT: 8083
            CONNECT_REST_ADVERTISED_HOST_NAME: kafka-connect
            CONNECT_PLUGIN_PATH: "/usr/share/java,/usr/share/confluent-hub-components,/usr/share/dockershare"
            CONNECT_BOOTSTRAP_SERVERS: kafka:9092
            CONNECT_GROUP_ID: kafka-connect
            CONNECT_KEY_CONVERTER: org.apache.kafka.connect.storage.StringConverter
            CONNECT_VALUE_CONVERTER: org.apache.kafka.connect.json.JsonConverter
            CONNECT_CONFIG_STORAGE_TOPIC: connect-configs
            CONNECT_OFFSET_STORAGE_TOPIC: connect-offsets
            CONNECT_STATUS_STORAGE_TOPIC: connect-status

        depends_on: 
            kafka:
                condition: service_healthy
        volumes: 
            - "./packages/kafka-clickhouse-connect/DockerShare/:/usr/share/dockershare/:ro"

With this dockerfile:

FROM confluentinc/cp-kafka-connect:7.0.9
RUN curl -k -SL "https://github.com/ClickHouse/clickhouse-kafka-connect/releases/download/v1.0.1/clickhouse-kafka-connect-v1.0.1.zip"  --output /tmp/clickhouse-kafka-connect-v1.0.1.jar
RUN confluent-hub install --no-prompt /tmp/clickhouse-kafka-connect-v1.0.1.jar

And this properties.json file in my own DockerShare folder:

{
    "name": "clickhouse-connect",
    "config": {
      "connector.class": "com.clickhouse.kafka.connect.ClickHouseSinkConnector",
      "tasks.max": "1",
      "consumer.override.max.poll.records": "5000",
      "consumer.override.max.partition.fetch.bytes": "5242880",
      "database": "default",
      "errors.retry.timeout": "60",
      "exactlyOnce": "true",
      "hostname": "localhost",
      "port": "8123",
      "ssl": "false",
      "username": "default",
      "topics": "my-topic",
      "value.converter": "org.apache.kafka.connect.json.JsonConverter",
      "value.converter.schemas.enable": "false",
      "clickhouseSettings": ""
    }
  }

But when I run this and try to find my configuration, there is nothing registered:

sh-4.4$ curl localhost:8083/connectors
[]

What am I doing wrong? Really appreciate any advice

[Paultagoras]

Hmm - usually when I fire up the docker image, I then make a POST REST call to actually create the connector (the endpoint is localhost:8083/connectors for me as well).

Have you tried using that just to see? The payload would look similar (this is the payload I usually send):

{
  "name": "clickhouse-connect",
  "config": {
    "connector.class": "com.clickhouse.kafka.connect.ClickHouseSinkConnector",
    "tasks.max": "1",
    "database": "default",
    "hostname": "default",
    "password": "A_PASSWORD",
    "port": "8443",
    "errors.retry.timeout": "60",
    "exactlyOnce": "false",
    "ssl": "true",
    "topics": "data_testing",
    "errors.tolerance": "none",
    "username": "default",
    "value.converter": "org.apache.kafka.connect.json.JsonConverter",
    "value.converter.schemas.enable": "false",
    "clickhouseSettings": "",
    "topic2TableMap": "",
    "consumer.override.max.poll.records": "50"
  }
}