Skip to content

How to custom a Converter using with WITHCONVERTER? #1662

@buxiua

Description

@buxiua

Hi guys! I'm a newer using Kafka. It's a awesome project about Kafka Connector.

I use the MqttSourceConnector to collect data from mqtt to kafka, just like the example.

name=mqtt-source
connector.class=io.lenses.streamreactor.connect.mqtt.source.MqttSourceConnector
tasks.max=1
connect.mqtt.kcql=INSERT INTO mqtt SELECT * FROM /mjson WITHCONVERTER=`io.lenses.streamreactor.connect.converters.source.JsonSimpleConverter`
[connect.mqtt.client.id](http://connect.mqtt.client.id/)=dm_source_id
connect.mqtt.hosts=tcp://mqtt:1883
connect.mqtt.service.quality=1

in property connect.mqtt.kcql,the WITHCONVERTER is io.lenses.streamreactor.connect.converters.source.JsonSimpleConverter.

When I test for it, I got message with some fields that i don't want to get, like schema, and I also need to get some other information from mqtt. So, It's great for me to custom Converter. Unfortunately, I can not find docs about How to custome Converter.

But get some information from the issue #1015

To implement your converter, you can check out the project and create a new module depending on kafka-connect-common and implement io.lenses.streamreactor.connect.converters.source.Converter. Ensure that the resulting JAR is placed in your plugin.path location.

so I did. with some steps like:

  1. git clone ....
  2. install scala and install sbt (because I'm a javaer)
  3. download many dependencies with bad network which make me crazy.
    ......

Finally, I complie and build the kafka-connect-common successfully and i can't find the interface io.lenses.streamreactor.connect.converters.source.Converter, but find it in kafka-connect-sql-common, and i build and using it to custom my own Converter com.inside.ByteArraySourceConverter just extends BytesConverter. But unfortunately it not work correctly.

Image

left is using io.lenses.streamreactor.connect.converters.source.JsonSimpleConverter and it run correctly, right is bad, it just like stoped before Creating task.

So anybody can tell me what wrong with it and How can i cutom a Converter using with WITHCONVERTER? Thank for you very much, Please forgive my poor English.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions