Skip to content

Expand Types of Sink Output to Match Clickhouse e.g. UInt64 #442

@PaulST2

Description

@PaulST2

Is your feature request related to a problem? Please describe.
Clickhouse supports a wide range of data types, as does Protobuf (for serdes in Kafka Schema Registry). Whilst the data types that Clickhouse-Kafka-Connect Sink supports is a much smaller subset.

Recently, using Kafka in conjunction with Schema Registry, Clickhouse-Kafka-Connect Sink, and Protobuf I attempted to write Protobuf uint64 data to a Clickhouse UInt64 table column. I discovered (from the logs) that Clickhouse-Kafka-Connect Sink converts the Protobuf uint64 to int64, attempts to write that (and fails).

I don't believe this is an intuitive expectation, and indeed a significant limitation on the usefulness of the Clickhouse-Kafka-Connect Sink.

Describe the solution you'd like
Please consider a much wider range of data type support to more closely match the range which Clickhouse & Protobuf supports. Most particularly, Clickhouse UInts of various widths, but indeed all types that can be matched or sensibly converted between, as available in Protobuf, and Avro.

(In this specific case I expected Protobuf uint64 to automatically convert to Clickhouse UInt64 type.)

Describe alternatives you've considered
CustomSMT and Custom Converters. These come with considerable complexity, however.

Additional context
Clickhouse-Kafka-Connect Sink - search the page for Supported Data Types:
https://clickhouse.com/docs/en/integrations/kafka/clickhouse-kafka-connect-sink

Clickhouse supported data types:
https://clickhouse.com/docs/en/sql-reference/data-types

Protobuf supported (scalar) data types:
https://protobuf.dev/programming-guides/proto3/#scalar

Avro supported data types:
https://avro.apache.org/docs/1.11.1/specification/.

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions