Inserts failing silently #458
Description
Describe the bug
Missing events in the target table. 3 events at offsets 7407350 - 7407352 are not present in clickhouse table. The connector has attempted to write them, we could see it from the DEBUG logs, but the insert had actually failed. Found insert query in query_log for id "eed49adf-feb9-46a7-abec-e44d9bdc03c2". it failed with MEMORY_LIMIT_EXCEEDED. Shouldn't this have made the sink connector retried or put the events in a DLQ like it was configured to? Instead, I could see this in the connector's logs
|task-0] Response Summary - Written Bytes: [7854], Written Rows: [3] - (QueryId: [eed49adf-feb9-46a7-abec-e44d9bdc03c2])
Steps to reproduce
Not sure how to reproduce
Expected behaviour
Failed inserts should have raised exception and caused it to be retried / put in dlq as configured.
Available logs
-
sink connector logs for query id eed49adf-feb9-46a7-abec-e44d9bdc03c2: https://gist.github.com/jonscyr/5f843f91f62429b5217a69007108c2c7
-
records from clickhouse's system.query_logs for same query id: https://gist.github.com/jonscyr/abff0e26c122dc1b8b3d80a00dcd93e3
Configuration
https://gist.github.com/jonscyr/ef2f400a30a6b63a019d77b8a77f23b4
Environment
We have half-stack setup (CH cloud + strimzi kafka). We've been facing this issue where some batches events gets lost every 1 or two months. We have a script for validator running every day which raises this.
Our kafka-connect are running on LOG_LEVEL=DEBUG.
- Kafka-Connect version:
v1.2.3 - Kafka Connect configuration: https://gist.github.com/jonscyr/6566d4f45db6dae63cb0b79a13ff127f
- Kafka version:
3.2 - Kafka environment:
Strimzi 0.33.0 - OS: Linux
ClickHouse server
- ClickHouse Server version:
24.6 CREATE TABLEstatements for tables involved: https://gist.github.com/jonscyr/6573e6cb84e37eca7547349af7fa1168- Sample data for all the table: https://gist.github.com/jonscyr/abbecbd14f56012b6a99bdca514951a6