Question on end to end deduplication

[azhur]

Does Glassflow guarantee that deduplicated records end up in ClickHouse exactly once?
I mean, even if records are deduplicated by Glassflow, the 'write to clickhouse' operation might still fail, and then retries might introduce duplicates to clickhouse so anyway ReplacingMergeTree engine is needed for the table?

Do I miss anything here?

[PabloPardoGarcia]

Hi Azhur, that's a great question. GlassFlow handles deduplication internally, but there are a couple of edge cases where duplicates can still make their way into ClickHouse:

• If the Kafka sink fails, the message is routed to the dead-letter queue (DLQ). At that point, it's up to the user to reprocess it, and depending on how that's handled, duplicates may be introduced.
• If the client’s Kafka is down for longer than the deduplication window, we can also introduce duplicates. This is because NATs (used internally for transport) currently checks the time window agains the timestamp when the message reached NATs, not the original Kafka event timestamp. As a result, delayed messages may fall outside the deduplication window. We are evaluating options for supporting custom timestamps for deduplication, which would help mitigate this, but for now, it’s something to be aware of.