Lack of truly guaranteed log delivery with Kafka Sink

[evevseev]

Hi, everyone!

I am facing a case where guaranteed delivery of logs from files to Kafka is required (not a single line can be lost).

During testing, I encountered the issue that with the default rdkafka options and disk buffering enabled, logs older than 5 minutes are lost when Kafka is down (this is the default timeout for rdkafka), and the buffer and acknowledgements do not facilitate resending since the logs have already been placed into rdkafka’s internal queue, as far as I understand.

Of course, one could disable timeouts at the rdkafka level and force it to always keep retrying, but then another problem arises — upon restart, logs that have already been queued in rdkafka will be lost.

I can see an issue #8468 for that, but maybe there are some workarounds? It is a blocker for us and I don't think this case is particularly rare :)