producer flush option with fluent bit

[manivannan-g]

Hi,

I'm using fluent bit for the log collecting and pushing to kafka. Upon fluent bit pod termination, I see the following message and also resulting in data loss:

[2022/02/28 08:20:44] [engine] caught signal (SIGTERM)
[2022/02/28 08:20:44] [ info] [input] pausing log-generator
[2022/02/28 08:20:44] [ info] [input] pausing storage_backlog.1
[2022/02/28 08:20:44] [ warn] [engine] service will shutdown in max 30 seconds
[2022/02/28 08:20:44] [ info] [engine] service has stopped (0 pending tasks)
[2022/02/28 08:20:44] [ info] [input:tail:log-generator] inotify_fs_remove(): inode=20022121 watch_fd=3
[2022/02/28 08:20:44] [ info] [output:s3:log-generator] Sending all locally buffered data to S3
[2022/02/28 08:20:44] [ info] [output:s3:log-generator] Successfully uploaded object /log-generator-logs/2022-02-28/08/19/44/tmp.log-generator.log-generator-5fcb44c9b7-zttbg.server.log-object4snCVYrI
[2022/02/28 08:20:44] [ warn] [output:kafka:kafka.1] fluent-bit#producer-1: [thrd:app]: Producer terminating with 436 messages (100238 bytes) still in queue or transit: use flush() to wait for outstanding message delivery

I tried finding a relevant option to enable flush https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md, unfortunately I couldn't find in there.

Here's my kafka output configuration:

    [OUTPUT]
        Name                                kafka
        Match                               *
        Brokers                             kafka-bootstrap.example.com:9094
        Topics                              fluentbit-log-gen-logs
        rdkafka.queue.buffering.max.kbytes  1000
        rdkafka.request.required.acks       1

Env: Kubernetes
Fluentbit version: 1.8.12

Is there a work around to ensure to wait till all messages flushed out before exit? Thanks!

[edenhill]

Are you calling rd_kafka_flush() as instructed?

[manivannan-g]

Hi @edenhill,
Yes, I did try setting this variable through the fluent bit rdkafka options. But I saw the below error message in fluent

[2022/02/28 06:52:38] [error] [output:kafka:kafka.1] cannot configure 'flush' property

Exact config:

    [OUTPUT]
        Name                                kafka
        Match                               *
        Brokers                             kafka-bootstrap.example.com:9094
        Topics                              fluentbit-log-gen-logs
        rdkafka.queue.buffering.max.kbytes  1000
        rdkafka.request.required.acks       1
        rdkafka.flush                       5000

I also tried setting rdkafka.flush -1 but both the times it shown the same message in fluent bit.

[edenhill]

This sounds like a fluent issue.

[manivannan-g]

Thanks! @edenhill I'll check with the fluent bit community.