Clarification on docs around max.poll.interval.ms

[rognorak]

Hi All -

For Ref: Maximum allowed time between calls to consume messages (e.g., rd_kafka_consumer_poll()) for high-level consumers. If this interval is exceeded the consumer is considered failed and the group will rebalance in order to reassign the partitions to another consumer group member. Warning: Offset commits may be not possible at this point. Note: It is recommended to set enable.auto.offset.store=false for long-time processing applications and then explicitly store offsets (using offsets_store()) after message processing, to make sure offsets are not auto-committed prior to processing has finished. The interval is checked two times per second. See KIP-62 for more information. Type: integer

Id like some clarity around the last part of this comment:

Note: It is recommended to set enable.auto.offset.store=false for long-time processing applications and then explicitly store offsets (using offsets_store()) after message processing, to make sure offsets are not auto-committed prior to processing has finished.

My consumer has:
enable.auto.offset.store is TRUE ( default )
also has "enable.auto.commit": false

I thought in that configuration, I would be safe from auto-committing? I read Kip-62, and I am still not clear -
but it sounds as though in the above comment, I might auto commit an un-processed message if something is long running, and potentially lose a message?.

Is this true, despite that enable.auto.commit is false ?

Please clarify,
Thank You

[edenhill]

You are fine with enable.auto.commit=false, nothing will be automatically committed.

But enable.auto.offset.store=false and enable.auto.commit=true is usually more convenient; you simply store the offset of the last processed message after processing and the auto-committer will commit it in the background according to its configured interval, as well as on rebalance.

The important part, though, is that the default (auto store=true, auto commit=true) is not suitable for long-running message processing since a message may be auto-commmitted prior to processing has completed.