Optimal scenario for a long-living producer with/without delivery-callback

[SoulfreezerXP]

I have questions regarding the optimal use of the producer once in a
scenario with delivery-callback and once without.

---

Without delivery-callback:

Is it generally mandatory to call poll() here? The application is
long-lived and the Kafka handling (produce-main-loop + potential poll() ) happens in
one separate thread. My tests with 10 million messages showed me that if you call poll()
after EACH msg, sending the whole amount is much slower by a factor of 10-100.
Also calling poll() once here only each 10000 iterations was not helping.
Completely without poll() it takes only 3-4 seconds (probably the implicit flush from
poll avoids bulk-sends out of the queue?).

Is it possible to omit poll() altogether here if you have NOT registered
a delivery-callback via config "dr_cb", or do you still need it
to keep the producer alive? If so, how often should poll
be called just for keeping it alive, because it will flush implicitly (?)
which would avoid bulk-sends? Is it even a bad thing if the producer
signs off? Wouldn't it reconnect automatically and promptly?

---

With delivery-callback:

Assuming that poll() flushes, how often should poll() be called ? Time bound and
greater than queue.buffering.max.ms ? A single poll() will then pick up and call
ALL completed callbacks (blocking, right)? Does it make sense here to make a
separate thread just for the polling so that the delivery callbacks are handled in this
thread context and the produce loop (Also in an own thread separated from application thread)
is free to run?

Maybe you can provide 2 optimal scenarios, one
with delivery-callbacks and one without?
Which method would you recommend (high throughput is very important!),
if you don't want to/can't handle faulty messages anyway?

[edenhill]

If you don't register any callbacks (dr_msg_cb, error_cb, stats_cb, etc) then you don't need to call poll().

But not having a delivery report callback (dr_msg_cb) means you will have no way of knowing if your messages were successfully produced or not, which is effectively equivalent to not produce():ing the messages at all - in drastical terms - so we highly advice against this approach.

As for poll() overhead; you don't need to call it once for each message produced, in practice you only need to call it when produce() returns ERR__QUEUE_FULL, but that will delay error reporting until the queue is full, so I'd recommend calling poll() once per second or every N messages produce()d.

[edenhill]

Having a separate thread that just calls poll() is a good approach too, yeah.

[SoulfreezerXP]

Thanks for the quick reply!

I think I will then stick with the recommendation with the poll()
for every second (+ own poll-thread if necessary).

Would produce() for sure return the "ERR__QUEUE_FULL" at some point, if I
never call poll() (with registered callback ) or is there something like an internal
auto-flush then (After -> queue.buffering.max.ms?)?
How does it behave if you don't have any callback registered?
Then there must be such an automatic mechanism,
if the poll() may be omitted there, right?