Producer application behavior on termination sequence #4129
Unanswered
asanchez1987
asked this question in
Q&A
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
-
In a producer application this is implemented at the end:
Does librdkafka attempt new deliveries after 4. but before 7. ?
Is there any extra activity happening with the messages (i.e. operation on msgs is not stopped at this point) ? or does operational stop require rd_kafka_destroy() ?
I see there's a rd_kafka_consume_stop() but not an analogous rd_kafka_produce_stop() that's why I'm asking.
With the end goal of having consistency with messages (save to disk undelivered messages, avoid duplicates on restart, and avoid losing msgs) but also minimize time needed for the termination sequence:
Is it possible to quantify/estimate the time a blocking rd_kafka_purge() operation needs to finish purging?
If in-flight msgs aren't purged, is this purge time dependent on request.timeout.ms (default 30000 ms, 30 s)? If it's the case it'd be too much blocking time for the end goal of this application, since I'm looking for a <1s purging completion time.
I'm thinking in doing the purge with RD_KAFKA_PURGE_F_QUEUE only, without the in-flight and in a blocking way. Then I guess in-flight for which the ack doesn't arrive are transformed back to in-queue when the purging thread finishes, or marked as delivered otherwise. Then the dr_msg_cb() should only care about requeuing to the application local queue and save to disk RD_KAFKA_RESP_ERR__PURGE_QUEUE and RD_KAFKA_RESP_ERR__MSG_TIMED_OUT.
Thanks for any suggestions/answers on this admittedly decent wall of questions around termination sequence.
Beta Was this translation helpful? Give feedback.
All reactions