r

[mkulandai]

rd_kafka_consumer_poll return only one record instead of array of records. Is there any other API available to get the list of records during that poll. e.g assume queue/partition is having 100 records and when hit the poll api can i go and get the array of 100 records and process in a loop instead of hitting the poll 100 times.

[Quuxplusone]

There is rd_kafka_consume_batch. I'm going to guess that you could have a loop body where you first call rd_kafka_consume_batch to get a bunch of messages, and then follow up with a single call to rd_kafka_consumer_poll to handle callbacks. My very ill-informed reading of the docs suggests that you can't just call rd_kafka_consume_batch-or- rd_kafka_consume in a loop, because you still need to call rd_kafka_consumer_poll-or-rd_kafka_poll every so often for correctness.

https://github.com/edenhill/librdkafka/blob/2aadc45974d32b7c7ff91805588be3fc9951f4ad/examples/rdkafka_performance.c#L1576 is an official example, and it shows a loop body that calls rd_kafka_consume_batch-or-rd_kafka_consume_callback_queue followed by rd_kafka_poll.