Can kcat/librdkafka tell me what the message batch sizes are, when I consume from a topic?

[wushujames]

I am auditing some of my kafka topics to see whether producers are using compression and if so, what are the batch sizes they use.

To do this, I consume from the topic with debuglogs turned way up (kcat -d all). It returns the following output:

%7|1644459346.210|CONSUME|rdkafka#consumer-1| [thrd:broker4:9092/4]: broker4:9092/4: Enqueue 1 message(s) (382 bytes, 1 ops) on topicname [0] fetch queue (qlen 0, v2, last_offset 4164, 0 ctrl msgs, lz4)

At the end, I see that "lz4" tells me that the messages were compressed.

Are there any debuglogs that will tell me the message batch size? I know that kcat/librdkafka have to receive and open up the message batches, so I was hoping it might print out that information somewhere.

I realize that this is a non-standard usage of kcat, and that debuglogs are not intended to be a stable API/interface. I'm okay with the instability.

Thanks!
-James

[edenhill]

Hi James,

you will have to rely on the debug logs (which you correctly say are not subject to stability/compat guarantees, but in practice rarely change).
The above log line tells you:

• there was 1 message in batch (Enqueue N message)
• the decompressed batch size for all messages was 382 bytes (including framing) (B bytes, ..)
• the batch was compressed with lz4

So the only thing missing is the compressed size.

There are also histograms tracking the number of messages and total size per batch, see the per topic batchsize and batchcnt metrics in https://github.com/edenhill/librdkafka/blob/master/STATISTICS.md.

[wushujames]

Thanks Magnus! Just to make sure I used the right vocab, when I said I was looking for "batch size", what I meant was the producer's idea of a "batch", where it accumulates a set of messages before sending them all to the broker as a single request. (Is that a MessageSet?) The one that (in Java) is controlled by batch.size.

Is THAT the value that is displayed by there was 1 message in batch (Enqueue N message)?

I just did some experiments:

• kcat -c 1 yielded Enqueue 1 message(s)
• kcat -c 10 yielded Enqueue 10 message(s)
• kcat -c 100 yielded Enqueue 100 message(s)

Which makes me think it's not actually measuring the batch/MessageSet size as stored in the topic, but rather that it's measuring something like the number of messages that the consumer retrieved.