Why do I use rdkafka_ Performance to test producers and consumers, their speed is only 6000 pieces / S?

[VisonZh]

My system is Contes, 32g, 16 core。
This is my running script：
./rdkafka_performance -P -t topic-5 -b 172.16.188.248:19092 -s 1024 -l -c 200000 -i 5000 -X queue.buffering.max.ms=1000 -X batch.size=5242880
./rdkafka_performance -G mygroup -t topic-5 -p 0 -b 172.16.188.248:19092 -l -c 200000 -i 5000

one topic and five partitions.
The Kafka cluster I deployed is installed on the same server. Three Kafka broker nodes are installed. It is a pseudo cluster

[VisonZh]

@edenhill

[edenhill]

Please provide some output from your run, like for example the last 100 or so lines.

[VisonZh]

./rdkafka_performance -P -t topic-50 -b 172.16.188.248:19092 -s 2048 -c 200000 -i 5000 -X queue.buffering.max.ms=1000 -X batch.size=5242880

% Sending 200000 messages of size 2048 bytes
%4|1632809369.795|CONFWARN|rdkafka#producer-1| [thrd:app]: Configuration property session.timeout.ms is a consumer property and will be ignored by this producer instance
%4|1632809369.795|CONFWARN|rdkafka#producer-1| [thrd:app]: Configuration property queued.min.messages is a consumer property and will be ignored by this producer instance
%4|1632809369.795|CONFWARN|rdkafka#producer-1| [thrd:app]: Configuration property offset_commit_cb is a consumer property and will be ignored by this producer instance
%4|1632809369.795|CONFWARN|rdkafka#producer-1| [thrd:app]: Configuration property auto.offset.reset is a consumer property and will be ignored by this producer instance
% 119400 messages produced (244531200 bytes), 19400 delivered (offset 4364, 0 failed) in 5005ms: 3875 msgs/s and 7.94 MB/s, 507 produce failures, 100000 in queue, no compression
% 135890 messages produced (278302720 bytes), 35890 delivered (offset 969, 0 failed) in 10011ms: 3584 msgs/s and 7.34 MB/s, 1018 produce failures, 100000 in queue, no compression
% 152380 messages produced (312074240 bytes), 52380 delivered (offset 3879, 0 failed) in 15013ms: 3488 msgs/s and 7.15 MB/s, 1532 produce failures, 100000 in queue, no compression
% 169840 messages produced (347832320 bytes), 69840 delivered (offset 969, 0 failed) in 20014ms: 3489 msgs/s and 7.15 MB/s, 2044 produce failures, 100000 in queue, no compression
% 185845 messages produced (380610560 bytes), 85845 delivered (offset 4849, 0 failed) in 25016ms: 3431 msgs/s and 7.03 MB/s, 2555 produce failures, 100000 in queue, no compression
% 200000 messages produced (409600000 bytes), 102820 delivered (offset 969, 0 failed) in 30101ms: 3415 msgs/s and 7.00 MB/s, 2994 produce failures, 97180 in queue, no compression
% 200000 messages produced (409600000 bytes), 119885 delivered (offset 4849, 0 failed) in 35184ms: 3407 msgs/s and 6.98 MB/s, 2994 produce failures, 80115 in queue, no compression
% 200000 messages produced (409600000 bytes), 136860 delivered (offset 46074, 0 failed) in 40202ms: 3404 msgs/s and 6.97 MB/s, 2994 produce failures, 63140 in queue, no compression
% 200000 messages produced (409600000 bytes), 153835 delivered (offset 63049, 0 failed) in 45263ms: 3398 msgs/s and 6.96 MB/s, 2994 produce failures, 46165 in queue, no compression
% 200000 messages produced (409600000 bytes), 171295 delivered (offset 80509, 0 failed) in 50500ms: 3391 msgs/s and 6.95 MB/s, 2994 produce failures, 28705 in queue, no compression
% 200000 messages produced (409600000 bytes), 188270 delivered (offset 97484, 0 failed) in 55618ms: 3384 msgs/s and 6.93 MB/s, 2994 produce failures, 11730 in queue, no compression
% 2994 backpressures for 200000 produce calls: 1.497% backpressure rate
% 200000 messages produced (409600000 bytes), 200000 delivered (offset 109214, 0 failed) in 59047ms: 3387 msgs/s and 6.94 MB/s, 2994 produce failures, 0 in queue, no compression

[VisonZh]

one topic and 50 partitions。

[edenhill]

What version of librdkafka?