RAM usage doesnot decrease while using rdkafka_performance

[shubhangiam]

TCPReplay playing packets 2500 packets per second , kafka_producer.sh produces msgs via tshark and rdkafka_performance in consumer running with r (sleep) 100 ms. Memory cosumption keeps on increasing above 200 MB and if stopped TCPreplay still after 20 mins , memory doesnot seem to decrease.... any idea..... I have same issue with my application so tried using rdkafka_performance but same issue..... memory doesnot go down.

[edenhill]

Is this RSS or Virtual memory?

[edenhill]

Try running it with valgrind again, but instead of Ctrl-C kill it with signal 6 (pkill -6 rdkafka_performance)

[shubhangiam]

tried with kill -6 pid ...

==80928==
==80928== 17,520 bytes in 1 blocks are still reachable in loss record 151 of 158
==80928== at 0x483DD99: calloc (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==80928== by 0x225018: rd_hdr_histogram_new (rdhdrhistogram.c:131)
==80928== by 0x1303A0: rd_avg_init (rdavg.h:242)
==80928== by 0x1303A0: rd_avg_init.constprop.0 (rdavg.h:229)
==80928== by 0x1429E6: rd_kafka_broker_add (rdkafka_broker.c:5453)
==80928== by 0x14306B: rd_kafka_broker_add_logical (rdkafka_broker.c:5614)
==80928== by 0x1942B3: rd_kafka_cgrp_new (rdkafka_cgrp.c:391)
==80928== by 0x12A557: rd_kafka_new (rdkafka.c:2305)
==80928== by 0x118FA2: main (rdkafka_performance.c:1658)
==80928==
==80928== 17,520 bytes in 1 blocks are still reachable in loss record 152 of 158
==80928== at 0x483DD99: calloc (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==80928== by 0x225018: rd_hdr_histogram_new (rdhdrhistogram.c:131)
==80928== by 0x1303A0: rd_avg_init (rdavg.h:242)
==80928== by 0x1303A0: rd_avg_init.constprop.0 (rdavg.h:229)
==80928== by 0x1429E6: rd_kafka_broker_add (rdkafka_broker.c:5453)
==80928== by 0x12A1A4: rd_kafka_new (rdkafka.c:2410)
==80928== by 0x118FA2: main (rdkafka_performance.c:1658)
==80928==
==80928== 17,520 bytes in 1 blocks are still reachable in loss record 153 of 158
==80928== at 0x483DD99: calloc (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==80928== by 0x225018: rd_hdr_histogram_new (rdhdrhistogram.c:131)
==80928== by 0x1303A0: rd_avg_init (rdavg.h:242)
==80928== by 0x1303A0: rd_avg_init.constprop.0 (rdavg.h:229)
==80928== by 0x1429E6: rd_kafka_broker_add (rdkafka_broker.c:5453)
==80928== by 0x1432AE: rd_kafka_brokers_add0 (rdkafka_broker.c:5933)
==80928== by 0x12A1C7: rd_kafka_new (rdkafka.c:2417)
==80928== by 0x118FA2: main (rdkafka_performance.c:1658)
==80928==
==80928== 17,520 bytes in 1 blocks are still reachable in loss record 154 of 158
==80928== at 0x483DD99: calloc (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==80928== by 0x225018: rd_hdr_histogram_new (rdhdrhistogram.c:131)
==80928== by 0x1303A0: rd_avg_init (rdavg.h:242)
==80928== by 0x1303A0: rd_avg_init.constprop.0 (rdavg.h:229)
==80928== by 0x1429E6: rd_kafka_broker_add (rdkafka_broker.c:5453)
==80928== by 0x14367C: rd_kafka_broker_update (rdkafka_broker.c:6008)
==80928== by 0x1C86B3: rd_kafka_parse_Metadata (rdkafka_metadata.c:470)
==80928== by 0x16DBAE: rd_kafka_handle_Metadata (rdkafka_request.c:1875)
==80928== by 0x15F1CB: rd_kafka_buf_callback (rdkafka_buf.c:492)
==80928== by 0x16B8CF: rd_kafka_op_handle_std (rdkafka_op.c:818)
==80928== by 0x16B97F: rd_kafka_op_handle (rdkafka_op.c:856)
==80928== by 0x163062: rd_kafka_q_serve (rdkafka_queue.c:497)
==80928== by 0x1295AB: rd_kafka_thread_main (rdkafka.c:2006)
==80928==
==80928== 503,200 bytes in 3,700 blocks are still reachable in loss record 155 of 158
==80928== at 0x483B7F3: malloc (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==80928== by 0x1F3030: rd_malloc (rd.h:127)
==80928== by 0x1F3030: rd_buf_init (rdbuf.c:369)
==80928== by 0x15E0DD: rd_kafka_buf_new0 (rdkafka_buf.c:112)
==80928== by 0x139020: rd_kafka_recv (rdkafka_broker.c:1822)
==80928== by 0x15CF17: rd_kafka_transport_io_event (rdkafka_transport.c:742)
==80928== by 0x15DADB: rd_kafka_transport_io_serve (rdkafka_transport.c:801)
==80928== by 0x145259: rd_kafka_broker_ops_io_serve (rdkafka_broker.c:3400)
==80928== by 0x145AD1: rd_kafka_broker_consumer_serve (rdkafka_broker.c:4982)
==80928== by 0x1471C2: rd_kafka_broker_serve (rdkafka_broker.c:5087)
==80928== by 0x14765C: rd_kafka_broker_thread_main (rdkafka_broker.c:5244)
==80928== by 0x49F76D9: start_thread (pthread_create.c:474)
==80928== by 0x4B33162: clone (clone.S:95)
==80928==
==80928== 1,657,600 bytes in 3,700 blocks are still reachable in loss record 156 of 158
==80928== at 0x483DD99: calloc (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==80928== by 0x15E0C3: rd_calloc (rd.h:121)
==80928== by 0x15E0C3: rd_kafka_buf_new0 (rdkafka_buf.c:108)
==80928== by 0x139020: rd_kafka_recv (rdkafka_broker.c:1822)
==80928== by 0x15CF17: rd_kafka_transport_io_event (rdkafka_transport.c:742)
==80928== by 0x15DADB: rd_kafka_transport_io_serve (rdkafka_transport.c:801)
==80928== by 0x145259: rd_kafka_broker_ops_io_serve (rdkafka_broker.c:3400)
==80928== by 0x145AD1: rd_kafka_broker_consumer_serve (rdkafka_broker.c:4982)
==80928== by 0x1471C2: rd_kafka_broker_serve (rdkafka_broker.c:5087)
==80928== by 0x14765C: rd_kafka_broker_thread_main (rdkafka_broker.c:5244)
==80928== by 0x49F76D9: start_thread (pthread_create.c:474)
==80928== by 0x4B33162: clone (clone.S:95)
==80928==
==80928== 60,404,408 bytes in 3,697 blocks are still reachable in loss record 157 of 158
==80928== at 0x483B7F3: malloc (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==80928== by 0x1F2C97: rd_malloc (rd.h:127)
==80928== by 0x1F2C97: rd_buf_alloc_segment0 (rdbuf.c:132)
==80928== by 0x1F2D7B: rd_buf_alloc_segment (rdbuf.c:165)
==80928== by 0x1F2EC1: rd_buf_write_ensure_contig (rdbuf.c:194)
==80928== by 0x138D97: rd_kafka_recv (rdkafka_broker.c:1895)
==80928== by 0x15CF17: rd_kafka_transport_io_event (rdkafka_transport.c:742)
==80928== by 0x15DADB: rd_kafka_transport_io_serve (rdkafka_transport.c:801)
==80928== by 0x145259: rd_kafka_broker_ops_io_serve (rdkafka_broker.c:3400)
==80928== by 0x145AD1: rd_kafka_broker_consumer_serve (rdkafka_broker.c:4982)
==80928== by 0x1471C2: rd_kafka_broker_serve (rdkafka_broker.c:5087)
==80928== by 0x14765C: rd_kafka_broker_thread_main (rdkafka_broker.c:5244)
==80928== by 0x49F76D9: start_thread (pthread_create.c:474)
==80928==
==80928== 62,471,688 bytes in 211,053 blocks are still reachable in loss record 158 of 158
==80928== at 0x483DD99: calloc (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==80928== by 0x165C1B: rd_calloc (rd.h:121)
==80928== by 0x165C1B: rd_kafka_op_new0 (rdkafka_op.c:247)
==80928== by 0x16A96F: rd_kafka_op_new_fetch_msg (rdkafka_op.c:739)
==80928== by 0x1D6C19: rd_kafka_msgset_reader_msg_v2 (rdkafka_msgset_reader.c:894)
==80928== by 0x1DA997: rd_kafka_msgset_reader_msgs_v2 (rdkafka_msgset_reader.c:989)
==80928== by 0x1DEBE7: rd_kafka_msgset_reader_v2 (rdkafka_msgset_reader.c:1135)
==80928== by 0x1D3CC4: rd_kafka_msgset_reader (rdkafka_msgset_reader.c:1273)
==80928== by 0x1D3CC4: rd_kafka_msgset_reader_run (rdkafka_msgset_reader.c:1333)
==80928== by 0x1E124D: rd_kafka_msgset_parse (rdkafka_msgset_reader.c:1441)
==80928== by 0x13E545: rd_kafka_fetch_reply_handle (rdkafka_broker.c:4604)
==80928== by 0x141B64: rd_kafka_broker_fetch_reply (rdkafka_broker.c:4664)
==80928== by 0x15F1CB: rd_kafka_buf_callback (rdkafka_buf.c:492)
==80928== by 0x138F8D: rd_kafka_req_response (rdkafka_broker.c:1789)
==80928== by 0x138F8D: rd_kafka_recv (rdkafka_broker.c:1907)
==80928==
==80928== LEAK SUMMARY:
==80928== definitely lost: 115 bytes in 1 blocks
==80928== indirectly lost: 0 bytes in 0 blocks
==80928== possibly lost: 0 bytes in 0 blocks
==80928== still reachable: 125,330,271 bytes in 222,301 blocks
==80928== suppressed: 21,818 bytes in 72 blocks
==80928==
==80928== For lists of detected and suppressed errors, rerun with: -s
==80928== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 5 from 5)
Aborted

[edenhill]

Perfect, can you provide your complete rdkafka_performance command line?
And also verify that you have made no modifications to rdkafka_performance or librdkafka?

[shubhangiam]

Sure Eden... Command line -
valgrind --suppressions=tests/librdkafka.suppressions --leak-check=full --show-leak-kinds=all --track-origins=yes examples/rdkafka_performance -G 1 -t Input_Master_Data -b localhost:9092 -r 100

I have not made change in single line in any of the rdkafka_performance or librdkafka ......

PFA Complete log -
rdkafka_performance_report.txt

[edenhill]

You are passing -r 100 which means the application will only read 100 msgs/s.
See the previously linked FAQ on the consumer memory usage and you will see why it will have a high memory usage (TLDR: all messages from a batch persist in memory until the last one has been read).

[shubhangiam]

-r is used as sleep in G mode if my understanding is correct ....

                rkmessage = rd_kafka_consumer_poll(rk, 1000);
                if (rkmessage) {
                        msg_consume(rkmessage, NULL);
                        rd_kafka_message_destroy(rkmessage);

                        /* Simulate processing time
                         * if `-r <rate>` was set. */
                        if (rate_sleep)
                                do_sleep(rate_sleep);
                }

[edenhill]

Run the code with valgrind again, as per previous instructions, but instead of quitting the program, send it signal 6 when it reaches the point where there are no more messages but it still consumes a lot of memory. This will give us an insight into what holds that memory.

[shubhangiam]

Ok..will try with valgrind.... but one more question, rdkafka_performance keeps on consuming msgs for veryyyy longgg time..... displays consuming 14 msgs for last half an hr, then 13 msgs for next half an hr....will decrease slowly ...thought i have mentioned r 1000..... why its consuming so slow ....

[shubhangiam]

At the EOD after 8 hrs, still rdkafka_performace is saying % 96330 messages (26619314 bytes) consumed in 12031137ms: 8 msgs/s (0.00 MB/s). Started with tcpreplay 1000 pps and consumer - r 1000 for 2 mins.

[shubhangiam]

rdkafka_performance doesnot reach till 0 msg/sec after 9 hrs......

valgrind output

==304862== 11,376 bytes in 1 blocks are still reachable in loss record 144 of 154
==304862== at 0x483DD99: calloc (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==304862== by 0x225018: rd_hdr_histogram_new (rdhdrhistogram.c:131)
==304862== by 0x1303A0: rd_avg_init (rdavg.h:242)
==304862== by 0x1303A0: rd_avg_init.constprop.0 (rdavg.h:229)
==304862== by 0x14298C: rd_kafka_broker_add (rdkafka_broker.c:5447)
==304862== by 0x14367C: rd_kafka_broker_update (rdkafka_broker.c:6008)
==304862== by 0x1C86B3: rd_kafka_parse_Metadata (rdkafka_metadata.c:470)
==304862== by 0x16DBAE: rd_kafka_handle_Metadata (rdkafka_request.c:1875)
==304862== by 0x15F1CB: rd_kafka_buf_callback (rdkafka_buf.c:492)
==304862== by 0x16B8CF: rd_kafka_op_handle_std (rdkafka_op.c:818)
==304862== by 0x16B97F: rd_kafka_op_handle (rdkafka_op.c:856)
==304862== by 0x163062: rd_kafka_q_serve (rdkafka_queue.c:497)
==304862== by 0x1295AB: rd_kafka_thread_main (rdkafka.c:2006)
==304862==
==304862== 11,376 bytes in 1 blocks are still reachable in loss record 145 of 154
==304862== at 0x483DD99: calloc (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==304862== by 0x225018: rd_hdr_histogram_new (rdhdrhistogram.c:131)
==304862== by 0x1303A0: rd_avg_init (rdavg.h:242)
==304862== by 0x1303A0: rd_avg_init.constprop.0 (rdavg.h:229)
==304862== by 0x1429AA: rd_kafka_broker_add (rdkafka_broker.c:5449)
==304862== by 0x14367C: rd_kafka_broker_update (rdkafka_broker.c:6008)
==304862== by 0x1C86B3: rd_kafka_parse_Metadata (rdkafka_metadata.c:470)
==304862== by 0x16DBAE: rd_kafka_handle_Metadata (rdkafka_request.c:1875)
==304862== by 0x15F1CB: rd_kafka_buf_callback (rdkafka_buf.c:492)
==304862== by 0x16B8CF: rd_kafka_op_handle_std (rdkafka_op.c:818)
==304862== by 0x16B97F: rd_kafka_op_handle (rdkafka_op.c:856)
==304862== by 0x163062: rd_kafka_q_serve (rdkafka_queue.c:497)
==304862== by 0x1295AB: rd_kafka_thread_main (rdkafka.c:2006)
==304862==
==304862== 13,424 bytes in 1 blocks are still reachable in loss record 146 of 154
==304862== at 0x483DD99: calloc (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==304862== by 0x225018: rd_hdr_histogram_new (rdhdrhistogram.c:131)
==304862== by 0x1303A0: rd_avg_init (rdavg.h:242)
==304862== by 0x1303A0: rd_avg_init.constprop.0 (rdavg.h:229)
==304862== by 0x1429C8: rd_kafka_broker_add (rdkafka_broker.c:5451)
==304862== by 0x14306B: rd_kafka_broker_add_logical (rdkafka_broker.c:5614)
==304862== by 0x1942B3: rd_kafka_cgrp_new (rdkafka_cgrp.c:391)
==304862== by 0x12A557: rd_kafka_new (rdkafka.c:2305)
==304862== by 0x118FA2: main (rdkafka_performance.c:1658)
==304862==
==304862== 13,424 bytes in 1 blocks are still reachable in loss record 147 of 154
==304862== at 0x483DD99: calloc (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==304862== by 0x225018: rd_hdr_histogram_new (rdhdrhistogram.c:131)
==304862== by 0x1303A0: rd_avg_init (rdavg.h:242)
==304862== by 0x1303A0: rd_avg_init.constprop.0 (rdavg.h:229)
==304862== by 0x1429C8: rd_kafka_broker_add (rdkafka_broker.c:5451)
==304862== by 0x12A1A4: rd_kafka_new (rdkafka.c:2410)
==304862== by 0x118FA2: main (rdkafka_performance.c:1658)
==304862==
==304862== 13,424 bytes in 1 blocks are still reachable in loss record 148 of 154
==304862== at 0x483DD99: calloc (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==304862== by 0x225018: rd_hdr_histogram_new (rdhdrhistogram.c:131)
==304862== by 0x1303A0: rd_avg_init (rdavg.h:242)
==304862== by 0x1303A0: rd_avg_init.constprop.0 (rdavg.h:229)
==304862== by 0x1429C8: rd_kafka_broker_add (rdkafka_broker.c:5451)
==304862== by 0x1432AE: rd_kafka_brokers_add0 (rdkafka_broker.c:5933)
==304862== by 0x12A1C7: rd_kafka_new (rdkafka.c:2417)
==304862== by 0x118FA2: main (rdkafka_performance.c:1658)
==304862==
==304862== 14,448 bytes in 1 blocks are still reachable in loss record 149 of 154
==304862== at 0x483DD99: calloc (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==304862== by 0x225018: rd_hdr_histogram_new (rdhdrhistogram.c:131)
==304862== by 0x14E400: rd_avg_init (rdavg.h:242)
==304862== by 0x14E400: rd_avg_init.constprop.0 (rdavg.h:229)
==304862== by 0x1501BA: rd_kafka_topic_new0 (rdkafka_topic.c:445)
==304862== by 0x1A548A: rd_kafka_toppar_get2 (rdkafka_partition.c:426)
==304862== by 0x1AE4C8: rd_kafka_topic_partition_ensure_toppar (rdkafka_partition.c:2991)
==304862== by 0x1B31B8: rd_kafka_assignment_add (rdkafka_assignment.c:740)
==304862== by 0x18ECDB: rd_kafka_cgrp_assign (rdkafka_cgrp.c:3369)
==304862== by 0x19E8CC: rd_kafka_cgrp_handle_assign_op (rdkafka_cgrp.c:4616)
==304862== by 0x19E8CC: rd_kafka_cgrp_op_serve (rdkafka_cgrp.c:4788)
==304862== by 0x163062: rd_kafka_q_serve (rdkafka_queue.c:497)
==304862== by 0x1295AB: rd_kafka_thread_main (rdkafka.c:2006)
==304862== by 0x49F76D9: start_thread (pthread_create.c:474)
==304862==
==304862== 14,448 bytes in 1 blocks are still reachable in loss record 150 of 154
==304862== at 0x483DD99: calloc (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==304862== by 0x225018: rd_hdr_histogram_new (rdhdrhistogram.c:131)
==304862== by 0x11B62A: rd_avg_rollover (rdavg.h:212)
==304862== by 0x11B62A: rd_kafka_stats_emit_avg (rdkafka.c:1343)
==304862== by 0x12D473: rd_kafka_stats_emit_all (rdkafka.c:1706)
==304862== by 0x12D473: rd_kafka_stats_emit_tmr_cb (rdkafka.c:1887)
==304862== by 0x15826C: rd_kafka_timers_run (rdkafka_timer.c:291)
==304862== by 0x1295C6: rd_kafka_thread_main (rdkafka.c:2010)
==304862== by 0x49F76D9: start_thread (pthread_create.c:474)
==304862== by 0x4B33162: clone (clone.S:95)
==304862==
==304862== 17,520 bytes in 1 blocks are still reachable in loss record 151 of 154
==304862== at 0x483DD99: calloc (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==304862== by 0x225018: rd_hdr_histogram_new (rdhdrhistogram.c:131)
==304862== by 0x1303A0: rd_avg_init (rdavg.h:242)
==304862== by 0x1303A0: rd_avg_init.constprop.0 (rdavg.h:229)
==304862== by 0x1429E6: rd_kafka_broker_add (rdkafka_broker.c:5453)
==304862== by 0x14306B: rd_kafka_broker_add_logical (rdkafka_broker.c:5614)
==304862== by 0x1942B3: rd_kafka_cgrp_new (rdkafka_cgrp.c:391)
==304862== by 0x12A557: rd_kafka_new (rdkafka.c:2305)
==304862== by 0x118FA2: main (rdkafka_performance.c:1658)
==304862==
==304862== 17,520 bytes in 1 blocks are still reachable in loss record 152 of 154
==304862== at 0x483DD99: calloc (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==304862== by 0x225018: rd_hdr_histogram_new (rdhdrhistogram.c:131)
==304862== by 0x1303A0: rd_avg_init (rdavg.h:242)
==304862== by 0x1303A0: rd_avg_init.constprop.0 (rdavg.h:229)
==304862== by 0x1429E6: rd_kafka_broker_add (rdkafka_broker.c:5453)
==304862== by 0x12A1A4: rd_kafka_new (rdkafka.c:2410)
==304862== by 0x118FA2: main (rdkafka_performance.c:1658)
==304862==
==304862== 17,520 bytes in 1 blocks are still reachable in loss record 153 of 154
==304862== at 0x483DD99: calloc (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==304862== by 0x225018: rd_hdr_histogram_new (rdhdrhistogram.c:131)
==304862== by 0x1303A0: rd_avg_init (rdavg.h:242)
==304862== by 0x1303A0: rd_avg_init.constprop.0 (rdavg.h:229)
==304862== by 0x1429E6: rd_kafka_broker_add (rdkafka_broker.c:5453)
==304862== by 0x1432AE: rd_kafka_brokers_add0 (rdkafka_broker.c:5933)
==304862== by 0x12A1C7: rd_kafka_new (rdkafka.c:2417)
==304862== by 0x118FA2: main (rdkafka_performance.c:1658)
==304862==
==304862== 17,520 bytes in 1 blocks are still reachable in loss record 154 of 154
==304862== at 0x483DD99: calloc (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==304862== by 0x225018: rd_hdr_histogram_new (rdhdrhistogram.c:131)
==304862== by 0x1303A0: rd_avg_init (rdavg.h:242)
==304862== by 0x1303A0: rd_avg_init.constprop.0 (rdavg.h:229)
==304862== by 0x1429E6: rd_kafka_broker_add (rdkafka_broker.c:5453)
==304862== by 0x14367C: rd_kafka_broker_update (rdkafka_broker.c:6008)
==304862== by 0x1C86B3: rd_kafka_parse_Metadata (rdkafka_metadata.c:470)
==304862== by 0x16DBAE: rd_kafka_handle_Metadata (rdkafka_request.c:1875)
==304862== by 0x15F1CB: rd_kafka_buf_callback (rdkafka_buf.c:492)
==304862== by 0x16B8CF: rd_kafka_op_handle_std (rdkafka_op.c:818)
==304862== by 0x16B97F: rd_kafka_op_handle (rdkafka_op.c:856)
==304862== by 0x163062: rd_kafka_q_serve (rdkafka_queue.c:497)
==304862== by 0x1295AB: rd_kafka_thread_main (rdkafka.c:2006)
==304862==
==304862== LEAK SUMMARY:
==304862== definitely lost: 112 bytes in 1 blocks
==304862== indirectly lost: 0 bytes in 0 blocks
==304862== possibly lost: 0 bytes in 0 blocks
==304862== still reachable: 293,398 bytes in 144 blocks
==304862== suppressed: 21,818 bytes in 72 blocks
==304862==
==304862== For lists of detected and suppressed errors, rerun with: -s
==304862== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 5 from 5)
Aborted

[edenhill]

Ok..will try with valgrind.... but one more question, rdkafka_performance keeps on consuming msgs for veryyyy longgg time..... displays consuming 14 msgs for last half an hr, then 13 msgs for next half an hr....will decrease slowly ...thought i have mentioned r 1000..... why its consuming so slow ....

The message rate is unfortunately sampled over the entire runtime of the program, so when messages stops coming in the rate will just gradually decrease.

The valgrind output looks fine, does not seem to consume more than 300kb of memory. I wonder if what you are seeing is memory fragmentation