Questions regarding number of rdkafka threads

[mohitreddy1996]

We have a rust application running which makes use of https://github.com/fede1024/rust-rdkafka. We started noticing unusual number of process threads and started looking into them.

Background:

Rust application creates an "AdminClient" which is a producer underneath (?) and holds for its lifetime. It then creates a consumer for each partition of topic_1 (by default we have 32 partitions for each topic) and topic_2. Creates producers for each partition of topic topic_3 and topic_4 and keeps them for its lifetime. We have a 4 broker setup.

Logged into the machine, did -

top -H -b -n 1 > threads.txt

awk '{print $12}' threads.txt | sort -n | uniq -c

Got the output -

      1 admin
  .....
     64 producer
    257 rdk:broker-1
    129 rdk:broker1
    129 rdk:broker2
    129 rdk:broker3
    129 rdk:broker4
    129 rdk:main
      ..
     33 tokio-runtime-w
      1 top

From, https://github.com/confluentinc/librdkafka/wiki/FAQ#number-of-internal-threads, 129 makes sense (4 brokers x 32 producers and consumer + 1 admin client). 64 producer is something which I am yet to debug (may be this is coming from Rust client?).

257 rdk:broker-1 is something which is not clear yet. AFAIU, from #3343 (comment), this corresponds to bootstrap server, but I don't understand how could there be 257 threads for 129 clients?