connection_close_rate is high

[amankothari04]

I am writing from around 1000 servers to AWS MSK using this plugin. They write continuously throughout the day.
I see the aws.kafka.connection_creation_rate metric graph and the aws.kafka.connection_close_rate metric graph are the same and are pretty high (Around 500 connections are made and killed every 1 minute). Ideally, I would expect a connection to be made and never close as the servers are writing continuously to Kafka.

I tried to reproduce the same on another similar Kafka cluster.
I produced messages using the "kafka-producer-perf-test.sh" script that comes with Kafka.
(I basically ran the script in a for-loop for 1000 times)
I made sure that the "message_in_per_sec" and "bytes_in_per_sec" was, if not the same, at least more than my older (problematic) cluster.
But the "connection_close_rate" is just 25 (not as high as the problematic cluster).
The CPU is also half of what I see on the older cluster.

Can anyone help me understand why I see such a high connection_close_rate on my first cluster?

Here is my plugin config:

<label @default_kafka >
  <match *.**>
    @type copy
  <store>
    @type kafka2

    # list of seed brokers
    brokers {{ fluentd_env_vars[env]["xyz"] }}
    use_event_time true

    ssl_ca_cert "/path/CA_cert.pem"
    ssl_client_cert "/path/client_cert.pem"
    ssl_client_cert_key "/path/client_key.pem"
    
  # buffer settings
    <buffer topic>
      @type file
      path /var/log/td-agent/buffer/kafka
      flush_interval 1s
    </buffer>
      # data type settings
      <format>
        @type json
      </format>

      # topic settings
      topic_key topic
      default_topic log-messages

      # producer settings
      required_acks -1
      compression_codec gzip
      </store>
  </match>
  </label>

[Gauravshah]

its because of fluent/fluent-plugin-kafka#454