Skip to content

Kafka-exporter stops returning metrics under high cluster load #497

@DDDFiish

Description

@DDDFiish

Description

I'm using the latest version of kafka-exporter (1.9.0) with the following deployment parameters:

--verbosity=0 \
--kafka.server=kafka1:9092 \
--kafka.server=kafka2:9092 \
--kafka.server=kafka3:9092 \
--kafka.version=3.9.0 \
--sasl.enabled \
--sasl.username=$(SCRAM_USERNAME) \
--sasl.password=$(SCRAM_PASSWORD) \
--sasl.mechanism=scram-sha256

The exporter is deployed independently for each Kafka 3.9.0 cluster (KRaft mode), with each cluster having 10 nodes and typically a few hundred topics.

When the Kafka clusters experience high load (high network bandwidth but stable production/consumption), kafka-exporter stops returning any metrics. This appears as sudden zero values in Grafana dashboards.

I increased the log verbosity to level 2 --verbosity=2, but still did not observe any useful information, including error logs.

Notably:

  1. The exporter continues to log INFO-level messages without any ERROR entries
  2. The problem occurs consistently under high load conditions
  3. Metrics resume when cluster load decreases

What could cause the exporter to silently stop serving metrics?

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions