Prometheus Exporter pod/metrics not being created/exposed

[rava-dosa]

Please use this to only for bug reports. For questions or when you need help, you can use the GitHub Discussions, our #strimzi Slack channel or out user mailing list.

Describe the bug
Prometheus Exporter pod is not being created

To Reproduce
Steps to reproduce the behavior:
kubectl apply -f file.yaml

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: stage-kafka
spec:
  kafka:
    replicas: 1
    listeners:
      - name: plain
        port: 9092
        type: internal
        tls: false
      - name: tls
        port: 9093
        type: internal
        tls: true
        authentication:
          type: tls
      - name: external
        port: 9094
        type: loadbalancer
        tls: false
        configuration:
          bootstrap:
            loadBalancerIP: {some_ip}
            annotations:
              cloud.google.com/load-balancer-type: "Internal"
          brokers:
            - broker: 0
              annotations:
                cloud.google.com/load-balancer-type: "Internal"
    resources:
      limits:
        memory: 14Gi
        cpu: "4"
    storage:
      type: jbod
      volumes:
        - id: 0
          type: persistent-claim
          size: 70Gi
          class: ssd
          deleteClaim: false
    config:
      offsets.topic.replication.factor: 1
      transaction.state.log.replication.factor: 1
      transaction.state.log.min.isr: 1
      default.replication.factor: 1
      min.insync.replicas: 1
    metricsConfig:
      type: jmxPrometheusExporter
      valueFrom:
        configMapKeyRef:
          name: kafka-metrics
          key: kafka-metrics-config.yml
  zookeeper:
    replicas: 1
    storage:
      type: persistent-claim
      size: 10Gi
      class: ssd
      deleteClaim: false
  kafkaExporter:
    topicRegex: ".*"
    groupRegex: ".*"
  entityOperator:
    topicOperator: {}
    userOperator: {}

---
kind: ConfigMap
apiVersion: v1
metadata:
  name: kafka-metrics
  labels:
    app: strimzi
data:
  kafka-metrics-config.yml: |
    # See https://github.com/prometheus/jmx_exporter for more info about JMX Prometheus Exporter metrics
    lowercaseOutputName: true
    rules:
    # Special cases and very specific rules
    - pattern: kafka.server<type=(.+), name=(.+), clientId=(.+), topic=(.+), partition=(.*)><>Value
      name: kafka_server_$1_$2
      type: GAUGE
      labels:
       clientId: "$3"
       topic: "$4"
       partition: "$5"
    - pattern: kafka.server<type=(.+), name=(.+), clientId=(.+), brokerHost=(.+), brokerPort=(.+)><>Value
      name: kafka_server_$1_$2
      type: GAUGE
      labels:
       clientId: "$3"
       broker: "$4:$5"
    - pattern: kafka.server<type=(.+), cipher=(.+), protocol=(.+), listener=(.+), networkProcessor=(.+)><>connections
      name: kafka_server_$1_connections_tls_info
      type: GAUGE
      labels:
        cipher: "$2"
        protocol: "$3"
        listener: "$4"
        networkProcessor: "$5"
    - pattern: kafka.server<type=(.+), clientSoftwareName=(.+), clientSoftwareVersion=(.+), listener=(.+), networkProcessor=(.+)><>connections
      name: kafka_server_$1_connections_software
      type: GAUGE
      labels:
        clientSoftwareName: "$2"
        clientSoftwareVersion: "$3"
        listener: "$4"
        networkProcessor: "$5"
    - pattern: "kafka.server<type=(.+), listener=(.+), networkProcessor=(.+)><>(.+):"
      name: kafka_server_$1_$4
      type: GAUGE
      labels:
       listener: "$2"
       networkProcessor: "$3"
    - pattern: kafka.server<type=(.+), listener=(.+), networkProcessor=(.+)><>(.+)
      name: kafka_server_$1_$4
      type: GAUGE
      labels:
       listener: "$2"
       networkProcessor: "$3"
    # Some percent metrics use MeanRate attribute
    # Ex) kafka.server<type=(KafkaRequestHandlerPool), name=(RequestHandlerAvgIdlePercent)><>MeanRate
    - pattern: kafka.(\w+)<type=(.+), name=(.+)Percent\w*><>MeanRate
      name: kafka_$1_$2_$3_percent
      type: GAUGE
    # Generic gauges for percents
    - pattern: kafka.(\w+)<type=(.+), name=(.+)Percent\w*><>Value
      name: kafka_$1_$2_$3_percent
      type: GAUGE
    - pattern: kafka.(\w+)<type=(.+), name=(.+)Percent\w*, (.+)=(.+)><>Value
      name: kafka_$1_$2_$3_percent
      type: GAUGE
      labels:
        "$4": "$5"
    # Generic per-second counters with 0-2 key/value pairs
    - pattern: kafka.(\w+)<type=(.+), name=(.+)PerSec\w*, (.+)=(.+), (.+)=(.+)><>Count
      name: kafka_$1_$2_$3_total
      type: COUNTER
      labels:
        "$4": "$5"
        "$6": "$7"
    - pattern: kafka.(\w+)<type=(.+), name=(.+)PerSec\w*, (.+)=(.+)><>Count
      name: kafka_$1_$2_$3_total
      type: COUNTER
      labels:
        "$4": "$5"
    - pattern: kafka.(\w+)<type=(.+), name=(.+)PerSec\w*><>Count
      name: kafka_$1_$2_$3_total
      type: COUNTER
    # Generic gauges with 0-2 key/value pairs
    - pattern: kafka.(\w+)<type=(.+), name=(.+), (.+)=(.+), (.+)=(.+)><>Value
      name: kafka_$1_$2_$3
      type: GAUGE
      labels:
        "$4": "$5"
        "$6": "$7"
    - pattern: kafka.(\w+)<type=(.+), name=(.+), (.+)=(.+)><>Value
      name: kafka_$1_$2_$3
      type: GAUGE
      labels:
        "$4": "$5"
    - pattern: kafka.(\w+)<type=(.+), name=(.+)><>Value
      name: kafka_$1_$2_$3
      type: GAUGE
    # Emulate Prometheus 'Summary' metrics for the exported 'Histogram's.
    # Note that these are missing the '_sum' metric!
    - pattern: kafka.(\w+)<type=(.+), name=(.+), (.+)=(.+), (.+)=(.+)><>Count
      name: kafka_$1_$2_$3_count
      type: COUNTER
      labels:
        "$4": "$5"
        "$6": "$7"
    - pattern: kafka.(\w+)<type=(.+), name=(.+), (.+)=(.*), (.+)=(.+)><>(\d+)thPercentile
      name: kafka_$1_$2_$3
      type: GAUGE
      labels:
        "$4": "$5"
        "$6": "$7"
        quantile: "0.$8"
    - pattern: kafka.(\w+)<type=(.+), name=(.+), (.+)=(.+)><>Count
      name: kafka_$1_$2_$3_count
      type: COUNTER
      labels:
        "$4": "$5"
    - pattern: kafka.(\w+)<type=(.+), name=(.+), (.+)=(.*)><>(\d+)thPercentile
      name: kafka_$1_$2_$3
      type: GAUGE
      labels:
        "$4": "$5"
        quantile: "0.$6"
    - pattern: kafka.(\w+)<type=(.+), name=(.+)><>Count
      name: kafka_$1_$2_$3_count
      type: COUNTER
    - pattern: kafka.(\w+)<type=(.+), name=(.+)><>(\d+)thPercentile
      name: kafka_$1_$2_$3
      type: GAUGE
      labels:
        quantile: "0.$4"

Environment (please complete the following information):

• Strimzi version: helm install strimzi strimzi/strimzi-kafka-operator --version 0.28.0
• Installation method: Helm
• Kubernetes cluster: Client Version: version.Info{Major:"1", Minor:"24+", GitVersion:"v1.24.10-dispatcher-dirty", GitCommit:"c4dc593360fd87cf0fe27b27e36a4a19b62d90e9", GitTreeState:"dirty", BuildDate:"2023-01-24T19:42:00Z", GoVersion:"go1.19.5", Compiler:"gc", Platform:"darwin/arm64"}
  Kustomize Version: v4.5.4
  Server Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.9-gke.3200", GitCommit:"92ea556d4e7418d0e7b5db1ee576a73f8fc47e91", GitTreeState:"clean", BuildDate:"2023-01-20T09:29:29Z", GoVersion:"go1.18.9b7", Compiler:"gc", Platform:"linux/amd64"}
• Infrastructure: GKE

Metrics Being exposed

# HELP vertx_pool_ratio Pool usage ratio, only present if maximum pool size could be determined
# TYPE vertx_pool_ratio gauge
vertx_pool_ratio{pool_type="worker",} 0.0
# HELP vertx_http_server_active_connections Number of opened connections to the server
# TYPE vertx_http_server_active_connections gauge
vertx_http_server_active_connections 2.0
# HELP jvm_buffer_total_capacity_bytes An estimate of the total capacity of the buffers in this pool
# TYPE jvm_buffer_total_capacity_bytes gauge
jvm_buffer_total_capacity_bytes{id="mapped",} 0.0
jvm_buffer_total_capacity_bytes{id="direct",} 1.6785443E7
# HELP vertx_http_server_bytes_written_total Number of bytes sent by the server
# TYPE vertx_http_server_bytes_written_total counter
vertx_http_server_bytes_written_total 14947.0
# HELP jvm_classes_loaded_classes The number of classes that are currently loaded in the Java virtual machine
# TYPE jvm_classes_loaded_classes gauge
jvm_classes_loaded_classes 9946.0
# HELP jvm_gc_memory_allocated_bytes_total Incremented for an increase in the size of the young generation memory pool after one GC to before the next
# TYPE jvm_gc_memory_allocated_bytes_total counter
jvm_gc_memory_allocated_bytes_total 6.92043928E8
# HELP strimzi_reconciliations_failed_total Number of reconciliations done by the operator for individual resources which failed
# TYPE strimzi_reconciliations_failed_total counter
strimzi_reconciliations_failed_total{kind="Kafka",namespace="platform-stage",selector="",} 9.0
# HELP vertx_http_server_response_time_seconds_max Request processing time
# TYPE vertx_http_server_response_time_seconds_max gauge
vertx_http_server_response_time_seconds_max{code="200",method="GET",route="",} 1.49709E-4
# HELP vertx_http_server_response_time_seconds Request processing time
# TYPE vertx_http_server_response_time_seconds summary
vertx_http_server_response_time_seconds_count{code="200",method="GET",route="",} 231.0
vertx_http_server_response_time_seconds_sum{code="200",method="GET",route="",} 0.061019742
# HELP jvm_memory_committed_bytes The amount of memory in bytes that is committed for the Java virtual machine to use
# TYPE jvm_memory_committed_bytes gauge
jvm_memory_committed_bytes{area="heap",id="Tenured Gen",} 5.4038528E7
jvm_memory_committed_bytes{area="nonheap",id="CodeHeap 'profiled nmethods'",} 1.900544E7
jvm_memory_committed_bytes{area="heap",id="Eden Space",} 1.5204352E7
jvm_memory_committed_bytes{area="nonheap",id="Metaspace",} 5.7450496E7
jvm_memory_committed_bytes{area="nonheap",id="CodeHeap 'non-nmethods'",} 2555904.0
jvm_memory_committed_bytes{area="heap",id="Survivor Space",} 1835008.0
jvm_memory_committed_bytes{area="nonheap",id="Compressed Class Space",} 7725056.0
jvm_memory_committed_bytes{area="nonheap",id="CodeHeap 'non-profiled nmethods'",} 5570560.0
# HELP vertx_pool_queue_time_seconds_max Time spent in queue before being processed
# TYPE vertx_pool_queue_time_seconds_max gauge
vertx_pool_queue_time_seconds_max{pool_type="worker",} 0.026851589
# HELP vertx_pool_queue_time_seconds Time spent in queue before being processed
# TYPE vertx_pool_queue_time_seconds summary
vertx_pool_queue_time_seconds_count{pool_type="worker",} 3647.0
vertx_pool_queue_time_seconds_sum{pool_type="worker",} 8.396185467
# HELP vertx_http_server_request_resets_total Number of request resets
# TYPE vertx_http_server_request_resets_total counter
vertx_http_server_request_resets_total{method="GET",} 2.0
# HELP strimzi_resource_state Current state of the resource: 1 ready, 0 fail
# TYPE strimzi_resource_state gauge
strimzi_resource_state{kind="Kafka",name="stage-kafka",reason="Exceeded timeout of 300000ms while waiting for Service resource stage-kafka-kafka-external-bootstrap in namespace platform-stage to be addressable",resource_namespace="platform-stage",} 0.0
# HELP jvm_memory_max_bytes The maximum amount of memory in bytes that can be used for memory management
# TYPE jvm_memory_max_bytes gauge
jvm_memory_max_bytes{area="heap",id="Tenured Gen",} 8.8080384E7
jvm_memory_max_bytes{area="nonheap",id="CodeHeap 'profiled nmethods'",} 1.22912768E8
jvm_memory_max_bytes{area="heap",id="Eden Space",} 3.5258368E7
jvm_memory_max_bytes{area="nonheap",id="Metaspace",} -1.0
jvm_memory_max_bytes{area="nonheap",id="CodeHeap 'non-nmethods'",} 5828608.0
jvm_memory_max_bytes{area="heap",id="Survivor Space",} 4390912.0
jvm_memory_max_bytes{area="nonheap",id="Compressed Class Space",} 1.073741824E9
jvm_memory_max_bytes{area="nonheap",id="CodeHeap 'non-profiled nmethods'",} 1.22916864E8
# HELP jvm_gc_live_data_size_bytes Size of old generation memory pool after a full GC
# TYPE jvm_gc_live_data_size_bytes gauge
jvm_gc_live_data_size_bytes 2.2352232E7
# HELP vertx_pool_completed_total Number of elements done with the resource
# TYPE vertx_pool_completed_total counter
vertx_pool_completed_total{pool_type="worker",} 3647.0
# HELP system_cpu_usage The "recent cpu usage" for the whole system
# TYPE system_cpu_usage gauge
system_cpu_usage 0.012146155605095542
# HELP vertx_pool_in_use Number of resources used
# TYPE vertx_pool_in_use gauge
vertx_pool_in_use{pool_type="worker",} 0.0
# HELP strimzi_reconciliations_periodical_total Number of periodical reconciliations done by the operator
# TYPE strimzi_reconciliations_periodical_total counter
strimzi_reconciliations_periodical_total{kind="KafkaMirrorMaker2",namespace="platform-stage",selector="",} 28.0
strimzi_reconciliations_periodical_total{kind="KafkaRebalance",namespace="platform-stage",selector="",} 28.0
strimzi_reconciliations_periodical_total{kind="Kafka",namespace="platform-stage",selector="",} 28.0
strimzi_reconciliations_periodical_total{kind="KafkaMirrorMaker",namespace="platform-stage",selector="",} 28.0
strimzi_reconciliations_periodical_total{kind="KafkaBridge",namespace="platform-stage",selector="",} 28.0
strimzi_reconciliations_periodical_total{kind="KafkaConnect",namespace="platform-stage",selector="",} 28.0
# HELP jvm_classes_unloaded_classes_total The total number of classes unloaded since the Java virtual machine has started execution
# TYPE jvm_classes_unloaded_classes_total counter
jvm_classes_unloaded_classes_total 0.0
# HELP vertx_pool_queue_pending Number of pending elements in queue
# TYPE vertx_pool_queue_pending gauge
vertx_pool_queue_pending{pool_type="worker",} 0.0
# HELP strimzi_reconciliations_duration_seconds The time the reconciliation takes to complete
# TYPE strimzi_reconciliations_duration_seconds histogram
strimzi_reconciliations_duration_seconds_bucket{kind="Kafka",namespace="platform-stage",selector="",le="1.0",} 0.0
strimzi_reconciliations_duration_seconds_bucket{kind="Kafka",namespace="platform-stage",selector="",le="5.0",} 0.0
strimzi_reconciliations_duration_seconds_bucket{kind="Kafka",namespace="platform-stage",selector="",le="10.0",} 0.0
strimzi_reconciliations_duration_seconds_bucket{kind="Kafka",namespace="platform-stage",selector="",le="30.0",} 0.0
strimzi_reconciliations_duration_seconds_bucket{kind="Kafka",namespace="platform-stage",selector="",le="60.0",} 0.0
strimzi_reconciliations_duration_seconds_bucket{kind="Kafka",namespace="platform-stage",selector="",le="120.0",} 0.0
strimzi_reconciliations_duration_seconds_bucket{kind="Kafka",namespace="platform-stage",selector="",le="300.0",} 0.0
strimzi_reconciliations_duration_seconds_bucket{kind="Kafka",namespace="platform-stage",selector="",le="+Inf",} 9.0
strimzi_reconciliations_duration_seconds_count{kind="Kafka",namespace="platform-stage",selector="",} 9.0
strimzi_reconciliations_duration_seconds_sum{kind="Kafka",namespace="platform-stage",selector="",} 2720.209161349
# HELP strimzi_reconciliations_duration_seconds_max The time the reconciliation takes to complete
# TYPE strimzi_reconciliations_duration_seconds_max gauge
strimzi_reconciliations_duration_seconds_max{kind="Kafka",namespace="platform-stage",selector="",} 0.0
# HELP jvm_buffer_count_buffers An estimate of the number of buffers in the pool
# TYPE jvm_buffer_count_buffers gauge
jvm_buffer_count_buffers{id="mapped",} 0.0
jvm_buffer_count_buffers{id="direct",} 7.0
# HELP jvm_memory_used_bytes The amount of used memory
# TYPE jvm_memory_used_bytes gauge
jvm_memory_used_bytes{area="heap",id="Tenured Gen",} 4.2380264E7
jvm_memory_used_bytes{area="nonheap",id="CodeHeap 'profiled nmethods'",} 1.896384E7
jvm_memory_used_bytes{area="heap",id="Eden Space",} 6148120.0
jvm_memory_used_bytes{area="nonheap",id="Metaspace",} 5.3987936E7
jvm_memory_used_bytes{area="nonheap",id="CodeHeap 'non-nmethods'",} 1430144.0
jvm_memory_used_bytes{area="heap",id="Survivor Space",} 789928.0
jvm_memory_used_bytes{area="nonheap",id="Compressed Class Space",} 6448768.0
jvm_memory_used_bytes{area="nonheap",id="CodeHeap 'non-profiled nmethods'",} 5544192.0
# HELP jvm_threads_live_threads The current number of live threads including both daemon and non-daemon threads
# TYPE jvm_threads_live_threads gauge
jvm_threads_live_threads 58.0
# HELP strimzi_resources_paused Number of custom resources the operator sees but does not reconcile due to paused reconciliations
# TYPE strimzi_resources_paused gauge
strimzi_resources_paused{kind="KafkaMirrorMaker2",namespace="platform-stage",selector="",} 0.0
strimzi_resources_paused{kind="KafkaRebalance",namespace="platform-stage",selector="",} 0.0
strimzi_resources_paused{kind="Kafka",namespace="platform-stage",selector="",} 0.0
strimzi_resources_paused{kind="KafkaMirrorMaker",namespace="platform-stage",selector="",} 0.0
strimzi_resources_paused{kind="KafkaBridge",namespace="platform-stage",selector="",} 0.0
strimzi_resources_paused{kind="KafkaConnect",namespace="platform-stage",selector="",} 0.0
# HELP vertx_pool_usage_seconds Time using a resource
# TYPE vertx_pool_usage_seconds summary
vertx_pool_usage_seconds_count{pool_type="worker",} 3647.0
vertx_pool_usage_seconds_sum{pool_type="worker",} 42.809582416
# HELP vertx_pool_usage_seconds_max Time using a resource
# TYPE vertx_pool_usage_seconds_max gauge
vertx_pool_usage_seconds_max{pool_type="worker",} 0.073131263
# HELP jvm_buffer_memory_used_bytes An estimate of the memory that the Java virtual machine is using for this buffer pool
# TYPE jvm_buffer_memory_used_bytes gauge
jvm_buffer_memory_used_bytes{id="mapped",} 0.0
jvm_buffer_memory_used_bytes{id="direct",} 1.6785444E7
# HELP vertx_http_server_requests_total Number of processed requests
# TYPE vertx_http_server_requests_total counter
vertx_http_server_requests_total{code="200",method="GET",route="",} 231.0
# HELP jvm_threads_daemon_threads The current number of live daemon threads
# TYPE jvm_threads_daemon_threads gauge
jvm_threads_daemon_threads 8.0
# HELP vertx_http_server_request_bytes_max Size of requests in bytes
# TYPE vertx_http_server_request_bytes_max gauge
vertx_http_server_request_bytes_max{method="GET",} 0.0
# HELP vertx_http_server_request_bytes Size of requests in bytes
# TYPE vertx_http_server_request_bytes summary
vertx_http_server_request_bytes_count{method="GET",} 234.0
vertx_http_server_request_bytes_sum{method="GET",} 0.0
# HELP jvm_gc_pause_seconds Time spent in GC pause
# TYPE jvm_gc_pause_seconds summary
jvm_gc_pause_seconds_count{action="end of major GC",cause="Metadata GC Threshold",} 2.0
jvm_gc_pause_seconds_sum{action="end of major GC",cause="Metadata GC Threshold",} 0.041
jvm_gc_pause_seconds_count{action="end of major GC",cause="Allocation Failure",} 1.0
jvm_gc_pause_seconds_sum{action="end of major GC",cause="Allocation Failure",} 0.083
jvm_gc_pause_seconds_count{action="end of minor GC",cause="Allocation Failure",} 57.0
jvm_gc_pause_seconds_sum{action="end of minor GC",cause="Allocation Failure",} 0.17
# HELP jvm_gc_pause_seconds_max Time spent in GC pause
# TYPE jvm_gc_pause_seconds_max gauge
jvm_gc_pause_seconds_max{action="end of major GC",cause="Metadata GC Threshold",} 0.0
jvm_gc_pause_seconds_max{action="end of major GC",cause="Allocation Failure",} 0.0
jvm_gc_pause_seconds_max{action="end of minor GC",cause="Allocation Failure",} 0.003
# HELP jvm_threads_peak_threads The peak live thread count since the Java virtual machine started or peak was reset
# TYPE jvm_threads_peak_threads gauge
jvm_threads_peak_threads 63.0
# HELP jvm_gc_memory_promoted_bytes_total Count of positive increases in the size of the old generation memory pool before GC to after GC
# TYPE jvm_gc_memory_promoted_bytes_total counter
jvm_gc_memory_promoted_bytes_total 1.6976016E7
# HELP vertx_http_server_response_bytes_max Size of responses in bytes
# TYPE vertx_http_server_response_bytes_max gauge
vertx_http_server_response_bytes_max{code="200",method="GET",route="",} 0.0
# HELP vertx_http_server_response_bytes Size of responses in bytes
# TYPE vertx_http_server_response_bytes summary
vertx_http_server_response_bytes_count{code="200",method="GET",route="",} 231.0
vertx_http_server_response_bytes_sum{code="200",method="GET",route="",} 14947.0
# HELP strimzi_resources Number of custom resources the operator sees
# TYPE strimzi_resources gauge
strimzi_resources{kind="KafkaMirrorMaker2",namespace="platform-stage",selector="",} 0.0
strimzi_resources{kind="KafkaRebalance",namespace="platform-stage",selector="",} 0.0
strimzi_resources{kind="Kafka",namespace="platform-stage",selector="",} 1.0
strimzi_resources{kind="KafkaMirrorMaker",namespace="platform-stage",selector="",} 0.0
strimzi_resources{kind="KafkaBridge",namespace="platform-stage",selector="",} 0.0
strimzi_resources{kind="KafkaConnect",namespace="platform-stage",selector="",} 0.0
# HELP jvm_gc_max_data_size_bytes Max size of old generation memory pool
# TYPE jvm_gc_max_data_size_bytes gauge
jvm_gc_max_data_size_bytes 8.8080384E7
# HELP jvm_threads_states_threads The current number of threads having NEW state
# TYPE jvm_threads_states_threads gauge
jvm_threads_states_threads{state="runnable",} 14.0
jvm_threads_states_threads{state="blocked",} 0.0
jvm_threads_states_threads{state="waiting",} 31.0
jvm_threads_states_threads{state="timed-waiting",} 13.0
jvm_threads_states_threads{state="new",} 0.0
jvm_threads_states_threads{state="terminated",} 0.0
# HELP strimzi_reconciliations_locked_total Number of reconciliations skipped because another reconciliation for the same resource was still running
# TYPE strimzi_reconciliations_locked_total counter
strimzi_reconciliations_locked_total{kind="Kafka",namespace="platform-stage",selector="",} 19.0
# HELP system_load_average_1m The sum of the number of runnable entities queued to available processors and the number of runnable entities running on the available processors averaged over a period of time
# TYPE system_load_average_1m gauge
system_load_average_1m 0.01
# HELP process_cpu_usage The "recent cpu usage" for the Java Virtual Machine process
# TYPE process_cpu_usage gauge
process_cpu_usage 0.01197452229299363
# HELP vertx_http_server_active_requests Number of requests being processed
# TYPE vertx_http_server_active_requests gauge
vertx_http_server_active_requests{method="GET",} 2.0
# HELP strimzi_reconciliations_total Number of reconciliations done by the operator for individual resources
# TYPE strimzi_reconciliations_total counter
strimzi_reconciliations_total{kind="Kafka",namespace="platform-stage",selector="",} 29.0
# HELP system_cpu_count The number of processors available to the Java virtual machine
# TYPE system_cpu_count gauge
system_cpu_count 1.0

YAML files and logs

Attach or copy paste the custom resources you used to deploy the Kafka cluster and the relevant YAMLs created by the Cluster Operator.
Attach or copy and paste also the relevant logs.

To easily collect all YAMLs and logs, you can use our report script which will automatically collect all files and prepare a ZIP archive which can be easily attached to this issue.
The usage of this script is:
./report.sh --namespace <string> --cluster <string> [--bridge <string>] [--connect <string>] [--mm2 <string>]

Additional context
Add any other context about the problem here.

[ppatierno]

The metrics you are pasted here are coming from the Strimzi operator. They are not Kafka metrics.
If you have a Prometheus instance and follow instruction here https://strimzi.io/docs/operators/latest/deploying.html#assembly-metrics-setup-str in the doc, you should be able to query the Kafka metrics as well.
Just to be clear, don't expect a Prometheus Exporter pod to run. The JMX exporter is running as an agent alongside the Kafka pod in order to get JMX metrics and exposing them in Prometheus format.

[scholzj]

What do you mean with Prometheus Exporter Pod? The Prometheus JMX Exporter runs as an agent in the corresponding pods (e.g. Kafka or ZooKeeper pods). It does not have its own pod.