zookeeper monitoring metrics error

[lanzhiwang]

Deploy kafka in operators 0.18 and configure zookeeper metrics as follows：

  zookeeper:
    ...
    metrics:
      lowercaseOutputName: true
      rules:
      - pattern: "org.apache.ZooKeeperService<name0=ReplicatedServer_id(\\d+)><>(\\w+)"
        name: "zookeeper_$2"
        type: GAUGE
      - pattern: "org.apache.ZooKeeperService<name0=ReplicatedServer_id(\\d+), name1=replica.(\\d+)><>(\\w+)"
        name: "zookeeper_$3"
        type: GAUGE
        labels:
          replicaId: "$2"
      - pattern: "org.apache.ZooKeeperService<name0=ReplicatedServer_id(\\d+), name1=replica.(\\d+), name2=(\\w+)><>(Packets.*)"
        name: "zookeeper_$4"
        type: COUNTER
        labels:
          replicaId: "$2"
          memberType: "$3"
      - pattern: "org.apache.ZooKeeperService<name0=ReplicatedServer_id(\\d+), name1=replica.(\\d+), name2=(\\w+)><>(\\w+)"
        name: "zookeeper_$4"
        type: GAUGE
        labels:
          replicaId: "$2"
          memberType: "$3"
      - name: zookeeper_$5
        pattern: org.apache.ZooKeeperService<name0=(.+).name1=(.+).name2=(.+).name3=(.+)><>(PacketsReceived)
        labels:
          memberType: $3
          replica: $2
        type: COUNTER
      - name: zookeeper_$5
        pattern: org.apache.ZooKeeperService<name0=(.+).name1=(.+).name2=(.+).name3=(.+)><>(PacketsSent)
        labels:
          memberType: $3
          replica: $2
        type: COUNTER
      - pattern: "org.apache.ZooKeeperService<name0=ReplicatedServer_id(\\d+), name1=replica.(\\d+), name2=(\\w+), name3=(\\w+)><>(\\w+)"
        name: "zookeeper_$4_$5"
        type: GAUGE
        labels:
          replicaId: "$2"
          memberType: "$3"
      - pattern: "org.apache.ZooKeeperService<name0=StandaloneServer_port(\\d+)><>(\\w+)"
        type: GAUGE
        name: "zookeeper_$2"
      - pattern: "org.apache.ZooKeeperService<name0=StandaloneServer_port(\\d+), name1=InMemoryDataTree><>(\\w+)"
        type: GAUGE
        name: "zookeeper_$2"

The deployed servicemonitors look like this:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  creationTimestamp: "2022-01-12T10:39:59Z"
  generation: 1
  labels:
    prometheus: kube-prometheus
  name: kafka-service-monitor
  namespace: operators
  resourceVersion: "1726888"
  uid: 5e25b2a3-2384-4fa7-b5b2-c6c0d49a3ee9
spec:
  endpoints:
  - honorLabels: true
    interval: 15s
    port: tcp-prometheus
  jobLabel: strimzi.io/cluster
  namespaceSelector:
    any: true
  selector:
    matchExpressions:
    - key: app.kubernetes.io/managed-by
      operator: In
      values:
      - strimzi-cluster-operator
    - key: strimzi.io/kind
      operator: In
      values:
      - Kafka

At this time, there are some very strange monitoring metrics in promethues

The metrics zookeeper_packetsrecevied and zookeeper_packetssend will use the metric name and corresponding value as the new metric name

The rule that outputs these two metrics is

      - pattern: "org.apache.ZooKeeperService<name0=ReplicatedServer_id(\\d+), name1=replica.(\\d+), name2=(\\w+)><>(\\w+)"
        name: "zookeeper_$4"
        type: GAUGE
        labels:
          replicaId: "$2"
          memberType: "$3"

      - name: zookeeper_$5
        pattern: org.apache.ZooKeeperService<name0=(.+).name1=(.+).name2=(.+).name3=(.+)><>(PacketsReceived)
        labels:
          memberType: $3
          replica: $2
        type: COUNTER
      

These two rules are repeated, but there is no direct use of the metrics value. Have you encountered this problem again?