kafka v-20 metrics

[saloni-2404]

We tried the following method:
https://github.com/strimzi/strimzi-kafka-operator/blob/master/examples/metrics/kafka-metrics.yaml

We have a prometheus deployment in the cluster, but we are not able to see any metrics from our kafka namespace with above deployment.
Is there anything we are missing?

[scholzj]

You should always use the examples from the tag for the release you use. The examples from the master branch might not work with your release. You can exec to your Kafka pods and see if you can get the metrics on the 9404 port. If you get them there and not in Prometheus, the issue is in your Prometheus configuration. Of you cannot get them on the port 9404 the issue is in how you configured them in Strimzi (which might be for example related to the first point).

[saloni-2404]

So when we individually define services for kafkaexporter, kafka, zookeeper exposing 9404 port, we were able to curl from there.
But we were looking for the metrics to come up in prometheus. Would we require specifying anything other than this as kafka.yaml: https://github.com/strimzi/strimzi-kafka-operator/blob/master/examples/metrics/kafka-metrics.yaml

[scholzj]

Well, you need to configure your Prometheus to scrape them. Kafka just exposes them. In the docs there is some guidance how to deploy and configure Prometheus using the Prometheus operator. But basically it is up to you how you run and manage your Prometheus, but you have to configure it to scrape the metrics from the pods.

[pashafirdous1]

HI Jakub,
is there a reason we disabled the port for monitoring and introduced Prometheus?
we had primarily deployed monitoring on port 9404, and we also wrote a service to enable the port for kafka, zk and exporter
now with prometheus as it is a design change we need to find a way to monitor the 22.1 metrics.

please advice how we can manage with minimal changes

[scholzj]

The monitoring is still the same - it is still on port 9404 and it still uses Prometheus as it did before. The only difference done (already some time ago) was to remove the port from the service and keep it on the Pod only. TBH I didn't do it, so I do not remember the exact motivation.