Example of using Jaeger with Prometheus and Grafana

[zhylinskiy-stryber]

Hello everyone.
My goal is to set up monitoring for my app, make alerts for pretty standard cases, like very long http request, or heavy SQL query.
I have an app written in PHP using Laravel framework. I got to the point where I can send traces to Jaeger with proper information attached, that works fine.
I run my environment using docker-compose. I have Jaeger all-in-one, Prometheus and Grafana. I managed to scrape metrics from Jaeger, using Prometheus. Little off topic: Would be nice if someone pointed me out, why cannot Grafana communicate directly to Jaeger, and instead uses Prometheus.
I copied default Grafana dashboard from repo and it didn't work for 100%, only few graphics work.
I also tried the way of using jb and jsonnet to generate dashboard JSON, but still it doesn't work for 100%

Please,
can anyone provide any useful information on how to:

1. Get Grafana dashboard work.
2. How can I expose traces log information (the one I send from my app) and utilise it using Prometheus and Grafana.

Thank you.

[jpkrohling]

Are you talking about this one here? https://github.com/jaegertracing/jaeger/tree/master/monitoring/jaeger-mixin

What does "didn't work 100% mean"? Did you see error messages?

[zhylinskiy-stryber]

No, Prometheus /metrics does not expose Jaeger's metrics, thanks for pointing out, is there a way to expose them ?

[jpkrohling]

I'm not sure we are talking about the same thing: Prometheus' own metrics won't show Jaeger metrics. Prometheus should be scrapping Jaeger metrics, making them available in the query. If you get Jaeger's /metrics, your goal should be to query them in Prometheus. If that's not happening, you'll have to check your Prometheus scrape jobs under Targets in your Prometheus UI.

[zhylinskiy-stryber]

Ah yes, I got your point. I have this in my prometheus.yml config file

It works fine:

[jpkrohling]

It means that all metrics available to monitor Jaeger are getting scraped by Prometheus. Are you by any chance expecting to see metrics related to your workload instead of internal Jaeger metrics? If so, then see the answer by @jkowall.

[zhylinskiy-stryber]

Alright, thanks.

[jkowall]

Looks like metric aggregation and alerting. @zhylinskiy-stryber can you have a look at this and see if this describes what you want to do? open-telemetry/opentelemetry-collector-contrib#403 (comment)

[zhylinskiy-stryber]

So you suggest using Opentelemtry-collector instead of Jaeger ?