Need help troubleshooting Micrometer integration

[chris-pinola-rf]

Hello! I was really excited to see the Micrometer bridge work land in v1.10 🎉#4919

I promptly tested out the new 1.10 javaagent, which is configured to push metrics over OTLP which I know is working because I already get JVM metrics from these apps, on some of my Kotlin apps that are instrumented with Micrometer but I'm still not seeing my Micrometer metrics flowing through the OTel pipeline.

When I turn on debug logs for the javaagent I can see the relevant instrumentation transforming the Micrometer library, and my app is definitely manipulating metrics via Micrometer because the existing Prometheus integration is still working. I also tried setting OTEL_METRICS_EXPORTER to logging but this only logged out one set of JVM platform metrics immediately after the app started up (but before any calls to Micrometer would have been made) and then never logged anything again.

So, I'm wondering, what else can I do to troubleshoot why the javaagent isn't picking up my Micrometer metrics? Any help would be greatly appreciated, thanks!

[trask]

hey @chris-pinola-rf! the agent registers its MeterRegistry into the micrometer global registry, maybe you are using dependency injection of MeterRegistry instead of using the global registry? it would be good to understand if this is the case and what we would need to do to cover more cases like that

[mateuszrzeszutek]

You should be able to list the registries and meters that are contained in Metrics.globalRegistry -- is the OtelMetricsRegistry there? And your meters?

[chris-pinola-rf]

Thank you both so much for the tip. It lead me to realize that we were defining all of our meters against a Prometheus registry instead of the global composite registry. 🤦‍♂️

For example, we were statically defining our Prometheus registry like so:

import io.micrometer.core.instrument.Metrics
import io.micrometer.prometheus.PrometheusMeterRegistry

object InternalMetricsWrapperThing {
    val prometheusRegistry = PrometheusMeterRegistry(...)
    
    init { Metrics.globalRegistry.add(prometheusRegistry) }
}

And then anytime we defined a new meter we were explicitly associating it with our own (non-global) registry:

Counter.builder(...)
    .register(InternalMetricsWrapperThing.prometheusRegistry)
    .increment(...)

After registering our meters against the global registry (instead of just the Prometheus one) I am now seeing exactly what I expected to: Micrometer metrics are flowing through both the OpenTelemetry pipeline and the existing Prometheus registry.

Thanks again, I really appreciate the help! 👍

[codeforaliving]

@mateuszrzeszutek We have a kind of similar issue, however we trying to use the MetricRegistry in our custom SpanProcessor to create our own span metrics in an OTEL extension. The extension is running along side a Spring Boot app that has all the MetricRegistry information. Do you know how we can get access to the same registries within the context of our extension? We are using 1.12.0 of OTEL. Thanks!

[mateuszrzeszutek]

That'd be extremely non-trivial, due to the classloader separation.
Micrometer and its MeterRegistry resides in the application classloader; javaagent extension run in the agent classloader (more or less). The agent classloader does not have direct access to classes in the application classloader, and cannot call them. To make this setup work you'd have to put all Micrometer classes in the bootstrap classloader, which is accessible from both application and agent CLs.
Instead of doing this, I'd recommend using the OTel metrics API and the Micrometer bridge instrumentation.

[codeforaliving]

@mateuszrzeszutek thanks for the response. I figured it wouldn’t be easy. What is the recommended/best way to add custom metrics in our extension using OTEL metrics API and Micrometer bridge instrumentation?