OpenTelemetry Metrics

[fage88]

I know quarkus has an opentelemetry extension, but it only seems to do traces.
Is there any way to also utilize OpenTelemetry metrics and push them to the same OTLP collector as the traces?

I am compiling my quarkus application as a GRAALVM native image and trying to implement observability on it with Elastic. We can't use the elastic apm java agent with GraalVM so we are tying to use the quarkus opentelemetry extension. It is working great for traces but we weren't seeing any metrics.

Then I added the micrometer extension and started pushing micrometer metrics to elastic via OTLP, but the metrics aren't really getting associated with the app well, I am wondering if I can figure out a way to use OpenTelemetry for both the traces and metrics if Elastic will bundle these together better.

-Jeff

[quarkus-bot[bot]]

/cc @ebullient, @jmartisk, @radcortez

[ebullient]

this is in the plan, but combined instrumentation is not there yet.

[vietk]

Hello,

I have an additional question regarding OpenTelemetry metrics.

OpenTelemetry has started issuing semantic convention about metrics used to monitor an application: from HTTP client/server to database driver.

So my question is: Do Quarkus plan to implement such standard bindings?
Because today, for example, even if Quarkus is using Vertx, the HTTP metric binding is implemented in Quarkus and specific to Vertx which is different from a HTTP servlet server, but finally report the same "kind" of information

Kevin

[ebullient]

No. Quarkus does not use Vert.x HTTP metrics.. it uses Vert.x hook points.

The metrics emitted are consistent with what Spring emits (which is fairly close to OTel semantic conventions already). There is an OTLP registry already (present in Quarkiverse) that I believe adapts naming conventions (which is the only significant delta).

MP Metrics is significantly different at the moment.

[vietk]

Hello,
Thanks for your response, I will see the OLTP registry adapts the metrics convention.

Kevin

[yunusevren]

Hi. Is there any update on when Quarkus OpenTelemetry plugin will start supporting metrics export? Is it perhaps already on the roadmap?

The suggested method of using Micrometer + Micrometer OTel exporter is not a feasible alternative. With this method, it's possible to push JVM metrics, however Micrometer's metric naming does not comply with the OpenTelemetry's Semantic Conventions for Runtime Environment Metrics. For example: jvm.memory.used (Micrometer) vs. process.runtime.jvm.memory.usage (OTel standard). This results in significant incompatibility on monitoring systems (New Relic, Datadog etc.) that use OTel metrics to build out of the box JVM health dashboards. Since they cannot find metrics with the expected names, none of the dashboards are rendered properly.

So I am looking forward to adding metrics support for Quarkus OpenTelemetry plugin. If metrics exporter is configured, I assume/hope that an application can easily use OpenTelemetry's runtime metrics library to register JVM observers: https://mvnrepository.com/artifact/io.opentelemetry.instrumentation/opentelemetry-runtime-metrics. Right now this approach does not seem to work:

import jakarta.enterprise.context.ApplicationScoped;
import jakarta.enterprise.event.Observes;
import jakarta.inject.Inject;

import io.opentelemetry.api.OpenTelemetry;
import io.opentelemetry.instrumentation.runtimemetrics.BufferPools;
import io.opentelemetry.instrumentation.runtimemetrics.Classes;
import io.opentelemetry.instrumentation.runtimemetrics.Cpu;
import io.opentelemetry.instrumentation.runtimemetrics.GarbageCollector;
import io.opentelemetry.instrumentation.runtimemetrics.MemoryPools;
import io.opentelemetry.instrumentation.runtimemetrics.Threads;
import io.quarkus.runtime.StartupEvent;

@ApplicationScoped
public class StartupService {

    @Inject
    OpenTelemetry openTelemetry;

    void onStart(@Observes StartupEvent ev) {
        MemoryPools.registerObservers(openTelemetry);
        BufferPools.registerObservers(openTelemetry);
        Classes.registerObservers(openTelemetry);
        Cpu.registerObservers(openTelemetry);
        Threads.registerObservers(openTelemetry);
        GarbageCollector.registerObservers(openTelemetry);
    }

}

Even if the observers are registered, OpenTelemetry is not exporting metrics. Is there maybe an alternative way to achieve this? Looking forward to your response. Thanks in advance!

[ebullient]

We will likely be bridging the micrometer registry to write to the OTLP extension, rather than using OTLP metrics directly. OTLP instrumentation (last I checked) is not compatible w/ build time vs. runtime instrumentation practices.

The Micrometer OTLP registry does not currently apply any naming conventions, though we could. The semantic conventions for metrics have fluctuated quite a bit, has the dust officially settled?

[yunusevren]

It seems to have settled since there are multiple observability vendors that add built in OTel runtime metric support based on the specs outlined in https://opentelemetry.io/docs/specs/semconv/runtime/jvm-metrics.

Is there a Github issue to keep track of adding OTel Metric support to Quarkus Opentelemetry plugin? I would like to follow that. Curious what the plans are, whether it will be picked up in near future and if there is possibility to contribute. Thanks.