How to use same instance of Prometheus default registry in agent as well as in application

[srinusanduri]

Hello,

I am building a custom agent using is distro approach mentioned here https://github.com/open-telemetry/opentelemetry-java-instrumentation/tree/main/examples/distro. This custom agent will be statically attached (using -javaagent argument) to our spring-boot services.

Our requirement is to emit a prometheus metric on a special span start. To achieve this, I have added a custom SpanProcessor MetricEmitSpanProcessor.java and registered it using AutoConfigurationCustomizerProvider. MetricEmitSpanProcessor will emit a prometheus metric on span start if a special condition is met. It does so by registering a Prometheus counter metric to Prometheus default Collector registry io.prometheus.client.CollectorRegistry.defaultRegistry. The spring-boot application also uses the same Prometheus default Collector registry io.prometheus.client.CollectorRegistry.defaultRegistry for registering prometheus metrics. The expectation here is, since both application and agent are using same prometheus registry, agent added metrics will also be available in applications's prometheus scrape endpoint. This is the approach we are using currently in our OpenTracing based custom agent. But this is not working with OpenTelemetry custom agent since application's ClassLoader and agent's ClassLoader are different.

The custom MetricEmitSpanProcessor code is:

public class MetricEmitSpanProcessor implements SpanProcessor { 

public static final io.prometheus.client.CollectorRegistry collectorRegistry = io.prometheus.client.CollectorRegistry.defaultRegistry;

  private io.prometheus.client.Counter specialSpans;

  public MetricEmitSpanProcessor(){
    this.specialSpans = io.prometheus.client.Counter.build().name("special_spans").help("Total special spans").register(collectorRegistry);
  }

  @Override
  public void onStart(Context parentContext, ReadWriteSpan span) {
    if (isSpecialSpan()) {
      this.specialSpans.inc();
    }
  } 
  ....
}

I have also tried loading io.prometheus.client.CollectorRegistry.defaultRegistry in bootstrap module and then make the prometheus default registry loaded by bootstrap classloader accessible to agent module like below.

package com.example.javaagent.bootstrap;
public final class PrometheusDefaultRegistryHolder {

  public static final CollectorRegistry DEFAULT_REGISTRY = CollectorRegistry.defaultRegistry;

  public static CollectorRegistry prometheusRegistry() {
    return DEFAULT_REGISTRY;
  }
}

This didn't work too. I expected this to work since CollectorRegistry is loaded by bootstrap class loader and its child class loader AgentClassLoader should use the class loaded by bootstrap classloader instead of AgentClassLoader class loader loading it again but I am seeing below error in this case

java.lang.LinkageError: loader constraint violation: loader io.opentelemetry.javaagent.bootstrap.AgentClassLoader @79b4d0f (instance of io.opentelemetry.javaagent.bootstrap.AgentClassLoader, child of io.opentelemetry.javaagent.bootstrap.AgentClassLoader$JdkHttpServerClassLoader @3cb5cdba io.opentelemetry.javaagent.bootstrap.AgentClassLoader$JdkHttpServerClassLoader) wants to load class io.prometheus.client.CollectorRegistry. A different class with the same name was previously loaded by 'bootstrap'.

I don't have much experience in dealing with class loaders, I tried multiple ways to achieve this behavior but in vain

How can I achieve this behaviour of same prometheus default registry object io.prometheus.client.CollectorRegistry.defaultRegistry to be used in agent as well as application? Please provide your suggestions. Thank you for your time.

[laurit]

AgentClassLoader uses child first delegation, if a class is present in inst it will be loaded by agent class loader regardless of whether bootstrap loader can also load it. This exception means that you have probably packaged prometheus classes in agent in inst directory (loaded by agent class loader) and also at the root of the jar (loaded by bootstrap loader). Check you agent jar and remove the prometheus classes from inst.

[srinusanduri]

Thank you for the suggestion @laurit. I got it working by adding io.prometheus:simpleclient as compileOnly dependency in agent module and as implementation dependency in bootstrap module.