Set custom MeterProvider for custom grpc log exporter to track failed exports.

[lokeshmavale]

Hi Team,
We are using otel java agent version 1.21.0,
In below setup, I have a requirement to setMeterProvider() OtlpGrpcLogRecordExporter. So that I can track log export failures in metrics.
So I tried setting OtlpGrpcLogRecordExporterBuilder.setMeterProvider(GlobalOTEL.getMeterProvider)
But when I try to set Global meter provider, Agent fails with error:
java.lang.IllegalStateException: GlobalOpenTelemetry.set has already been called. GlobalOpenTelemetry.set must be called only once before any calls to GlobalOpenTelemetry.get. If you are using the OpenTelemetrySdk, use OpenTelemetrySdkBuilder.buildAndRegisterGlobal instead. Previous invocation set to cause of this exception.

How can I add meter provider in such case?
(I also tried setting CutsomizedMeterProvider from configureSdkMeterProvider() to class variable,
But its null, as this method doesnt gets called before logProvider()

Any other approach is also welcome.

Setup details:

@Override
    public void customize(AutoConfigurationCustomizer autoConfiguration) {
        autoConfiguration.addMeterProviderCustomizer(this::configureSdkMeterProvider);
        autoConfiguration.addMetricExporterCustomizer(this::configureSdkMetricExporterProvider);
        autoConfiguration.addTracerProviderCustomizer(this::configureSdkTracerProvider);

        //Added custom logger which enriches log records with custom attributes and exports via custom exporter.
        autoConfiguration.addLoggerProviderCustomizer(this::configureSdkLoggerProvider);
        autoConfiguration.addResourceCustomizer(this::configureSdkResourceProvider);
        autoConfiguration.addSpanExporterCustomizer(this::configureSdkSpanExporterProvider);
    }
    
  //  And we use below to wrap custom log record exporter (Internally OtlpProxyLogExporter uses OtlpGrpcLogRecordExporterBuilder)
    
    private SdkLoggerProviderBuilder configureSdkLoggerProvider(SdkLoggerProviderBuilder sdkLoggerProviderBuilder,
            ConfigProperties configProperties) {
        LogRecordExporter exporter = LogRecordExporter.composite(OtlpProxyLogExporter.INSTANCE.getLogRecordExporter(),
                                                                 SystemOutLogRecordExporter.create());
        sdkLoggerProviderBuilder.addLogRecordProcessor(new EnrichLogRecordProcessor(exporter));
        return sdkLoggerProviderBuilder;
    }
    
    // We register and configure custom metric reader for Prometheus Server, which enriches all the outging metrics with custom attributes.
    
    private SdkMeterProviderBuilder configureSdkMeterProvider(SdkMeterProviderBuilder sdkMeterProviderBuilder,
            ConfigProperties configProperties) {
        sdkMeterProviderBuilder.registerMetricReader(new WrappedMetricReader(PrometheusHttpServer.create()));
        return sdkMeterProviderBuilder;
    }

Thanks in Advance!

[mateuszrzeszutek]

Hey @lokeshmavale ,

The agent sets the GlobalOpenTelemetry at the end of the auto-configuration process. When you call GlobalOpenTelemetry.getMeterProvider() during the auto-configuration, the global gets initialized with a noop OpenTelemetry instance, and the agent's attempt to set the global again fails.

Do you actually need to set that? The built-in exporters should use the global by default: https://github.com/open-telemetry/opentelemetry-java/blob/0530e63579fda4f844b2ba09fbdfc2070d926759/exporters/common/src/main/java/io/opentelemetry/exporter/internal/grpc/GrpcExporterBuilder.java#L59

[lokeshmavale]

Thanks for the reply @mateuszrzeszutek ,
Understood,
I have a requirement to enrich log record with custom attributes, to do that, I had to add configureSdkLoggerProvider() and added log record processor which enriches before exporting.

Is there any way to achieve this for logging so that it has meterProvider?
Or Any way to provide MeterProvider in current setup?

[mateuszrzeszutek]

Hmm, if you need to add some metrics to your LoggerProvider decorator, how about you use the same trick that we use internally?
You could define a field of Supplier<MeterProvider> with value GlobalOpenTelemetry::getMeterProvider -- as long as you don't call GlobalOpenTelemetry during the construction process you should be fine.

[lokeshmavale]

Thanks @mateuszrzeszutek , I have tried doing something like this:

And declared supplier as
static Supplier b = ()-> GlobalOpenTelemetry.getMeterProvider();
Used it as:

public static LogRecordExporter createLogRecordExporter() {
final OtlpGrpcLogRecordExporterBuilder builder = OtlpGrpcLogRecordExporter.builder();
builder.setMeterProvider(b.get());
RetryUtil.setRetryPolicyOnDelegate(builder, RetryPolicy.getDefault());
return builder.build();
}

Still the problem persists, Is there a way I am missing on?

[mateuszrzeszutek]

I don't really get why you need to set the meter provider on the delegate builder:

builder.setMeterProvider(b.get());

The OtlpGrpcLogRecordExporter already uses GlobalOpenTelemetry.getMeterProvider() by default, you don't have to provide that yourself.