Agent class loader is not loading custom instrumentation classes

[srinusanduri]

I want to make a minor change to grpc server and client auto instrumentation rules. I want to prefix server_ for grpc server span names and prefix client_ for grpc client span names. I tried to achieve this by overriding only io.opentelemetry.instrumentation.grpc.v1_6.GrpcSpanNameExtractor in my custom agent which being built with distro approach.

I have overriden io.opentelemetry.instrumentation.grpc.v1_6.GrpcTelemetryBuilder in my distro project (kept overridden class in the same package io.opentelemetry.instrumentation.grpc.v1_6) and passed custom SpanNameExtracters for server and client instrumentations. I have created 2 new span name exstracters similar to io.opentelemetry.instrumentation.grpc.v1_6.GrpcSpanNameExtractor, they are io.opentelemetry.instrumentation.grpc.v1_6.GrpcServerSpanNameExtractor and io.opentelemetry.instrumentation.grpc.v1_6.GrpcClientSpanNameExtractor.

I built the agent, extracted the contents of agent and verified that the new classes are shaded and placed in correct location besides GrpcTelemetryBuilder at path inst/io/opentelemetry/javaagent/shaded/instrumentation/grpc/v1_6 in agent jar. Screenshot of contents attached below.

But agent class loader is not able to load io.opentelemetry.instrumentation.grpc.v1_6.GrpcServerSpanNameExtractor class. Its throwing java.lang.NoClassDefFoundError exception.

Caused by: java.lang.ExceptionInInitializerError: Exception java.lang.NoClassDefFoundError: io/opentelemetry/javaagent/shaded/instrumentation/grpc/v1_6/GrpcServerSpanNameExtractor [in thread "main"]
	at io.opentelemetry.javaagent.shaded.instrumentation.grpc.v1_6.GrpcTelemetryBuilder.build(GrpcTelemetryBuilder.java:117)
	at io.opentelemetry.javaagent.instrumentation.grpc.v1_6.GrpcSingletons.<clinit>(GrpcSingletons.java:33)

However, if I override io.opentelemetry.instrumentation.grpc.v1_6.GrpcSpanNameExtractor (overriding already existing class), overrides are effective but new classes are not being loaded.

Grpc instrumentation code: https://github.com/open-telemetry/opentelemetry-java-instrumentation/tree/main/instrumentation/grpc-1.6/library/src/main/java/io/opentelemetry/instrumentation/grpc/v1_6

Am I missing anything here? Could you help understand why this is not working? Please suggest. Thank you.

[laurit]

At runtime these classes are injected into application class loader so it is not agent class loader but application class loader that fails loading your class. We use a custom gradle plugin named muzzle that among other things adds https://github.com/open-telemetry/opentelemetry-java-instrumentation/blob/main/muzzle/src/main/java/io/opentelemetry/javaagent/tooling/muzzle/InstrumentationModuleMuzzle.java to InstrumentationModule classes and also implement the methods from that interface. One of these methods returns a list of classes that need to be defined in application class loader for this instrumentation. If you add a new class to the instrumentation that is not in this list it can't be loaded. Try building the javaagent instrumentation with your changes and replace all modified class files (if you do it correctly then you should see that GrpcInstrumentationModule bytes have changed compared to what is inside the agent after your modifications).

[srinusanduri]

@laurit Thank you for the response and detailed explanation.
Are you suggesting to override GrpcInstrumentationModule class and by overriding getAdditionalHelperClassNames() method to include my extra helper classes?

I tried overriding GrpcInstrumentationModule but build is failing at byteBuddyJava task. Build error is

> Task :instrumentation:grpc:byteBuddyJava FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':instrumentation:grpc:byteBuddyJava'.
> Failed to transform class files in /Users/sreenivasulusanduri/Projects/Work/au-opentelemetry-java/instrumentation/grpc/build/classes/java/mainraw

Existence of GrpcInstrumentationModule class is causing this issue. Build succeeds if I remove this class.

Below is the screenshot of package structure I have to override contents of classes.

Sorry if I misunderstood your suggestion. I missing something here. Could you help resolve this?

[laurit]

I'd start by running gradle with --stacktrace and try to figure out why it fails. I'm not aware of anybody trying to extend and instrumentation they way you are doing it so it is hard to predict what kind of issues you'll run into. If you can't get it working this way you could try copying all the classes in grpc instrumentation (including the library instrumentation) move them to a different package, do the modifications you need and disable the original grpc instrumentation.

[srinusanduri]

@laurit, thank you for taking time to look into this. I tried multiple ways to resolve this without luck. May be I will have to rewrite entire instrumentation as you suggested. I will try.