Netty Class Not Found Exception

[codeforaliving]

We recently ran into an issue with agent/instrumentation v1.14 when initializing an AWS Kinesis client. We are getting the following exception:

java.lang.NoClassDefFoundError: io/opentelemetry/javaagent/instrumentation/netty/common/FutureListenerWrappers

We tried including io.opentelemetry.javaagent.instrumentation:opentelemetry-javaagent-netty-4-common:1.14.0-alpha as a runtime dependency in our Spring Boot app but still see the error. Can someone please help?

Thanks

[laurit]

Full stack trace could help. Instructions to reproduce would also be nice.

[codeforaliving]

@laurit ok, makes sense. We are currently migrating from DataDog to OTEL and can't turn off DataDog just yet. Do you have any suggestions for how to run both?

[codeforaliving]

@laurit regarding "switching order of the agents", are you referring to changing the order in the command line?

[laurit]

@codeforaliving yes, command line. Another way you probably could get rid of this exception is when you add a ignored types configurer like https://github.com/open-telemetry/opentelemetry-java-instrumentation/blob/main/examples/extension/src/main/java/com/example/javaagent/DemoIgnoredTypesConfigurer.java with content

builder.allowClass("org.springframework.boot.loader.LaunchedURLClassLoader");

to your extension. I didn't try this, it is just a guess.

[codeforaliving]

@laurit great, thanks! It looks like that would be a separate SPI? What dependencies should I include with that?

[codeforaliving]

@laurit will that prevent auto-instrumentation supplied by OTEL?

[codeforaliving]

@laurit we found that swapping the order of OTEL and DataDog agents fixed the majority of issues but now another one crept up:

Caused by: java.lang.NoClassDefFoundError: io/opentelemetry/javaagent/instrumentation/apachehttpasyncclient/ApacheHttpAsyncClientInstrumentation$DelegatingRequestProducer
…
…
org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:52)

One thing that I maybe didn’t mention was that all of this happened when we added a custom Sampler to our extension. Why would that make a difference? We literally copied the parent based trace id class and added logic to make the sample rate reloadable. Is there some dependency we can add?

[codeforaliving]

@trask @laurit any recommendations how to avoid NoClassDefFoundErrors while running the DataDog and Otel Java agents at the same time?

I’ve tried:

• changing the order on the command line, which fixed some errors but created different errors (more NoClassDefFoundErrors)
• Tried implementing an IgnoredTypesConfigurer in our extension (not sure if I did it right) and that caused the agent to just choke

I am not sure what else to try.

[laurit]

The easiest way to avoid these issues is not to use datadog and otel agent at the same time. While we don't plan to say that we'll support running these together, we do feel that it would be nice to improve the situation, and if such pull requests are submitted we won't reject them.
I did a quick test and it seems that when you have datadog agent before otel agent there is and error when applying instrumentations during startup (you'll have to enable debug logging with -Dotel.javaagent.debug=true to see it) that could explain the initial NoClassDefFoundError you got. When the order of the agents is reversed, so that otel agent is before datadog agent, I didn't notice any errors in the log. Though as otel agent evolved from the datadog agent it would not be too surprising if datadog agent errored this way. I didn't try to enable their debug logging. If you plan to run the agents together you might wish to check whether there are any errors in the datadog agent logs.
Now for the second NoClassDefFoundErrors, as it comes without stack trace or any additional info I'm not able to tell what could be the cause of it. If you wish for someone to look at this then you'll have to provide a minimal application that reproduces the issue. If this only happens with your extension then the first thing I'd check is the debug output of the agent (add -Dotel.javaagent.debug=true). It is possible that the agents are stepping on each others toes and adding the extension causes something to error.