Opentelemetry and Appdynamics agent causing exceptions in JFR recordings

[ashishsanodia-harness]

Hi Team,

We have multiple java applications which are configured to use opentelemetry java agent, Appd java agent and JFR. JFR recordings for these applications being captured at certain interval.
While analysing JFR recordings we are seeing lots of "java.lang.IllegalAccessError" under exceptions section. These are being thrown because of AppDynamics agent not exporting some packages. Also count of these exception are very high which is resulting in very big JFR recording files.

Exceptions:

1. class io.opentelemetry.javaagent.instrumentation.log4j.appender.v2_17.Log4jHelper (in unnamed module @0x5befbac1) cannot access class org.apache.logging.log4j.Logger (in module com.appdynamics.appagent) because module com.appdynamics.appagent does not export org.apache.logging.log4j to unnamed module @0x5befbac1
2. class io.opentelemetry.javaagent.instrumentation.apachehttpclient.v4_0.ApacheHttpClientRequest (in unnamed module @0x5befbac1) cannot access class org.apache.http.HttpRequest (in module com.appdynamics.appagent) because module com.appdynamics.appagent does not export org.apache.http to unnamed module @0x5befbac1
3. class io.opentelemetry.javaagent.instrumentation.apachehttpclient.v4_0.ApacheHttpClientRequest (in unnamed module @0x6d4f388a) cannot access class org.apache.http.client.methods.HttpUriRequest (in module MultiTenantAgentClassLoader) because module MultiTenantAgentClassLoader does not export org.apache.http.client.methods to unnamed module @0x6d4f388a
4. class io.opentelemetry.javaagent.instrumentation.apachehttpclient.v4_0.ApacheHttpClientRequest (in unnamed module @0x5befbac1) cannot access class org.apache.http.client.methods.HttpUriRequest (in module com.appdynamics.appagent) because module com.appdynamics.appagent does not export org.apache.http.client.methods to unnamed module @0x5befbac1

Steps to reproduce:

1. Download AppServerAgent-1.8-21.11.2.33305 agent in current directory
2. Download & Install JDK Mission Control https://www.oracle.com/java/technologies/jdk-mission-control.html
3. Download attached default.jfc in current directory
4. Download opentelemetry java agent from https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases/latest/download/opentelemetry-javaagent.jar [We are using version 1.19.2]
5. Use any java application (attaching sample spring web app spring-boot-hello.jar)
6. Use below command to run application
   java -Xmx1024m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=./heapdump -Xloggc:mygclogfilename.gc -XX:+UseG1GC -XX:InitiatingHeapOccupancyPercent=40 -XX:MaxGCPauseMillis=1000 -Dfile.encoding=UTF-8 -Dotel.traces.exporter=none -Dotel.metrics.exporter=none -XX:NativeMemoryTracking=summary -XX:ErrorFile=hs_err_pid%p.log -XX:StartFlightRecording=disk=true,name=jfrRecording,maxage=30s,dumponexit=true,filename=jfr_dumponexit.jfr,settings=default.jfc -XX:FlightRecorderOptions=repository=./repo,maxchunksize=2m --add-reads jdk.jfr=ALL-UNNAMED -javaagent:AppServerAgent-1.8-21.11.2.33305/javaagent.jar -Dappdynamics.jvm.shutdown.mark.node.as.historical=true -Dappdynamics.agent.nodeName=spring-web-app-local -javaagent:opentelemetry-javaagent.jar -Dotel.service.name=spring-web-app-local -Dotel.traces.exporter=none -Dotel.metrics.exporter=none -jar spring-boot-hello.jar
7. It will periodically generate JFR recordings in ./repo folder under current directory.
8. Open any JFR file present under repo folder using JDK Mission Control. (Attaching sample JFR file as well)
9. Once JFR is opened in JMC navigate to "Java Application"->"Exception" you will see "java.lang.IllegalAccessError". (Screenshot attached)

I had discussion with AppDynamics team as well they suggested couple of configuration changes which resolved first 3 errors but 4th is still not resolved.
class io.opentelemetry.javaagent.instrumentation.apachehttpclient.v4_0.ApacheHttpClientRequest (in unnamed module @0x5befbac1) cannot access class org.apache.http.client.methods.HttpUriRequest (in module com.appdynamics.appagent) because module com.appdynamics.appagent does not export org.apache.http.client.methods to unnamed module @0x5befbac1

Also AppDynamics team suggested

• disable apache-httpclient instrumentation by setting -Dotel.instrumentation.apache-httpclient.enabled=false But this will disable all our apache-httpclient instrumentation.
• Remove opentelemetry agent altogether. Not an option as we are using it for tracing.
• Create ticket with Opentelemetry as it seem like opentelemetry agent issue.

So here I am. Can you please help with this?

[laurit]

Hopefully fixed with #8065

[ashishsanodia-harness]

Thanks @laurit @mateuszrzeszutek . When this fix will be released?

[mateuszrzeszutek]

In around ~3 weeks.

[laurit]

@ashishsanodia-harness try with the latest snapshot build from https://oss.sonatype.org/content/repositories/snapshots/io/opentelemetry/javaagent/opentelemetry-javaagent/1.25.0-SNAPSHOT/ and see if this fixes the issue you had

[ashishsanodia-harness]

Thank @laurit @mateuszrzeszutek
I tried to reproduce with opentelemetry-javaagent-1.25.0-20230323.123337-31.jar and not seeing this exception anymore.