Enabling agent breaks nested spans

[lassemaatta]

Hi, I'm implementing opentelemetry tracing for a clojure web application. I've built the tracing using the various Java classes (io.opentelemetry.[api, context, sdk, ..].*) and everything seems to be working fine. When accessing services through HTTP endpoints, I can see the resulting traces in e.g. Jaeger UI and spans are nested nicely. I use a Tracer to get a SpanBuilder, create a Span and finally make it the current span.

However, if I include the opentelemetry-java-instrumentation agent, I notice a couple of changes. First, the root span is created by the agent from within the Java HTTP server I use the moment a HTTP request arrives, which is nice. Also, I now get spans from my DB driver.

Great.

Except, now all my spans are direct descendants of this root span and are no longer nested correctly. In other words, if I previously saw in Jaeger:

<span "handle request">
  -> <span "step 1">
       -> <span "nested step 1a">
       -> <span "nested step 1b">
  -> <span "step 2">

Instead, now I get:

<span "HTTP server>
-> <span "handle request">
-> <span "step 1">
-> <span "nested step 1a">
-> <span "nested step 1b">
-> <span "step 2">

Any ideas on how to fix this? I assume this has something to do with context propagation, but I haven't been able to figure what exactly I need to do. I did find some similar discussions related to Kotlin, but I'm doing just basic Java interop here.

Not sure if relevant, but when activating a new Span by calling span.makeCurrent() it would previously return a io.opentelemetry.context.ThreadLocalContextStorage$ScopeImpl class. But when the agent is running, the same call returns a io.opentelemetry.context.ThreadLocalContextStorage$NoopScope/INSTANCE object.

[laurit]

Please provide a minimal sample application that reproduces the issue along with any necessary instructions.

[lassemaatta]

Here you go: https://github.com/lassemaatta/otel-example

README contains instructions for starting the project if you want to run it locally.

https://github.com/lassemaatta/otel-example/blob/master/src/otel_example/core.clj contains the actual implementation; basic scaffolding to setup otel, some helper functions for creating spans and a simple HTTP server.

It uses the 2.13.3 version of the BOM + agent, but I also checked the latest version.

[lassemaatta]

and just to clarify, the issue I'm facing

when running without the agent: outer span is the parent of the inner span

when running with the agent: outer and inner spans are siblings and both are children of the jetty span.

Things get worse if we add even more spans

vs

[lassemaatta]

I think I figured out the problem; I guess the Java instrumentation agent creates its own OpenTelemetry instance and installs it using GlobalOpenTelemetry/set. I was creating my own separate instance for internal telemetry and assumed the agent would use it.

[laurit]

Great that you figured it out. When using agent you can get access to the sdk configured by the agent with GlobalOpenTelemetry.get().