Span processor cannot extract otel-context key

[thangcircle]

Hello,

I have a java agent extension with a span processor which tries to extract otel-context entry from the context. The problem is: the entry always return null, despite it is showing up when I try to print the context:

{otel-context=com.johndoe.tracer.ContextWrapper@206b1661}

I try to print out other context entries, and they all show up. Only otel-context is the only entry which never gets printed out. Would anyone be able to explain to me whats going on behind the scene ?

I am using AWS OpenTelemetry Agent version 1.6.0

Many thanks !

[thangcircle]

Also a little bit of context to what Im trying to do:

I am injecting some entries into the context from the application, and in the span processor, I pick those entries up and add them to the span attributes.

[mateuszrzeszutek]

Hey @thangcircle ,
Are you using different ContextKey instances in your app and in the SpanProcessor? Context keys are compared by reference (take a look at its javadoc for a small example), so if you're using different instances they are not the same context keys, even though they're named the same.
To solve that, you can either make the ContextKey constant available both to the agent and the app (by putting it in the bootstrap classloader), or use Baggage instead to propagate your custom entries.

[thangcircle]

Hey there!

Thanks for the reply. You are indeed correct about the ContextKey comparison, but it seems like the AgentContextStorage wraps all the context entries within the application code and put it in otel-context ContextKey entry. The ContextKey entry is package protected, which I cannot reference in my span processor to get my inner ContextKey. Would you know anyway to get around this otel-context entry ?

[anuraaga]

Hi @thangcircle - the AgentContextStorage is a bridge between the agent and the app to allow using the normal Context API. The otel-context and similar keys used by our code is intentionally left private since things could go quite wrong if it's modified by a user, it is basically an implementation detail of our instrumentation.

To inject information into the context, you should just be able to use methods on Context without thinking about the contents. For example, this sort of code would inject something to context and read in spanprocessor.

static final ContextKey<String> MY_DATA = ContextKey.named("my-data");

class Controller {
businessLogic() {
  try (Scope ignored = Context.current().with(MY_DATA, "data").makeCurrent()) {
    Span span = tracer.spanBuilder().start().end();
  }
}
}

class SpanProcessor {
  onStart(Context parentContext, ReadWriteSpan span) {
    assert parentContext.get(MY_DATA) == "data";
  }
}

If this isn't working for you, it'd be a bug and a code reproduction would help us see what could be going on.

The context is updated in the app and the spanprocessor is in the javaagent extension, but the context bridge ensures the contents are the same (the otel-context is an implementation detail handling this, but the Context API methods should work fine directly)

[thangcircle]

Hey there !
Thanks for the detailed reply. I tried the above code sample, but it didn't work for me. The debugger shows the following for parentContext when it hops through SpanProcessor#onStart: {otel-context={my-data=data}} . It seems like otel-context is still in the way of fetching the entry from the context.

I tried to circumvent this by using Baggage context. It seems like BaggageContextKey is a separate bridge per these lines of code, which I can reference the entry using Baggage.fromContext(context) and get my data.

[trask]

hi @thangcircle, if you can create a repository with a minimal repro of what you have tried, maybe that would help us to identify the issue (either in the repro, or in the javaagent)