Support of custom ContextStorageProvider?

[gpoulin]

Hi,

We are using Finagle as our primary I/O library. Finagle already have a mechanism for propagating context between thread within a process. It should be pretty trivial for us to write a custom ContextStorageProvider that is backed by this mechanism ensuring that trace context are properly passed within our applications. While it's pretty clear that using custom ContextStorageProvider is supported when using manual instrumentation, it's not clear to us if using custom ContextStorageProvider with the java agent is supported. So I was wondering if it was officially supported? If not, would you expect it to work?

[laurit]

You can customize the agent with an extension, see https://github.com/open-telemetry/opentelemetry-java-instrumentation/tree/main/examples/extension There you could use ContextStorageProvider SPI to provide your own context storage just like with manual instrumentation. Keep in mind that agent code is in a different class loader from your application and does not have access to your application classes. It is clear that if you choose to replace the default context storage with something that uses finagle from your application you'll have to apply some pretty serious magic. I personally would attempt instrumenting finagle to propagate otel context instead of trying to replace the agent context storage.

[gpoulin]

Thanks for the reply. When you say "apply some pretty serious magic", what kind of magic are you thinking about? I was thinking it should be pretty trivial since the interface for com.twitter.util.Local is perfect for implementing ContextStorage. For example, implementing grpc's ContextStorage is trivial and works well for us for interfacing finagle and grpc.

If we were to attempt to instrument finagle, what kind of work would be required? Would we need to patch every method where there's a thread change over which the context needs to be propagated? Or will it be possible to simply patch com.twitter.util.Local.save ref and com.twitter.util.Local.restore ref since those method should be called every time there's a thread change? Those methods are called every time there's a thread change where the context should be propagated. Doing a quick search show that those methods are called often in the code base https://github.com/search?q=org%3Atwitter+Local.save&type=code&p=1 .

[laurit]

Agent and your application are in different class loaders. You can't directly use the finagle classes that are in your application in the ContextStorage that you have inside the agent. You'll need to figure out how to bridge the code that is on the application side with the code that is on the agent side. Typical strategy would be to place some classes in the bootstrap loader, which is visible to both agent and application, and instrument the application classes to use the api in boot loader to communicate with the agent. Like when whatever is responsible for finagle context propagation is created you could call the api in boot loader to pass it to the agent so that your ContextStorage could start using it. Perhaps calling it serious magic was an overstatement, it definitely isn't trivial. Unless you add finagle to boot loader so you could use the same finagle classes from your app and agent, maybe then it would be easy to implement.

For example, implementing grpc's ContextStorage is trivial and works well for us for interfacing finagle and grpc.

That implements grpc context on top of otel context. You are talking about doing it the other way around.

Or will it be possible to simply patch com.twitter.util.Local.save ref and com.twitter.util.Local.restore ref since those method should be called every time there's a thread change?

These should be enough. We have something similar for kotlin coroutines https://github.com/open-telemetry/opentelemetry-java/blob/main/extensions/kotlin/src/main/java/io/opentelemetry/extension/kotlin/KotlinContextElement.java