Intended way to interact with openetelemetry-grpc

[Thrillpool]

Hello, I'm using the openetelemetry-grpc instrumentation. I set up interceptors for my clients and servers. All my other traces are manually instrumented, and I avoid using context (by which I mean, the thread local variable) anywhere because I prefer to manually pass things around (mostly because I find it easier to reason about).

I'm having an issue though, the way I've interacted with the interceptor is, when I make my client call, I briefly set the context using my current span, and promptly close the scope. On the server side, in my grpc server code, I call Span.current() in the method that is supposed to handle the request,

it looks basically like so (this is scala but that shouldn't matter much)

  override def methodHandler(
      request: Req,
      responseObserver: StreamObserver[Res]
  ): Unit = {
    val span = Span.current()
    ... other stuff
    }

And my intent is basically to pick up the context that the serverInterceptor sets.

This normally all works super, for several thousand requests. At some point though, it just stops working, and the span starts coming through as Span.getInvalid. In my traces, the traces terminate at the traces inside the server interceptor. I see the SENT and RECEIVED events, https://github.com/open-telemetry/opentelemetry-java-instrumentation/blob/main/instrumentation/grpc-1.6/library/src/main/java/io/opentelemetry/instrumentation/grpc/v1_6/TracingServerInterceptor.java#L129. Once it stops working, it stops working everywhere and doesn't recover until I restart the application.

All this makes me think, what actually is the intended way to interact with opentelemetry-grpc? I just sort of made this approach up I'm using because it seemed logical, but maybe I'm doing something fundamentally wrong.

[laurit]

There is some sample code for setting up the grpc library instrumentation in https://github.com/open-telemetry/opentelemetry-java-instrumentation/blob/main/instrumentation/grpc-1.6/library/README.md You can also examine the tests on how they use the library https://github.com/open-telemetry/opentelemetry-java-instrumentation/blob/main/instrumentation/grpc-1.6/library/src/test/java/io/opentelemetry/instrumentation/grpc/v1_6/GrpcTest.java
If you believe you have found a bug, which from your description sounds plausible, create a minimal application that reproduces the problem and create an issue with it.

[Thrillpool]

That readme only concerns how to setup the interceptors (and does a great job of telling you how to, actually), which I'm confident I've done correctly. I suppose that there's no documentation on how to interact with the span is probably a bit telling that maybe one's not supposed to?

That test isn't helpful, for similar reasons. The place I want to obtain the span would be this line
https://github.com/open-telemetry/opentelemetry-java-instrumentation/blob/main/instrumentation/grpc-1.6/library/src/test/java/io/opentelemetry/instrumentation/grpc/v1_6/GrpcTest.java#L81
It is a test of the span that the server interceptor injects.

I do sort of feel I might have found a bug, but even if I have, it's a bug in an undocumented usecase - and also I can't reliably reproduce it so I don't think I can raise such a bug.

If there is no intended way to interact with the span I think I might just replace the instrumentation with my own code and put the spans into the grpc requests so that they can be explicitly handled.

[laurit]

Using Span.current() should work and it is the intended way to interact with the containing span. To confirm whether this is a bug and to fix this we need a way to reproduce it. In grpc instrumentation we use grpc context for context propagation, our thread local context is populated when grpc context is activated. If SERVER span is created, but you can't observe it in the handler then probably something went wrong with the grpc context propagation. It is possible that this could be worked around by using context.makeCurrent() in TracingServerInterceptor.

[Thrillpool]

That's helpful, thank you. I will try to reproduce the issue, but it's not especially clear to me how to do so, so this might take a while.