Support tracing via auto-instrumentation with Kafka broker

[manuelnucci]

Hello!

I've been trying to find a way to enable tracing in our microservice architecture and leverage OTEL for that. So far we haven't been able to generate the correct traces of two apps communicated between each other through a Kafka broker.

After reading this article it seems we could leverage the auto-instrumentation approach in our applications and add a specific environment variable in the consumer application to decode the specific headers that should come with the event consumed.

I was able to find that variable here but the current behaviour is that only the trace from the producer app is generated.

I want to know if this behaviour is currently supported, and if there's some additional configuration that we must set for this to work.

Thanks!

P.S.: I've heard about the Strimzi Kafka Bridge as well as the Micrometer library that could solve our problem, but we're interested in the simplest approach possible.

[mateuszrzeszutek]

Hey @manuelnucci ,

Before diving into any extra configuration/software components: are you instrumenting both ends (producer & consumer)? What kind of Kafka library are you using? E.g. plain kafka-clients, spring-kafka etc.
What version of the javaagent are you running?

[manuelnucci]

Hi @mateuszrzeszutek. Let me provide some details about our current setup:

1. Yes, we have instrumented boths ends so we have the Java agent being injected into each container via the K8s operator and then we have an additional sidecar that handles all the signals export process to the appropriate collector.
2. The library being used to talk with Kafka is spring-cloud-stream-binder-kafka. Does it need to support OTEL in some way? Or the agent should handle the headers injection and everything else on their own?
3. We're running Java agent 1.26.0:

Manifest-Version: 1.0
Implementation-Title: javaagent
Implementation-Version: 1.26.0
Implementation-Vendor: OpenTelemetry
Implementation-URL: https://github.com/open-telemetry/opentelemetry-java
-instrumentation
Main-Class: io.opentelemetry.javaagent.OpenTelemetryAgent
Agent-Class: io.opentelemetry.javaagent.OpenTelemetryAgent
Premain-Class: io.opentelemetry.javaagent.OpenTelemetryAgent
Can-Redefine-Classes: true
Can-Retransform-Classes: true

Let me know if you need anything else!

[mateuszrzeszutek]

2. The library being used to talk with Kafka is spring-cloud-stream-binder-kafka. Does it need to support OTEL in some way? Or the agent should handle the headers injection and everything else on their own?

I see. Are you getting any CONSUMER spans at all?

I think this particular library (the Spring Cloud version of Kafka client) might need an extra instrumentation to make it work across the board, based on my previous experiences with the Spring Cloud utilities. The javaagent probably is not able to stitch the trace correctly as it is now.

[manuelnucci]

No span on the consumer side at all. I'll check what specific configuration this library needs in order to work with OTEL.

[mateuszrzeszutek]

Yeah, this might be a case of a missing instrumentation.

[manuelnucci]

Could this be the root cause of the issue? That's the only promising lead we've been able to find out.

[mateuszrzeszutek]

You're not getting any consumer spans, right? This makes me think that there's a matter of a missing instrumentation on the consumer side, rather than just context propagation issues. Although, perhaps, is there a chance you actually have some consumer spans, but they're completely disconnected from the producer?