Trace/span lost during stateful processing in Kafka streams

[paulklos]

Hi,

We have an application that consumes messages from a Kafka topic. It's configured with Kafka bindings using Spring Cloud Stream.

One of our processes uses does an aggregation on a KStream, which results in a KTable, which is then converted back into a KStream and picked up by the next processor.

The whole process consists of 4 processors.

The application is instrumented with opentelemetry-javaagent.jar, the latest version.

Up until the aggregation we get the same trace id in our log messages, and a different span id in each processor, which is very nice, and exactly what we would expect.

But after the aggregation the span and trace ids disappear. Inspection in the debugger of the current span shows only zeros.

Is this a known limitation, or is there something we can configure to have proper propagation?

[mateuszrzeszutek]

Hey @paulklos ,
This looks like an existing limitation -- perhaps we're missing a piece of instrumentation that'd propagate the context across the aggregation.
If it's feasible for you, please create a small app that reproduces your scenario and create an issue -- without a repro scenario it'll be really hard for us to find out what we're missing.

[paulklos]

Hi @mateuszrzeszutek,

I haven't had time for this yet due to holidays, but I'll try to put something together.

[paulklos]

Hi @mateuszrzeszutek,

I created an issue and a small project to illustrate what happens.

[amirhar]

Any updates on this? I'm running into the same issue.

[laurit]

Create an issue, include minimal application that reproduces the problem.

[gvisoc]

Hi @laurit and @mateuszrzeszutek -- I have created this application that reproduces the problem https://github.com/gvisoc/kafka-streams-instrumentation --please comment if any clarification or screenshots are needed (or if any newer versions are already mitigating the problem).

Thanks

[laurit]

@gvisoc could you create an issue to ensure that this doesn't get lost

[gvisoc]

I'll take some time this morning to update the app, take some screenshots and will open the issue.

[gvisoc]

Sorry all, I've seen that the issue is already there in #6569 with better steps to reproduce. I'll monitor that one.