Problem propagate kafka header telemetry

[jkljajic]

HI.
I trying to implement Opentelemetry in some java project and I am stuck I do not know if I missing something or is just BUG and I would like help who has much more experience.

prodProps.put(ProducerConfig.INTERCEPTOR_CLASSES_CONFIG, `TracingProducerInterceptor.class.getName());`

consProps.put(ConsumerConfig.INTERCEPTOR_CLASSES_CONFIG, TracingConsumerInterceptor.class.getName());

ConsumerRecords<String, DataRecord> records = consumer.poll(100);
for (ConsumerRecord<String, DataRecord> record : records) {
      //in from header => 00-d9b067759918fd20a204b4773550501a-8d6256f0a54a47a7-01 
       System.out.printf("%s %s \n", Span.current().getSpanContext().getTraceId(),Span.current().getSpanContext().getSpanId());
       //System out print => 00000000000000000000000000000000 0000000000000000 

       producer.send(new ProducerRecord<String, DataRecord>(topic, key1, value));
       //out to header => 00-94914ed9ca249880b13f0674644c5376-2ec87bbf67ddb96a-01 
}

some more print out

in => 00-e0370ab7807f0c9f0baa0188578e7821-f83bfdd76207e892-01
00000000000000000000000000000000 0000000000000000
out => 00-cd3183d543438b3d82b78f0a793f9835-735d047592de3452-01
Consumed record with key alice and value {"count":1}, and updated total count to 13769

in => 00-b96211623482f5ed81c014eaa66230e1-dcf9cc3b6cc6b021-01
00000000000000000000000000000000 0000000000000000
out => 00-c1dce4f2162be65633a8cd0908116aa6-74f95a73e4a244e6-01
Consumed record with key alice and value {"count":1}, and updated total count to 13770

in => 00-ad0e223562b6b79b7287462da02b7c65-7fb272c2128d0bb9-01
00000000000000000000000000000000 0000000000000000
out => 00-652b5bdc6b6bd09c0800c8e9c8ba4839-a16fb24db6003d8c-01
Consumed record with key alice and value {"count":1}, and updated total count to 13771

in => 00-3f2cf2f9c4d4cb8fb75c9d01e761eb95-fdbb95608fe45417-01
00000000000000000000000000000000 0000000000000000
out => 00-6331a8bd5c4d4195aafba90588e7413f-2a5bf754857ae297-01
Consumed record with key alice and value {"count":1}, and updated total count to 13772

I expected that in and out header would be the same always just the parent SpanId

I am really confused

any help is welcome

[trask]

hi @jkljajic, what version of the javaagent are you using? If you're not using 1.12.x, try with the latest (see #5385)

[jkljajic]

Hi @trask I try to use the latest 1.12.x but also try to implement the manual approach and same effect no Span.Current().
and yes we also try suppress-messaging-receive-spans to setup to the true same result with JavaAgent

[trask]

ah, I may know what's going on.

the instrumentation is not able to automatically make the incoming span "current" when using poll() and looping through the results.

you will need to extract the traceparent header into a Span, and and make that span current before you call producer.send

ConsumerRecords<String, DataRecord> records = consumer.poll(100);
for (ConsumerRecord<String, DataRecord> record : records) {
      //in from header => 00-d9b067759918fd20a204b4773550501a-8d6256f0a54a47a7-01 
       System.out.printf("%s %s \n", Span.current().getSpanContext().getTraceId(),Span.current().getSpanContext().getSpanId());
       //System out print => 00000000000000000000000000000000 0000000000000000 

       producer.send(new ProducerRecord<String, DataRecord>(topic, key1, value));
       //out to header => 00-94914ed9ca249880b13f0674644c5376-2ec87bbf67ddb96a-01 
}

[jkljajic]

@trask that means we cant use 100% only Javaagent we need to adopt for loop that after getting record extract and makespan current. But is not point Javaagent that you get that out of the box?

[mateuszrzeszutek]

Hey @jkljajic ,
Unfortunately, there are limits to what we can do with the javaagent - the Kafka client API isn't particularly friendly when it comes to instrumentation, and in some cases it is not possible to correctly determine which record is currently being processed.
If it is possible, try using one of the wrappers over Kafka - for example, spring-kafka.

[trask]

@jkljajic I'm not super sure, but this PR may resolve your issue: #5826