Multiple Kafka message keys as span attributes?

[Ocramius]

Heyo!

I'm working on a relatively generic messaging system in Go, and am currently writing/integrating instrumentation for Kafka consumers.

I noticed that semconv declares various attributes for connecting messages with each other:

• opentelemetry-go/semconv/v1.17.0/trace.go

  Line 2461 in 6192b14

  MessagingMessageIDKey = attribute.Key("messaging.message.id")

• opentelemetry-go/semconv/v1.23.1/attribute_group.go

  Line 1556 in 6192b14

  MessagingKafkaMessageOffsetKey = attribute.Key("messaging.kafka.message.offset")

What I'm struggling with is writing code that does (at logical level, pseudo-code) following batching:

batcher := someBatcher(func(messages []Message, session KafkaSession) {
  handleBatch(messages) // I want to create a span around this

  for _, message := range messages {
    session.MarkProcessed(message)
  }
})

messageConsumer := makeConsumer(consumerGroupOptions, batcher); // this retrieves messages one-by-one, passes them to `batcher`

In my setup, this means that the start of a span is not a single message, but a larger slice of messages.

How would I approach such correlation, if I wanted to make it visible in span attributes?

For now, I used this very brutal approach, which is of little use for the span collector services:

			semconv.MessagingKafkaMessageKey(strings.Join(function.Pipe1(
				messages,
				array.Map(func(message *sarama.ConsumerMessage) string {
					return string(message.Key)
				}),
			), ","),

[dmathieu]

Hi,

Have you looked at the messaging semantic conventions otep?
The semantic conventions repository (which is the source of truth for semconv. The code here is auto-generated from that data) also contains docs for kafka that may provide more insight.
https://github.com/open-telemetry/semantic-conventions/blob/main/docs/messaging/kafka.md

[Ocramius]

Yep, I checked those docs.

RocketMQ has multi-key support, as it seems: https://github.com/open-telemetry/semantic-conventions/blob/cd0cbf6e3b3b677742f891154d3702bf1e6a27c9/docs/messaging/rocketmq.md - messaging.rocketmq.message.keys

For other messaging systems, I don't see anything pertinent to multiple keys, other than messaging.batch.message_count ( https://github.com/open-telemetry/semantic-conventions/blob/cd0cbf6e3b3b677742f891154d3702bf1e6a27c9/docs/attributes-registry/messaging.md#general-messaging-attributes )

[dmathieu]

If you think semantic conventions are missing an issue on the semantic-conventions repository would be the right place to ask.

[Ocramius]

Raised: open-telemetry/semantic-conventions#1937

Thanks @dmathieu!

[pellared]

Consider looking at https://github.com/dnwe/otelsarama which an instrumentation library for https://github.com/IBM/sarama.

[Ocramius]

Yep, checked that, and it's doing per-message, doesn't have anything per-batch: https://github.com/dnwe/otelsarama/blob/9388d9d40bc0a3a2e41e0df5c969250c035645a2/dispatcher.go#L58-L83