Type of a message is lost for ActivityKind: Server and Internal

[prybski]

During incorporation of this project as a NuGet package we have encountered an issue, which results in missing type of a message for ActivityKind: Server and Internal (for Client it works fine and is visible as it should be). I saw in the source code that you are extracting type of a message with use of .GetMessageType() method, but it seems to not always return it:

/// <summary>
/// Gets the message type from the message
/// </summary>
public static string GetMessageType(this TransportMessage message)
{
    if (message == null) throw new ArgumentNullException(nameof(message));
    return message.Headers.GetValueOrNull(Headers.Type)
           ?? "<unknown>";
}

Maybe there is a better way to extract type of a message(?), because right now we have ended up with our custom implementation, which uses Rebus.Events NuGet package and with is the type of a message is always present.

I am including screenshots from our Datadog instance to illustrate what the problem is in a better way:

[zlepper]

I'm sorry, i'm not sure i follow the problem here? I'm not familiar with Datadogs UI, so maybe i'm missing something? Can you describe exactly what you are expecting vs what you are getting? It would be especially nice if you can include the actual headers you have on the messages that are not working as expected.

[prybski]

The issue is basically what I have described @zlepper 😄 For some reason when using NuGet package from this repository Activity with ActivityKind set to Client is working properly. Rest of them miss the message type name. I have included method which should be responsible for that, and that is GetMessageType. Maybe there is a better way of fetching type of message than from headers?

This issue forced us to write our custom Rebus instrumentation, because having message type name on all Activities related to Rebus is crucial to us.

[prybski]

I expect that Activities with any ActivityKind related to Rebus will contain message type name 😉 Right now it only works like that for Activities with ActivityKind set to Client.

[zlepper]

We are using Grafana where i work and it works fine without any modifications there:

What did you add the the Activity in your custom instrumentation?

[prybski]

We do not use DiagnosticSource like you do in this repository. Instead we make use of Rebus.Events 😉 And then we have possibility to create Activities with desired ActivityKind and message type name (it does not get lost at some point when we implement it like that).

[prybski]

Maybe something was off with Rebus configuration on our side. Can you share with me how does it look in you project? 🤔

[zlepper]

The DiagnosticSource is the key to making sure everything actually works. If the DiagnosticSource is not enabled, the Rebus does not populate the activity: https://github.com/rebus-org/Rebus.OpenTelemetry/blob/master/Rebus.Diagnostics/Diagnostics/Incoming/IncomingDiagnosticsStep.cs#L47 Though if you are actually getting the "receive" types and such, then this part probably isn't the problem.

Can you maybe setup a unit test in this repository that reproduces the issue? You can see an example here: https://github.com/rebus-org/Rebus.OpenTelemetry/blob/master/Rebus.Diagnostics.Tests/IntegrationTests.cs#L23

[prybski]

We have enabled DiagnosticSource, so for sure this is not the problem unfortunately 😕 I will try to setup the unit test soon 😉

[prybski]

Can you show me @zlepper how is Rebus configured in your project that works properly with this NuGet package? 🤔

[robertopaixao]

I'm having the same issue in Datadog, but when using Jagger locally, I can see the span name as expected.
So I'm assuming it might be something related to Datadog.

[robertopaixao]

I found a way to fix it.
In my OpenTelemetry processor, I've added the following transformations.
Datadog requires the type to be located in the span attribute messaging.destination.name

transform:
trace_statements:
- context: span
statements:
- set(span.attributes["messaging.destination.name"],span.attributes["rebus.message.type"]) where span.attributes["messaging.operation"] == "receive" and span.attributes["rebus.message.type"] != nil
- set(span.attributes["messaging.destination.name"],span.attributes["rebus.message.type"]) where span.attributes["messaging.operation"] == "process" and span.attributes["rebus.message.type"] != nil
- set(kind.string, "Consumer") where span.attributes["messaging.operation"] == "process"

[robertopaixao]

Additionally, Rebus, instead of having the process spans as type Consumers, as required in OpenTelemetry, has it as internal.
And the receive should be of the type client.

https://opentelemetry.io/docs/specs/semconv/messaging/messaging-spans/#span-kind

[zlepper]

Not to be sarcastic or hostile, I mean this in the friendliest way possible:

The code that sets the different values is in this folder: https://github.com/rebus-org/Rebus.OpenTelemetry/tree/master/Rebus.Diagnostics%2FDiagnostics%2FIncoming

if you submit a pr I'll be happy to take a look and merge it :)

It's not really a problem for us as we use Honeycomb which doesn't care about any of that stuff and only looks for the parent span id to be part of the same trace, so I personally don't really have the motivation to fix it myself :)

[robertopaixao]

I haven't taken it at all as a hostile reply, no worries. I will check the code more carefully and then open a PR if it is really the case.

[zlepper]

If you have any questions about the code, feel free to ask :)