AddCustomTokenRequestValidator OpenTelemetry Invalid Parent Span

[mkchmiel]

BFF version

7.0.8

.NET version

8.0.405

Description

When using opentelemetry along side jaeger, we are seeing the parent span get dropped and it doesnt matter whether or not we include all of the Duende instrumentation.
An example of the issues is the following:
"invalid parent span IDs=b1986611a6cbc7f2; skipping clock skew adjustment"

Due to the above issue using code such as
Activity.Current?.AddTag("ApiKey", request.ApiKey);
will not do anything because the parent is lost and the data along with it.

Reproduction steps

Setup a standard Duende Identity server setup where we use AddCustomTokenRequestValidator.

Expected behavior

The parent span is not missing or dropped. When all works well using code like Activity.Current?.AddTag can aid in debugging issues.

Logs

No response

Additional context

No response

[AndersAbel]

Can you please provide some more context? Is it inside your custom token request validator that the parent activity is lost? Is there a parent activity you've seen in our source code that should be there?

[mkchmiel]

Good morning,

Yes it appears as though before hitting the custom token request validator gets hit, the parent span gets dropped by disassociation or so thats my theory. You can do something like this for example by overwriting the parent activity context when starting an activity (StartActivity).

Ill attempt to locate such a scenario in the source code.

[mkchmiel]

Well I went through the source code and didn't identify any code that would cause this sadly... What I would have expected is that the activity.Parent/ParentId/ParentSpanId had been modified to disassociate the span from another one.

[mkchmiel]

The cause was some custom logic to filter or hide some spans, closing this out.