Add support for Filter and Enrich for OpenTelemetry activities

[aradalvand]

Resolves #857

Several important considerations:

• 1 dependency added: OpenTelemetry.Api to NATS.Client.Core — needed for the TracerProviderBuilder.AddNatsInstrumentation extension method. This makes NATS dependent on the so-called OpenTelemetry API, which is expected. But if you're strongly opposed to this, we could also remove this method. Won't take away much from the core functionality added here which are Filter and Enrich. (Dependency removed)
• Since Telemetry.Send* methods were also used in places without dependency injection available (e.g. the NatsMsg struct), I couldn't integrate NatsInstrumentationOptions into the IOptions system (which would've probably been more ideal) — so I created a static Default property instead, which the Action<NatsInstrumentationOptions> input parameter effectively configures, and that property is then used by the Telemetry class.
• I think it would also be useful to add an IsInternal flag to NatsInstrumentationContext (equivalent to the old NATS.Net.Internal activity source name) so that people could easily filter out lower-level communication (e.g. for request-replies), which could be verbose and chatty.

Made a number of tangential improvements too (if you disagree with any of them let me know and I could revert them back):

• Made the methods of the internal Telemetry class public — this is the most conventional pattern in .NET (which this same codebase also follows in almost all other places).
• Added isRemote: true to ActivityContext.TryParse — the OpenTelemetry specification states that span contexts extracted from text maps using propagators (e.g. DistributedContextPropagator) should be isRemote: true — see this link.
• In NatsMsg<T>.Build I moved the code for activity creation before Deserialize is called, so that if the user's custom deserializer spawns an activity (which mine does, for example, so this would be useful for me), that's included in the trace. This makes things symmetric with how RequestAsync and PublishAsync work as well (they encompasses the ser/de step in their activities).

[thompson-tomo]

See couple of comments but I would also suggest waiting for #859 to be merged which would enable the context & options to reside in the abstractions package.

[aradalvand]

See couple of comments but I would also suggest waiting for #859 to be merged which would enable the context & options to reside in the abstractions package.

@thompson-tomo Waiting for what to be merged, sorry? #859 is this PR.

[thompson-tomo]

Ahh my mistake @aradalvand I meant #858

[aradalvand]

I have no idea why the Windows test is failing... Weird.
Update: It seems to be failing without this PR's changes too:

Another update: looks like the main branch is also failing so we're in good company haha:
https://github.com/nats-io/nats.net/commits/main/

[mtmk]

test fail maybe a flapper. just kicked again

[mtmk]

may i move this PR onto release/2.7 branch? it might make it easier to coordinate with other otel stuff for us also we can hopefully release preview quicker

[aradalvand]

Sure @mtmk feel free to do that.

[aradalvand]

@mtmk Any idea about the timeout exception in two of the tests? Weird.

[mtmk]

@mtmk Any idea about the timeout exception in two of the tests? Weird.

Fetch_dispose_test is a flapper and the can not start to connect nats server is because test couldn't setup the nast server which is not related to test itself. i can re run them but need #875 to be merged first.