Replace ApplicationInsights dependency with OpenTelemetry libraries

[baronfel]

• Implement Azure Monitor OpenTelemetry exporter integration
• Update Telemetry.cs and related telemetry infrastructure
• Coordinate with .NET team to bring OpenTelemetry libraries into source-build
• Ensure telemetry data format compatibility for existing analytics
• Update TelemetryFilter.cs and TelemetryCommonProperties.cs for new infrastructure

[marcpopMSFT]

Talking offline with baronfel, here's the next steps for this:

• Finish Minimal work to make System.Diagnostics.Activity usable by the dotnet CLI application and codebase #49749 (baronfel)
• Split out just the otel onboarding of events into a separate PR (miyanni)
  • Confirm it's sending and being processed correctly
  • Confirm with A&D team if we can just ifdef these out rather than source build otel
• Onboard basic spans within program .cs (baronfel)
• Explore whether spans are supported by DevDiv data as is or need to be converted to events
• Explore if sampling of spans is needed to minimize the impact to the customer and data volume back to DevDiv data
• Explore additional spans in other parts of the code base
• Investigate impact of sending data post-command and if we need a deferred delivery solution

[marcpopMSFT]

Synced offline on this with @MichaelSimons RE source build. The proposal is to add a new flag dotnetTelemetry that's part of the build that includes the otel components in the build. The source build would not set that flag but the MS build would set it. This would allow any downstream partners who want telemetry enabled to set this and include otel as a prebuilt but allow us to now have to onboard otel and it's dependencies to the official source build (as telemetry is already disabled for that build).

[mmitche]

@baronfel @marcpopMSFT We already have a telemetry opt-in, opt-out based on the "OfficialBuilder" property. This property, when set to Microsoft, enables telemetry. Distro partner builds don't get telemetry.

sdk/Directory.Build.props

Lines 66 to 68 in ab9e137

	<DefineConstants Condition="'$(ContinuousIntegrationBuild)' == 'true'">$(DefineConstants);CI_BUILD</DefineConstants>
	<DefineConstants Condition="'$(OfficialBuilder)' == 'Microsoft'">$(DefineConstants);MICROSOFT_ENABLE_TELEMETRY</DefineConstants>
	<DefineConstants Condition="'$(DotNetBuildSourceOnly)' == 'true'">$(DefineConstants);DOT_NET_BUILD_FROM_SOURCE</DefineConstants>

.

Could that be used?

[marcpopMSFT]

Potentially though OfficialBuilder just sets whether telemetry is on/off whereas this would be if telemetry was included in the files or not. I don't know if we want to use the same property or not. We'll have to think on whether there are scenarios we want telemetry not tied to official builder.

[KalleOlaviNiemitalo]

Is there going to be any interaction between telemetry of the .NET SDK and telemetry of user applications? For example, if a parent process starts dotnet to run a .NET tool and passes it environment variables (OTEL_SERVICE_NAME, TRACEPARENT) to control how the tool measures or exports telemetry, will those environment variables also affect telemetry of the .NET SDK?

I think OTel environment variables should not affect telemetry of the .NET SDK by default. As telemetry of the .NET SDK is normally exported to Microsoft only, a user who sets a trace context or an OTLP endpoint via environment variables presumably wants them to affect telemetry of the application only and does not expect to receive telemetry from the .NET SDK.

However, an opt-in mode in which the user controls the telemetry of the .NET SDK too might be valuable. This could be enabled via a DOTNET_CLI_TELEMETRY_CUSTOM environment variable, with a name similar to the existing DOTNET_CLI_TELEMETRY_OPTOUT. Possible scenarios:

• Export the .NET SDK telemetry to a local endpoint in order to review exactly what would be sent to Microsoft.
• Propagate the trace context from a CI system to .NET SDK, and from .NET SDK to a local NuGet server and to a local container image registry. Export telemetry from all of these to the same analysis service, and view the end-to-end trace.