Determine usability in Messaging

[stijnmoreels]

Since the repository is rather inactive, the following area's of the codebase become deprecated. If this state continues, they should be removed to make Messaging more lightweight and easier to maintain.

• ❌ Arcus.Security is being used to retrieve connection strings when authenticating message pumps;

  • ➡️ Should be removed in favor of recommending Managed Identity. Connection strings can still be used with the IConfiguration overload.
  • This includes the extensions overloads to register Azure Service bus message pumps as this takes in the ISecretProvider, which is the point-of-contact for the secret store.

• ❌ .NET Standard 2.1 and .NET 6 are still being added for backwards compatibility reasons. This means harder and 'split' package management when using in combination with .NET 8.

  • ➡️ Remove .NET Standard and .NET 6 in next major.
  • ➡️ Remove Newtonsoft.Json in favor of the built-in System.Text.Json.
  • This also includes the TCP health probe. Starting from v3.0, we can remove the .NET 6/.NET Standard 2.1 versions, but until then, we will need to reference the System.Text.Json package directly for those two target frameworks.

• Observability

  • ❌ Azure Functions are by-default supported, but also means that even without using Azure Functions, you pull in the AZ Func packages. The reason for this, is that Azure Functions uses a different telemetry tracking system (and both in-process and isolated uses also a different one each).
    • 👀 Note that for Azure Functions, it is often required in the past to have a string as incoming message; hence the general messaging routing that supported this. If we decide to remove any Azure Functions-related code, we can also remove the general messaging with string.
  • ❌ Arcus.Observability is being used, and with it Serilog for telemetry tracking.
  • ❌ Remove ServiceBusClient extensions, as they are only there to support our own telemetry tracking.

• ➡️ We will migrate to using OpenTelemetry in the future, plus we should stop supporting Azure Functions directly to lower the maintainence cost.

• ❓ Do we take this opportiunity to also remove the old/deprecated Hierachical correlation support?

• As a proposal, it would be good that the 'core' messaging functionality is independent of any telemetry system, and that either Serilog/Arcus.Observability or OpenTelemetry can be plugged in. To do that, we should:

  1. Deprecate the MessageCorrelationInfoEnricher in the 'core' messaging, as this is solely being used for Serilog;
  2. Release a minor version so that the deprecation is shown;
  3. Create a new 'telemetry' interface that acts as a 'tracker' for Azure Service bus requests;
  4. Creates two new packages/libraries, one for Serilog, one for OpenTelemetry, which both implements this new 'tracker' interface.
  5. Once the v3.0 is released, we can now support both OpenTelemetry, as well as being backwards compatible with the original setup.
  6. In a last phase, once OpenTelemetry is supported by-default, we cn remove our own version of OpenTelemetry.

• ❌ EventHubs has it's own 4 libraries to support similar functionlity like Service bus, but it sheldom being used this way and brings with it a big maintainence cost.

  • ➡️ Remove support for EventHubs all together.

• ❌ General message routing and handling: it was previously made to be as an abstraction for other messaging solutions, but it is become rather tedious to introduce pump-specific features this way. And since people mostly will not implement their own messaging solution this way, it is best to deprecate it instead of endlessly maintain it.

  • This includes the general MessageContext, IMessageRouter, and IFallbackMessageHandler with string/MessageContext.
  • The MessageRouter could become a 'template class' which exposes the functionality commonly needed for message routing, without being a router by itself.
  • The IMessageRouter interface will become deprecated, as it only was using the string as message (which is commonly used for Azure Functions, but not for Azure Service bus message pumps);
  • The IFallbackMessageHander with string/MessageContext will become deprecated as it is only using string as message (which is commonly used for Azure Functions, but not for Azure Service bus message pumps).
  • The MessageContext can become a 'template class' which can act as a common way of providing context to message handlers, without being a context by itself (since the general message routing is being removed).

[fgheysels]

Why would we remove .NET Standard ? I think the package is more lightweight by targetting .NET standard rather than targetting a specific version of .NET ?

.edit: Ok, I've read this article: https://learn.microsoft.com/en-us/dotnet/standard/net-standard?tabs=net-standard-1-0#net-5-and-net-standard
I think for Arcus, it makes sense to fully migrate to .NET x (in our case .NET 8), and drop the support for .NET standard. It doesn't make sense to use Arcus in a .NET Framework or .NET Core project anymore. For teams that are using Arcus for Azure development in a .NET Framework or .NET Core solution, I think it's more than time to upgrade :)

I'm also wondering if we should really drop support for EventHubs. As far as I know, we only need the message-pump that is specific for EventHubs to support EventHubs; all other components are just reused ?

On Observability: I would only remove Observability once we have full support for Open Telemetry.

[stijnmoreels]

I'm also wondering if we should really drop support for EventHubs. As far as I know, we only need the message-pump that is specific for EventHubs to support EventHubs; all other components are just reused ?

Yes and no. Because we also support our own telemetry, there is a lof going on behind the scenes. There are also a lot of overloads available to register message handlers. and then there is also Azure Functions-specific stuff.
So, all in all, still big maintainence cost against less and sheldom use.

On Observability: I would only remove Observability once we have full support for Open Telemetry.

OpenTelemetry is currently 'experimental' for Service bus service-to-service telemetry. So, let's wait indeed a bit. We could already remove the older 'Hierarcal', though.

[fgheysels]

The main advantage of Arcus.Messaging is that it abstracts the boilerplate code that is required to retrieve messages from the bus from the developer. Since Azure Function ServiceBus bindings already provide this abstraction (in another way), I agree with removing Azure Functions support from Arcus.Messaging. Moreover, some functionality -like the messagepump circuitbreaker- is not supported in the Azure-Functions version of Arcus.Messaging I believe.

[stijnmoreels]

To move the observability/service-to-service functionality outside the abstractions, some considerations need to be discussed.

The concrete implementations (current with Serilog, new with OpenTelemetry) will need to have access to the following information:

• FullyQualifiedNamespace ➡️ currently accessible via ServiceBusReceiver;
• EntityName ➡️ currently accessible via ServiceBusReceiver;
• TransactionId/OperationParentId ➡️ currently accessibly via AzureServiceBusMessageContext (which takes it from ServiceBusReceivedMessage);
• IsSuccessful ➡️ from the MessageRoutingResult returned by the message router;
• Duration ➡️ stopwatch around the message routing.

The following proposal should be able to handle both concrete implementations:

• AzureServiceBusMessageContext:
  • ➕ Add FullyQualifiedNamespace
  • ➕ Add EntityName
• ➕ Add IsSuccessful get/set property to MessageCorrelationResult;

Both Serilog and OpenTelemetry will need to create an implementation of an interface that takes in the above described information and returns a MessageCorrelationResult. That way, the message pump (👀 notice that it is not the message router) can initialize the instance and dispose it when the message routing is finished. Completely seperating what kind of observability system was used in the background.

[stijnmoreels]

@fgheysels , happy to discuss this separately.

[stijnmoreels]

Q: What to do with the IMessagePumpLifetime and IRestartableMessagePump interfaces/implementtions? Is this also not providing enough added-value/out-of-scope and can be deprecated?

Q: Should we move the circuit breaker functionlity solely to the Azure Service bus message pump, as not all message pumps could handle circuit breakers anyway (like EventHubs).

[stijnmoreels]

The IMessagePumpLifetime has been deprecated.