Add more resiliency features

[joaofbantunes]

Need to think about some additional resiliency features. Some loose ideas:

• When not all messages in a batch are produced, keep retrying the remainder, instead of breaking off the production loop
  • This is particularly important if using pure push approaches, like PostgreSQL with WAL, where we can't mark things partially completed
  • Not relevant for polling providers
• When there are issues with message production, have some backoff to avoid retying too much in a short amount of time
• Retry acknowledging produced messages
• Include some sort of "health check" abstraction, that the user can implement and OutboxKit checks, in order to pause trying to produce when either the database or the messaging infrastructure is having problems
• Handle poison messages?
  • Not a fan of this, given it might mess up ordering guarantees, but may be worth having the option
• Detect when the database is in readonly mode (e.g. during a blue/green deployment) and stop trying to produce, as it won't be able to mark messages as completed

[wellingtonjhn]

You probably already have good solutions for these features in mind, but here are my 2 cents:

• Retry acknowledging produced messages

Consumers need to be idempotent. I think this problem is inherent to the pattern itself, it’s a known problem, but to try to mitigate it may make sense the implementation.

So, thinking in a scenario in which it is not possible to mark the record in the outbox table as processed or delete it, an already produced message will return in the next loop. There may be some in-memory state mechanism of the latest successfully produced messages, so if you can't mark it in the database, you could mark it in memory and avoid processing it in the next loop. At some point, this state in memory may be released, or persisted in bulk when the database is operational again. (probably this is a bad solution, there must be another better way to avoid this problem, but I’m not reaching at the moment).

• Include some sort of "health check" abstraction, that the user can implement and OutboxKit checks, to pause trying to produce when either the database or the messaging infrastructure is having problems

With the result of the "health check", you could have something like a "Circuit Breaker" when your infrastructure (database, broker) has N failures in X time, so you can pause the processing. I think this should not be left to the user to implement, but it could be a core feature of the OutboxKit, or the user could have the option to enable/disable this feature.

• Handle poison messages?

Maybe it makes sense to have a "retry_count" and "failure_reason" in the outbox table. Imagine that you are trying to publish a certain message, and it always fails for some reason, mark that tried X times, until a limit in which it does not try this message, and then raises an alert. With the option to enable/disable in the OutboxKit configuration.

[joaofbantunes]

You probably already have good solutions for these features in mind, but here are my 2 cents:

I have some ideas, not sure if "good solutions" 😅

• Retry acknowledging produced messages

This can go from simple(ish) to very complex, depending on how much one wants to invest in this. Personally, I'm tilting towards simple(ish). Not sure how yet, but my idea is indeed to keep the messages pending ack in memory, then keep retrying to ack them on an interval, not producing any additional messages in the meantime (to ensure order).

• Include some sort of "health check" abstraction...

I think this should not be left to the user to implement

It has to be left to the user to implement. OutboxKit may include some database specific features to aid, but having an extensibility point for unaccounted for possibilities might be useful. Additionally, OutboxKit will not include broker specific code, so if we want to pause trying to produce in the face of broker issues, we'll need to rely on the users to do it.

• Handle poison messages?

Yeah, that's probably the way to go, have something opt-in, for scenarios where order isn't relevant and we don't want all the messages to get blocked due to a problematic one.

[joaofbantunes]

#47 and #45 directly help with the following:

• When there are issues with message production, have some backoff to avoid retying too much in a short amount of time
• Retry acknowledging produced messages

Indirectly, they also help with the following:

• Include some sort of "health check" abstraction, that the user can implement and OutboxKit checks, in order to pause trying to produce when either the database or the messaging infrastructure is having problems
• Detect when the database is in readonly mode (e.g. during a blue/green deployment) and stop trying to produce, as it won't be able to mark messages as completed

While not directly addressing them, the impacts are greatly minimized by including better error handling, identifying the stage at which errors happen to tailor the behavior to it, as well as introducing backoffs.