How to test exception handling logic with chaos enabled

[rynoV]

Hi, thanks in advance for your help

Question

Requirements for context:

1. You need a resilience pipeline to retry when an exception is thrown
2. In addition to the retry, you need exception handling logic that updates a local variable when that exception is thrown
3. You want to inject chaos faults simulating that exception for testing

1 and 2 are easy enough with a try-catch within ResiliencePipeline.ExecuteAsync that updates the variable and re-throws, but 3 is not satisfied because the logic within the catch won't get tested when chaos simulates an exception.

What is the recommended way to set this up such that the local exception handling logic is also tested by the chaos faults?

Ideal

What I was hoping for was an argument to ResiliencePipeline.ExecuteAsync that would let me execute a callback when my code or the chaos sim throws an exception (or maybe more generally when it returns an Outcome), before the pipeline starts handling it.

Options considered

1. Move the ExecuteAsync call within the try-catch: doesn't work because the local variable needs to be updated before the retry is executed
2. Use ResiliencePipelineBuilder to add another strategy (like a Fallback which handles the local variable update) to the existing pipeline: doesn't work because the chaos step would only be included in the existing pipeline, and wouldn't be applied to the new strategy
   • So an existing pipeline couldn't be used, it would have to be constructed from scratch at the point when the local variable is in scope
   • Do-able but not great
3. Use two pipelines, separating out the chaos and retry, put the chaos pipeline in the try-catch and the retry outside of the try-catch
   • This would work, but it's more code and messier
   • Part of the appeal of having "chaos" and "resilience" in the same library is the "chaos" can be injected somewhat transparently, i.e. one doesn't have to sprinkle "chaos injection" points throughout their code which are only there for testing purposes

[martincostello]

Try refactoring your code so that the state you want to update is passed by reference in a ResilienceContext so you can manipulate it from your test code.

If that doesn't work, then we'd need to see a concrete example of what you're trying to do with code we can run.

[rynoV]

My use case is to simulate an exception in businessLogicThatMayThrow which is difficult/impossible to trigger during tests, but is known to occur in production

[martincostello]

It isn't intended to be a mock, I'm just working with as much as I can with your small out-of-context sample to illustrate the use of the resilience context to mutate some shared state.

The main point is that if you want to mutate a local variable, you can't. Instead you need to store it in the context so that your chaos code can access it to change it as whatever it is you need in your real code to drive the test.

I could try to help more if you could provide a code sample that's pretty much representative of your real code to see what is missing to go down the code path you want to be tested. Then I could try converting it to C# (unless you could provide it as C# to begin with) and refactoring it to get the intended behaviour.

[rynoV]

The above sample is pretty much representative of the real code, except it's not using dependency injection to get the pipeline.

Instead you need to store it in the context so that your chaos code can access it to change it as whatever it is you need in your real code to drive the test

This is what I mean by "mock". When the test code executes a different code path from the production code, this is a mock which I would like for Polly to provide a path to avoid. I understand that I could use ResilienceContext and write the customLocalExceptionHandlingLogic such that it could be called from both the production code and the chaos code, but I would prefer to avoid any chance of the two calls to customLocalExceptionHandlingLogic getting out of sync. It would also introduce an unfortunate coupling where my resilience layer needs to know about the implementation details of my other code.

The main point is that if you want to mutate a local variable, you can't

It seems like this is a gap in the API more generally described as: it's not possible to define exception handling logic in-line which is executed in production and tests. Would you agree?

I see it as a gap in the ability of the library to simulate a fault:

Production code execution:
BusinessLogic -> RealException -> CustomErrorHandling -> ResiliencePipelineRetry -> BusinessLogic -> ...

Chaos code execution:
SimulatedException -> ____ -> ResiliencePipelineRetry -> BusinessLogic -> ...

So in the tests, SimulatedException fully replaces BusinessLogic -> RealException -> CustomErrorHandling, but I would like it to only replace BusinessLogic -> RealException so that I can still test what happens when BusinessLogic runs again after CustomErrorHandling.

I think something like an OnError parameter for ResiliencePipeline.Execute would be a good addition to enable writing code that is closer to transparent injection of chaos faults.

[martincostello]

This is what I mean by "mock".

Maybe you're missing my point - I'm not suggesting you have two different code paths. I'm suggesting you refactor your production code path to make it testable.

I would prefer to avoid any chance of the two calls to customLocalExceptionHandlingLogic getting out of sync.

I don't get how this wouldn't affect your existing code if you didn't refactor it. If you were still referencing a local variable, two simultaneous calls could still get out of sync unless you specifically synchronised them.

It seems like this is a gap in the API more generally described as: it's not possible to define exception handling logic in-line which is executed in production and tests. Would you agree?

I wouldn't based on the discussion so far, no - so far it sounds like the problem you have is that your code is not testable for the specific error scenario you have, Polly or not. Typically code using the resilience is intended to be transparent to the configuration of the resilience itself - for scenarios where the callbacks need additional context to make decisions on what to do, that's what ResilienceContext is intended for to support passing such information along.

I think something like an OnError parameter for ResiliencePipeline.Execute would be a good addition to enable writing code that is closer to transparent injection of chaos faults.

This isn't a workable suggestion - resilience pipelines are generic to all of the resilience strategies - we can't have chaos specific overloads. That would also then make your own code know about chaos.

Maybe because I'm not an F# developer I'm finding your code hard to understand with a C# viewpoint so I'm having difficulty trying to trace what you're actually trying to do with your real code and how that isn't testable as you'd like with your configured chaos strategy.

If I'm understanding the use case properly, I would refactor the code.

1. The code that might throw an exception is some code that does something - treat it as a black box, it doesn't matter what it does, other than it might throw an exception of some kind you want to specifically deal with.
2. The resilience strategy is configured somehow (e.g. retries, timeouts etc.). You execute the strategy with some code that causes an error of the intended types and you verify it behaves as expected.

This basically gives you two unit tests, one for your production code, another for your correctly configured resilience.

If you need to put the two together, then because you need the code in your black box to throw an error, then you need some sort of hook outside of Polly to do that. Polly can't see into the code that it is executing to apply resilience (of any kind, not just chaos) to - it is an atomic unit of execution, which is why you can't make the chaos happen in the try in your code as-written.

The API you're asking for is basically asking us to reinvent try-catch as a strategy, where you provide both parts. OnError is effectively what Polly already does - that's the whole something went wrong, so we'll retry/circuit-break/rate-limit part - but we provide the logic of what goes in the finally (or wraps the try).

If you rewrote the initial sample without Polly, you would have this:

while (true)
{
    try
    {
        throw new IOException("Actual IO fault");
    }
    catch (Exception)
    {
        customLocalExceptionHandlingLogic(e);
    }
}

It would just retry forever, and there's no where to inject a fault into the body of the while loop - that's the fundamental issue here. Even if we added OnError, there's no way to have it invoke inside your code (i.e. the body of the while loop) - it could only ever be called at the level of the scope of the loop, which would still mean your custom catch code wouldn't be executed. It needs refactoring to be testable in the way you desire - there's no API we can add to make arbitrary code run inside the code you're executing with the strategy.

[rynoV]

Thanks for taking the time to discuss

Typically code using the resilience is intended to be transparent to the configuration of the resilience itself

It's true my code was relying on the global configuration of the resilience (to perform retries) and it didn't really make sense to have them separated like that. I still wanted to use the retry logic from polly instead of a loop, so I've refactored to build the pipeline within the scope of the variable that needs to be changed, and now the OnRetry option for the retry strategy does what I need.