Enhancement of our testing strategy

[deeprobin]

Enhancement proposal in our testing strategy: Mutation Testing

I've been thinking hard on the topic of testing lately. And I guess we are all interested in increasing the quality of the runtime and the BCL so we can eliminate bugs in the bud.
Currently, our primary goal is to maximize test coverage. But has any thought ever been given to test quality?

That is one reason why the concept of mutation testing exists.
Mutation testing mutates the program so that certain sections of code in the code are changed and the test case is applied to the changed code. If it fails, it is called killed. If it passes it is a survivor. And that is bad.
From these two values we calculate the mutation score. Similar to the test coverage, it is necessary to achieve the highest possible value.

MutationScore = \frac{KilledMutants}{SurvivedMutants} * 100%

Goals ¹

The goals of mutation testing are multiple:

• identify weakly tested pieces of code (those for which mutants are not killed)
• identify weak tests (those that never kill mutants)
• compute the mutation score, the mutation score is the number of mutants killed / total number of mutants.
• learn about error propagation and state infection in the program

Application in runtime

In my opinion, this is a very good concept that we should apply in the runtime and base class libraries as well as extensions.
I have to say that a rate of 100% would be too good to be true (similar to coverage) but a steady improvement is a good start in my opinion.

We could, in my opinion, start to expand the BCL with these tests in small steps.

One idea would be to create a manually triggerable CI job for this purpose.

Why manually?
The CI takes a long time and the mutation testing will take much longer than a normal test run because it will test multiple mutations.

Approach to use this strategy in .NET

There is actually a tool called Stryker ² which is currently becoming more and more popular that can be installed via the dotnet CLI.
I'm not sure to what extent we can use it in our project, as our project structure is quite different from other projects (Usage of Shared projects, directory structure and many other parameters which could influence this).

Approach to use this strategy in C++

Such a process would also be useful for the CoreCLR and Mono Runtime. There is a C++ library for this purpose.

Useful references

Here is a small list of links that may help you to understand the topic a little better.

• Wikipedia Article: Mutation testing
• Guru99 - Mutation Testing in Software Testing: Mutant Score & Analysis Example
• PIT Mutation Testing - JVM Library for Mutation Testing
• Introduction to Mutation Testing

I would very much like to hear about your opinion on this subject 😄

Footnotes

1. Source - 02/21/2022 17:42 CEST ↩

2. GitHub Repository - Documentation ↩

[stephentoub]

Fuzzing is a form of mutation testing and is something we actively do. I don't see us investing much beyond fuzzing in other forms. You're of course welcome to explore on your own and demonstrate its value.