From d5099dd5fd98354cb8222e78abe86c51bddaf5d4 Mon Sep 17 00:00:00 2001 From: Cosmin Vladutu Date: Fri, 9 May 2025 16:23:25 +0300 Subject: [PATCH 01/11] Update mutation-testing.md Added section on how to interpret Mutation Testing Results --- docs/core/testing/mutation-testing.md | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/docs/core/testing/mutation-testing.md b/docs/core/testing/mutation-testing.md index 14be11aaf2fec..1b3b014b5fb2a 100644 --- a/docs/core/testing/mutation-testing.md +++ b/docs/core/testing/mutation-testing.md @@ -110,6 +110,18 @@ Stryker supports several types of mutations: For additional mutation types, see the [Stryker.NET: Mutations](https://stryker-mutator.io/docs/stryker-net/mutations) documentation. +## Interpreting Mutation Testing Results + +After running Stryker.NET, you’ll receive a report that categorizes mutants as **killed**, **survived**, or **timeout**. Here's how to interpret and act on these results: + +- **Killed Mutants**: These are changes that your tests successfully caught. A high number of killed mutants indicates that your test suite effectively detects logic errors. + +- **Survived Mutants**: These changes were not caught by your tests. Review them to identify gaps in test coverage or assertions that are too weak. Focus on adding targeted unit tests that would fail if the mutant were real. + +- **Timeout Mutants**: These mutations caused your code to hang or exceed the allowed time. This can happen with infinite loops or unoptimized paths. Investigate the code logic or increase the timeout threshold if needed. + +> **Note**: Don't chase a 100% mutation score. Focus instead on high risk or business critical areas where undetected bugs would be most costly. + ## Incremental improvement If, after changing your code, the unit tests pass successfully, then they aren't sufficiently robust, and the mutant survived. From bb1a6e281a1fc054f4078eeed37d546a9201230f Mon Sep 17 00:00:00 2001 From: Cosmin Vladutu Date: Fri, 9 May 2025 23:13:02 +0300 Subject: [PATCH 02/11] Update mutation-testing.md Removed spaces before the list items from interpreting mutation results --- docs/core/testing/mutation-testing.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/core/testing/mutation-testing.md b/docs/core/testing/mutation-testing.md index 1b3b014b5fb2a..f86b542e977e1 100644 --- a/docs/core/testing/mutation-testing.md +++ b/docs/core/testing/mutation-testing.md @@ -114,13 +114,13 @@ For additional mutation types, see the [Stryker.NET: Mutations](https://stryker- After running Stryker.NET, you’ll receive a report that categorizes mutants as **killed**, **survived**, or **timeout**. Here's how to interpret and act on these results: -- **Killed Mutants**: These are changes that your tests successfully caught. A high number of killed mutants indicates that your test suite effectively detects logic errors. +- **Killed Mutants**: These are changes that your tests successfully caught. A high number of killed mutants indicates that your test suite effectively detects logic errors. -- **Survived Mutants**: These changes were not caught by your tests. Review them to identify gaps in test coverage or assertions that are too weak. Focus on adding targeted unit tests that would fail if the mutant were real. +- **Survived Mutants**: These changes were not caught by your tests. Review them to identify gaps in test coverage or assertions that are too weak. Focus on adding targeted unit tests that would fail if the mutant were real. -- **Timeout Mutants**: These mutations caused your code to hang or exceed the allowed time. This can happen with infinite loops or unoptimized paths. Investigate the code logic or increase the timeout threshold if needed. +- **Timeout Mutants**: These mutations caused your code to hang or exceed the allowed time. This can happen with infinite loops or unoptimized paths. Investigate the code logic or increase the timeout threshold if needed. -> **Note**: Don't chase a 100% mutation score. Focus instead on high risk or business critical areas where undetected bugs would be most costly. +> **Note**: Don't chase a 100% mutation score. Focus instead on high risk or business critical areas where undetected bugs would be most costly. ## Incremental improvement From 28300beebfb566ccdc75df0bb8b3db07944b7a6e Mon Sep 17 00:00:00 2001 From: Cosmin Vladutu Date: Sat, 10 May 2025 08:16:50 +0300 Subject: [PATCH 03/11] Added 2 new sections Added some more information about CI/CD integration and custom configuration of the tool --- docs/core/testing/mutation-testing.md | 33 ++++++++++++++++++++++++++- 1 file changed, 32 insertions(+), 1 deletion(-) diff --git a/docs/core/testing/mutation-testing.md b/docs/core/testing/mutation-testing.md index f86b542e977e1..0ddfe8e531242 100644 --- a/docs/core/testing/mutation-testing.md +++ b/docs/core/testing/mutation-testing.md @@ -1,6 +1,6 @@ --- title: Mutation testing -author: sigmade +author: sigmade , cosmminvladutu description: Learn about the Stryker.net tool for mutation testing, to evaluate the quality of your unit tests. ms.date: 03/11/2025 --- @@ -122,6 +122,37 @@ After running Stryker.NET, you’ll receive a report that categorizes mutants as > **Note**: Don't chase a 100% mutation score. Focus instead on high risk or business critical areas where undetected bugs would be most costly. +## CI/CD Pipelines +Mutation testing can be easily integrated into your CI/CD pipelines. For example, you can configure Stryker.NET to run as part of your Azure Pipelines (or GitHub Actions) workflow and enforce quality gates based on thresholds. +``` + "thresholds": { + "high": 85, + "low": 65, + "break": 0 + } +``` +## Costumization +Besides setting thresholds for your pipeline, Stryker.NET offers the possibility of having different configurations for each of your project needs. You can do this customisation of behaviour using the stryker-config.json file. +``` +{ + "ignoreMutations": [ + "ToString", + "Equals", + "GetHashCode" + ], + "ignore-methods": [ + "*Logs" + ] + "mutate": [ + "!**/Migrations/*", + "!**/*.Designer.cs" + ] +} +``` +- **IgnoreMutations**: These are the methods or expressions that you want to ignore because they are noisy or you consider not needed based on your application logic needs. They will show up in your reports as Ignored. +- **Ignore-methods**: You can use this to skip entire methods based on their signatures. Also, those will appear in your reports as Ignored. In our example, we ignore all methods ending in "Logs". +- **Mutate**: Without this option, Styker will try to mutate all the files in your project. With this, you can ignore files or entire folders. In the example above, we ignore everything inside a Migrations folder and all .Designer.cs files (which are usually autogenerated) + ## Incremental improvement If, after changing your code, the unit tests pass successfully, then they aren't sufficiently robust, and the mutant survived. From 4754d76e7807a163de4afca180015f8b84832794 Mon Sep 17 00:00:00 2001 From: Cosmin Vladutu Date: Sat, 10 May 2025 08:58:31 +0300 Subject: [PATCH 04/11] Update mutation-testing.md Removed myself from the authors since I saw it should be only one --- docs/core/testing/mutation-testing.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/core/testing/mutation-testing.md b/docs/core/testing/mutation-testing.md index 0ddfe8e531242..5b558cbeb66e7 100644 --- a/docs/core/testing/mutation-testing.md +++ b/docs/core/testing/mutation-testing.md @@ -1,6 +1,6 @@ --- title: Mutation testing -author: sigmade , cosmminvladutu +author: sigmade description: Learn about the Stryker.net tool for mutation testing, to evaluate the quality of your unit tests. ms.date: 03/11/2025 --- From 29c2a224d9de447fafd869f6095c2ff1d47f86e0 Mon Sep 17 00:00:00 2001 From: Cosmin Vladutu Date: Mon, 2 Jun 2025 10:20:17 +0300 Subject: [PATCH 05/11] Update docs/core/testing/mutation-testing.md Co-authored-by: David Pine --- docs/core/testing/mutation-testing.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/core/testing/mutation-testing.md b/docs/core/testing/mutation-testing.md index 5b558cbeb66e7..613ca10e9ecfd 100644 --- a/docs/core/testing/mutation-testing.md +++ b/docs/core/testing/mutation-testing.md @@ -110,7 +110,7 @@ Stryker supports several types of mutations: For additional mutation types, see the [Stryker.NET: Mutations](https://stryker-mutator.io/docs/stryker-net/mutations) documentation. -## Interpreting Mutation Testing Results +## Interpret mutation testing results After running Stryker.NET, you’ll receive a report that categorizes mutants as **killed**, **survived**, or **timeout**. Here's how to interpret and act on these results: From 22e11cc93e3965ab12b09c8ae579fe0a6f9fb594 Mon Sep 17 00:00:00 2001 From: Cosmin Vladutu Date: Mon, 2 Jun 2025 10:21:02 +0300 Subject: [PATCH 06/11] Update docs/core/testing/mutation-testing.md Co-authored-by: David Pine --- docs/core/testing/mutation-testing.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/core/testing/mutation-testing.md b/docs/core/testing/mutation-testing.md index 613ca10e9ecfd..caec5c60fdd62 100644 --- a/docs/core/testing/mutation-testing.md +++ b/docs/core/testing/mutation-testing.md @@ -112,7 +112,7 @@ For additional mutation types, see the [Stryker.NET: Mutations](https://stryker- ## Interpret mutation testing results -After running Stryker.NET, you’ll receive a report that categorizes mutants as **killed**, **survived**, or **timeout**. Here's how to interpret and act on these results: +After running Stryker.NET, you'll receive a report that categorizes mutants as **killed**, **survived**, or **timeout**. Here's how to interpret and act on these results: - **Killed Mutants**: These are changes that your tests successfully caught. A high number of killed mutants indicates that your test suite effectively detects logic errors. From f1b979b762a2adc332cc086109208db992e09e93 Mon Sep 17 00:00:00 2001 From: Cosmin Vladutu Date: Mon, 2 Jun 2025 10:21:23 +0300 Subject: [PATCH 07/11] Update docs/core/testing/mutation-testing.md Co-authored-by: David Pine --- docs/core/testing/mutation-testing.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/core/testing/mutation-testing.md b/docs/core/testing/mutation-testing.md index caec5c60fdd62..e1f85bc11e68f 100644 --- a/docs/core/testing/mutation-testing.md +++ b/docs/core/testing/mutation-testing.md @@ -114,7 +114,7 @@ For additional mutation types, see the [Stryker.NET: Mutations](https://stryker- After running Stryker.NET, you'll receive a report that categorizes mutants as **killed**, **survived**, or **timeout**. Here's how to interpret and act on these results: -- **Killed Mutants**: These are changes that your tests successfully caught. A high number of killed mutants indicates that your test suite effectively detects logic errors. +- **Killed**: These are changes that your tests successfully caught. A high number of killed mutants indicates that your test suite effectively detects logic errors. - **Survived Mutants**: These changes were not caught by your tests. Review them to identify gaps in test coverage or assertions that are too weak. Focus on adding targeted unit tests that would fail if the mutant were real. From 4fc3f1d5ef31e4c7dea4f5e3e3606e7549547ff1 Mon Sep 17 00:00:00 2001 From: Cosmin Vladutu Date: Mon, 2 Jun 2025 10:21:32 +0300 Subject: [PATCH 08/11] Update docs/core/testing/mutation-testing.md Co-authored-by: David Pine --- docs/core/testing/mutation-testing.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/core/testing/mutation-testing.md b/docs/core/testing/mutation-testing.md index e1f85bc11e68f..8b1a5e167d0f7 100644 --- a/docs/core/testing/mutation-testing.md +++ b/docs/core/testing/mutation-testing.md @@ -116,7 +116,7 @@ After running Stryker.NET, you'll receive a report that categorizes mutants as * - **Killed**: These are changes that your tests successfully caught. A high number of killed mutants indicates that your test suite effectively detects logic errors. -- **Survived Mutants**: These changes were not caught by your tests. Review them to identify gaps in test coverage or assertions that are too weak. Focus on adding targeted unit tests that would fail if the mutant were real. +- **Survived**: These changes weren't caught by your tests. Review them to identify gaps in test coverage or assertions that are too weak. Focus on adding targeted unit tests that would fail if the mutant were real. - **Timeout Mutants**: These mutations caused your code to hang or exceed the allowed time. This can happen with infinite loops or unoptimized paths. Investigate the code logic or increase the timeout threshold if needed. From be74f018aadacd3a5495f737d5b36144c71c1523 Mon Sep 17 00:00:00 2001 From: Cosmin Vladutu Date: Mon, 2 Jun 2025 10:21:47 +0300 Subject: [PATCH 09/11] Update docs/core/testing/mutation-testing.md Co-authored-by: David Pine --- docs/core/testing/mutation-testing.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/core/testing/mutation-testing.md b/docs/core/testing/mutation-testing.md index 8b1a5e167d0f7..5bd67b655ef5c 100644 --- a/docs/core/testing/mutation-testing.md +++ b/docs/core/testing/mutation-testing.md @@ -118,7 +118,7 @@ After running Stryker.NET, you'll receive a report that categorizes mutants as * - **Survived**: These changes weren't caught by your tests. Review them to identify gaps in test coverage or assertions that are too weak. Focus on adding targeted unit tests that would fail if the mutant were real. -- **Timeout Mutants**: These mutations caused your code to hang or exceed the allowed time. This can happen with infinite loops or unoptimized paths. Investigate the code logic or increase the timeout threshold if needed. +- **Timeout**: These mutations caused your code to hang or exceed the allowed time. This can happen with infinite loops or unoptimized paths. Investigate the code logic or increase the timeout threshold if needed. > **Note**: Don't chase a 100% mutation score. Focus instead on high risk or business critical areas where undetected bugs would be most costly. From 5fa49676bc2120f1d8b1fc57046593dd2c6fcb9b Mon Sep 17 00:00:00 2001 From: Cosmin Vladutu Date: Mon, 2 Jun 2025 10:29:42 +0300 Subject: [PATCH 10/11] Apply suggestions from code review Co-authored-by: David Pine --- docs/core/testing/mutation-testing.md | 18 ++++++++++++------ 1 file changed, 12 insertions(+), 6 deletions(-) diff --git a/docs/core/testing/mutation-testing.md b/docs/core/testing/mutation-testing.md index 5bd67b655ef5c..635a4d835e53c 100644 --- a/docs/core/testing/mutation-testing.md +++ b/docs/core/testing/mutation-testing.md @@ -120,10 +120,12 @@ After running Stryker.NET, you'll receive a report that categorizes mutants as * - **Timeout**: These mutations caused your code to hang or exceed the allowed time. This can happen with infinite loops or unoptimized paths. Investigate the code logic or increase the timeout threshold if needed. -> **Note**: Don't chase a 100% mutation score. Focus instead on high risk or business critical areas where undetected bugs would be most costly. +> [!NOTE] +> Don't chase a 100% mutation score. Instead, focus on high risk or business critical areas where undetected bugs would be most costly. -## CI/CD Pipelines -Mutation testing can be easily integrated into your CI/CD pipelines. For example, you can configure Stryker.NET to run as part of your Azure Pipelines (or GitHub Actions) workflow and enforce quality gates based on thresholds. +## Adding mutation testing to your CI/CD workflow + +You can seamlessly integrate mutation testing into your continuous integration and delivery workflows. For instance, Stryker.NET can be configured to run within your Azure Pipelines or GitHub Actions setup, allowing you to enforce quality thresholds as part of your automated testing process. ``` "thresholds": { "high": 85, @@ -131,9 +133,11 @@ Mutation testing can be easily integrated into your CI/CD pipelines. For example "break": 0 } ``` -## Costumization -Besides setting thresholds for your pipeline, Stryker.NET offers the possibility of having different configurations for each of your project needs. You can do this customisation of behaviour using the stryker-config.json file. -``` + +## Customization + +Besides setting thresholds for your pipeline, Stryker.NET offers the possibility of having different configurations for each of your project needs. You can do this customization of behavior using the _stryker-config.json_ file. +```json { "ignoreMutations": [ "ToString", @@ -153,6 +157,8 @@ Besides setting thresholds for your pipeline, Stryker.NET offers the possibility - **Ignore-methods**: You can use this to skip entire methods based on their signatures. Also, those will appear in your reports as Ignored. In our example, we ignore all methods ending in "Logs". - **Mutate**: Without this option, Styker will try to mutate all the files in your project. With this, you can ignore files or entire folders. In the example above, we ignore everything inside a Migrations folder and all .Designer.cs files (which are usually autogenerated) +For more information, see [Stryker: Configuration](https://stryker-mutator.io/docs/stryker-net/configuration/). + ## Incremental improvement If, after changing your code, the unit tests pass successfully, then they aren't sufficiently robust, and the mutant survived. From 5a1f8696dd58f8efb9f871985003262e2e7a6a93 Mon Sep 17 00:00:00 2001 From: Cosmin Vladutu Date: Mon, 2 Jun 2025 10:40:47 +0300 Subject: [PATCH 11/11] Apply suggestions from code review Co-authored-by: David Pine --- docs/core/testing/mutation-testing.md | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/docs/core/testing/mutation-testing.md b/docs/core/testing/mutation-testing.md index 635a4d835e53c..db3da385e2017 100644 --- a/docs/core/testing/mutation-testing.md +++ b/docs/core/testing/mutation-testing.md @@ -139,7 +139,7 @@ You can seamlessly integrate mutation testing into your continuous integration a Besides setting thresholds for your pipeline, Stryker.NET offers the possibility of having different configurations for each of your project needs. You can do this customization of behavior using the _stryker-config.json_ file. ```json { - "ignoreMutations": [ + "ignore-mutations": [ "ToString", "Equals", "GetHashCode" @@ -153,9 +153,10 @@ Besides setting thresholds for your pipeline, Stryker.NET offers the possibility ] } ``` -- **IgnoreMutations**: These are the methods or expressions that you want to ignore because they are noisy or you consider not needed based on your application logic needs. They will show up in your reports as Ignored. -- **Ignore-methods**: You can use this to skip entire methods based on their signatures. Also, those will appear in your reports as Ignored. In our example, we ignore all methods ending in "Logs". -- **Mutate**: Without this option, Styker will try to mutate all the files in your project. With this, you can ignore files or entire folders. In the example above, we ignore everything inside a Migrations folder and all .Designer.cs files (which are usually autogenerated) + +- **ignore-mutations**: These are the methods or expressions that you want to ignore because they are noisy, or you consider not needed based on your application logic needs. They will show up in your reports as `Ignored`. +- **ignore-methods**: You can use this to skip entire methods based on their signatures. Also, those will appear in your reports as Ignored. In our example, we ignore all methods ending in "Logs". +- **mutate**: Without this option, Styker will try to mutate all the files in your project. With this, you can ignore files or entire folders. In the example above, we ignore everything inside a _Migrations_ folder and all _.Designer.cs_ files (which are usually autogenerated) . For more information, see [Stryker: Configuration](https://stryker-mutator.io/docs/stryker-net/configuration/).