Failed scenarios marked as passed

[dacid44]

I recently migrated a large project from SpecFlow, NUnit, .NET 6.0, and Allure 1 (the old SpecFlow.Allure package) to Reqnroll, .NET 8.0, and Allure 2 (the Allure.Reqnroll package, still using NUnit). However, since the migration, I noticed that some scenarios that throw exceptions now are marked as test passes when they should instead be failures. The stdout output shows that the step failed (the step in question shows an error and all following steps show "skipped due to previous errors", but the test exits as if no exception was thrown, no exception or error trace is printed, and the test is marked as passed, both in the Visual Studio test explorer and in the dotnet test task of an Azure DevOps pipeline. The generated allure report properly marks the test as failed, though.

I suspect it may have something to do with NUnit, as failed assertions properly fail the test, and I know NUnit sometimes treats assertions differently from other exceptions, but this could be a red herring.

This was not happening before the migration, and I'm having trouble writing an MRE because the project is fairly complex. What might be causing this behavior and how can I fix it?

If I can reproduce it in a simpler project I will post it here.

Thanks!

Other installed reqnroll plugins: Reqnroll.Assist.Dynamic, ViaData.Reqnroll.Variants

reqnroll.json:

{
    "$schema": "https://schemas.reqnroll.net/reqnroll-config-latest.json",
    "pluginparameters": {
        "variantkey": "Browser"
    }
}

allureConfig.json:

{
    "$schema": "https://raw.githubusercontent.com/allure-framework/allure-csharp/2.12.1/Allure.Reqnroll/Schemas/allureConfig.schema.json",
    "allure": {
        "title": "Web Test Suite",
        "directory": "../../../allure-results",
        "gherkinPatterns": {
            "stepArguments": {
                "createFromDataTables": true,
                "nameColumn": "a.*",
                "valueColumn": "v.*"
            },
            "grouping": {
                "suites": {
                    "suite": "(BuildTest|ReleaseTest|SmokeTest|StagedTest|MIPS)"
                }
            }
        }
    }
}

[dacid44]

Well, I found the problem, it's the ViaData.Reqnroll.Variants package. I don't know how or why, but adding it to my minimum example causes it to show this behavior. Removing it makes the test fail as expected.

[clrudolphi]

Please do post a minimal reproduction. We can look into what has recently changed that would cause that plug-in to fail.

[dacid44]

Sure, here you go. Thanks for looking into it! It happens on both versions of ViaData.Reqnroll.Variants available on NuGet, I haven't tried with Reqnroll versions other than 2.4.0 though. Just open it in VS and run the test from the test explorer. The test step throws an exception, but the test is still marked as passed. If you remove the ViaData.Reqnroll.Variants plugin, the test fails as expected.

I would open an issue on the repository for that plugin, but it doesn't have issues enabled.
https://github.com/bponneviadata/Reqnroll.Contrib.Variants

ReqnrollMre.zip

[304NotModified]

I would open an issue on the repository for that plugin, but it doesn't have issues enabled.

@bponneviadata is that a mistake? (It the default because it's a fork)

[304NotModified]

@dacid44 there is also another fork, but not sure if it the same

https://github.com/anthivaios/VariantsPlugin

[dacid44]

Good to know. I've actually opted to take the plugin out of the project completely, as I'm no longer doing cross-browser testing. I'll keep that alternative in mind if I start it up again at some point, though.