How to update test scenario status at runtime with Reqnroll+MSTest

[singh-swati28]

Hello,

In my C# Reqnroll + MSTest + Allure.Reqnroll plugin project, test scenario execution is being intentionally skipped based on some flag, for eg: execute test for US and skip using Assert.Inconclusive("This test is intentionally skipped for other countries") if any other country is specified in Feature file test data.

Note: Initially, when I am getting TestResults object --> testResult.status for current Test is set to none

After the scenario is executed, I am updating test status in AfterScenario hook using following code

AllureLifecycle.Instance.UpdateTestCase(testResult =>
{
    foreach (var x in testResult.steps)
    {
        if ((x.status.Equals(Status.failed) || x.status.Equals(Status.broken)) && x.statusDetails != null && !string.IsNullOrWhiteSpace(x.statusDetails.message) && (x.statusDetails.message.Contains("intentionally skipped") || x.statusDetails.message.Contains("ignored tests")))
        {
            x.status = Status.skipped;    
        }
    }
    testResult.status = Status.skipped;
    testResult.statusDetails.message = "intentionally skipped";
});

While debugging, I can see that the testResult.Status is updated to skipped in the Current TestContext .
However, when the execution is over and the -results.json is generated, test 'status' is overwritten and marked as broken, however, Steps.status is updated to Skipped

Can anyone please look into this and suggest the right way to update test 'status' based on exception thrown?

Note: This is working fine with Specflow+Xunit and Specflow +MSTest using AllureLifecycle.Instance.UpdateTestcase.
In Specflow, the status is set to broken initially and I was able to set it to skipped based on status details message but with Reqnroll, initially the status is none and I am not able to update status at runtime

@gasparnagy

[304NotModified]

Isn't this a question for Allure.Reqnroll?

[singh-swati28]

I posted the question for Allure.Reqnroll plugin as well and awaiting response. However, when I debug the code, Allure methods are working and I am able to update all the test data - status, steps.status, and any other field without any issue. Only test status is getting overwritten, (other updated data still persist) after the code exists the AfterScenario hook.
So, its the Test framework runner, that is overwriting the status set by AllureLifecycle.UpdateTestCase method but I may be wrong. Hence, reaching out here to seek some help/suggestion.

[gasparnagy]

@singh-swati28 Have you tried to set an order for your after scenario hook to be delayed? Like [AfterScrnario(Order = int.MaxValue)]?

[singh-swati28]

Yeah, I tried with [AfterScenario(Order = int.MaxValue)] but no luck
Steps.status (which was set to broken when the test intentionally failed) is updated to skipped but overall test status is still set to broken.

[singh-swati28]

Hi, @gasparnagy !

Allure.Reqnroll plugin developer confirmed that this is a bug which will be fixed eventually. They have not given any ETA for this.
(https://github.com/orgs/allure-framework/discussions/2921)

In the meantime, could you please suggest some workarounds or how to override Reqnroll test listener to update test status based on certain exceptions/stacktrace? Something like, override method OnScenarioFailure/OnStepFailure to update status?

Thank you for your help!

Regards,
Swati