Test failed during retry is added to the final test case count in the report portal

[laxminidawani]

Hi Team,
I am working on a Playwright C# framework with Reqnroll. We are using the NUnitRetry.ReqnrollPlugin (version 1.2.0) NuGet package to retry test cases in the regression suite. The framework is integrated with GitLab, and the reports are published on the report portal.

However, we have encountered an issue where the tests that fail during a retry are added to the final count, which is incorrect. For example, if the regression suite has a total of 228 test cases and one test case fails, then retries, and passes on the first retry attempt, the report portal shows a total count of test cases to 229 with a fail count of 1. Please see the attached screenshot below. The final results of the pass/fail count in GitLab are correct, though (Total = 228 and fail =0 ).

Here is the reqnroll.json configuration we are using:
{
"$schema": "https://schemas.reqnroll.net/reqnroll-config-latest.json",
"language": {
"binding": "en-AU",
"feature": "en-AU"
},
"trace": {
"coloredOutput": true
},
"ide": {
"editor": {
"gherkinFormat": {
"indentAndSteps": true,
"indentExamples": true,
"indentRuleChildren": true
}
}
},
"NRetrySettings": {
"maxRetries": 3,
"applyGlobally": true
},
"reporting": {
"excludeRetryFails": true,
"logRetryAttempts": true,
"finalStatusOnly": true,
"retryPlugin": "ReqnrollRetryPlugin"
}
}

[laxminidawani]

Hi @chrisbillson , can you please help here?

[clrudolphi]

In case it's not clear, the Reqnroll.Retry plugin marks each test method with an NUnit.Retry attribute. The NUnit runtime controls the actual retry behavior. Further, the NUnit runtime is tracking test executions and retries, and writes the results to the TRX file which is consumed by the GitLab report portal. Reqnroll has no direct ability to affect this or solve this problem.

Perhaps this question should also be asked on the NUnit and NUnit.Retry github discussion boards?

I do think we should leave this question open in the hopes that you get an answer and can share it here with other Reqnroll users who might have the same concern.