How to collect information about Ignored (@ignore) scenarios?

[volodymyr-shulha]

I want to collect how many tests are ignored in test run to make report with how many tests are ignored.

[BeforeScenario("ignore")] public void AddIgnoredScenarios() => _slackHandler!.AddResult(scenarioContext.ScenarioInfo.Title, "Ignored");

[BeforeScenario] public void AddIgnoredScenarios2() { if (scenarioContext.ScenarioInfo.Tags.Contains("ignore") || scenarioContext.ScenarioInfo.Tags.Contains("skip")) { _slackHandler!.AddResult(scenarioContext.ScenarioInfo.Title, "Ignored"); } }
both cases doesn't work. It seems that Ignore tag doesn't include scenarios at all. Can somebody suggest how to collect ignored test cases in such case?

[clrudolphi]

When a scenario is marked with the @ignore tag, none of the user Binding code will be invoked. The code generated from the feature file will first check for that tag and if found will skip execution of its steps.

There is an Execution Event published (the ScenarioSkippedEvent) which you could capture by writing a run-time plug-in. Unfortunately, that event won't tell you which scenario it was that was skipped. 😞

Here is the relevant code within the TestExecutionEngine that handles the scenario skipped situation:

        public virtual void OnScenarioSkipped()
        {
            // after discussing the placement of message sending points, this placement causes far less effort than rewriting the whole logic
            _contextManager.ScenarioContext.ScenarioExecutionStatus = ScenarioExecutionStatus.Skipped;

            // in case of skipping a Scenario, the OnScenarioStart() is not called, so publish the event here
            _testThreadExecutionEventPublisher.PublishEvent(new ScenarioStartedEvent(FeatureContext, ScenarioContext));
            _testThreadExecutionEventPublisher.PublishEvent(new ScenarioSkippedEvent());
        }

It would be a pretty simple enhancement to the constructor of the ScenarioSkippedEvent class to capture the FeatureContext and ScenarioContext. Once that is done, then a run-time plugin could listen for that event and write out to a log or other system as you desire.
As always, contributions welcome... 😄

[volodymyr-shulha]

If I add contexts to ScenarioSkippedEven object it should be resolved?
I'm not sure if I didn't affect anything :D

[volodymyr-shulha]

I think if I add context it'll be just added info about the scenario/feature..
Regarding plugin: if I create separate plugin and add its dll to my project via reqnroll.json, I have to support it when reqnroll version is updated, so there is no any other option if I understand correctly.

[SunNeverSets]

Is there any possibility not to skip all user Bindigs or create a rule by any chance?

[clrudolphi]

Sorry, there is nothing out-of-the-box that will provide that, to my knowledge.
Depending upon your requirements, something as simple as a text/regex search over your feature files might provide the info you need.

[volodymyr-shulha]

I've created the following plugin:

public class ScenarioSkippedPlugin : IRuntimePlugin
{
    public void Initialize(RuntimePluginEvents runtimePluginEvents, RuntimePluginParameters runtimePluginParameters, UnitTestProviderConfiguration unitTestProviderConfiguration)
    {
        runtimePluginEvents.CustomizeTestThreadDependencies += (sender, args) =>
        {
            args.ObjectContainer.RegisterTypeAs<ScenarioSkippedEventListener, IExecutionEventListener>();
        };
    }
}

public class ScenarioSkippedEventListener : IExecutionEventListener
{
    private static readonly List<string> _skippedScenarios = new List<string>();

    public void OnEvent(IExecutionEvent executionEvent)
    {
        if (executionEvent is ScenarioSkippedEvent)
            _skippedScenarios.Add("Skipped Scenario");
    }

    public static List<string> GetSkippedScenarios() => _skippedScenarios;

    public static void ClearSkippedScenarios() => _skippedScenarios.Clear();
}`

And I'm getting into Initialize method but not OnEvent(in Listener)
do you see what is the problem from the code above?

Regarding collaboration, I could add featureContext and scenarioContext to ScenarioSkippedEvent so I would add info about scenario but how could I listen or read these data?

Sorry for a lot of questions :(

[SunNeverSets]

NUNit

[clrudolphi]

Sorry, I'm an idiot. I overlooked the fact that the code generator adds Test Framework specific attributes on the method that represents the Scenario. When the @ignore tag is present that includes a framework-specific attribute to tell it to ignore that test.

In the case of NUnit, you'll find the following on your generated code:

        [NUnit.Framework.TestAttribute()]
        [NUnit.Framework.DescriptionAttribute("Add two numbers")]
        [NUnit.Framework.IgnoreAttribute("Ignored scenario")]
        public async System.Threading.Tasks.Task AddTwoNumbers()

So, none of Reqnroll's code get's executed.
Sorry that I misled you down that path.

Given that, however, you should be able to write a quick program that reflects over each test assembly to find methods decorated with the NUnit.Framework.IgnoreAttribute. The method names will match the Scenario name (with whitespace removed).

[SunNeverSets]

don't judge yourself, all fine, I'm so excited that you have time to help me anyway :)
I have never written any of the plugins, good staff to learn xD

I just discussed it with your colleague @olegKoshmeliuk who helped me as well to find it out :D
just wanted to tell about it

[volodymyr-shulha]

@olegKoshmeliuk suggested to use the following approach and it works for me in the best way it could without any additional methods, some hacks but still working easier than it could.

THANK YOU FOR YOUR TIME!!!

[volodymyr-shulha]

using custom tag and throw a native NUnit IgnoredException works perfect