Filter/Run/Generate tests only for given tag

[wgala-tomra]

Hey,

I was trying to incorporate reqnroll project into our own code base. We have few applications that are serving as whole product but generally are interconnected.
Also, we have many scenarios already created for whole product. Now, here's a catch, not all scenarios apply for given stack. What's more, not all scenarios are going to be covered by tests right away. Probably quite standard problem that many of us face.

My idea of using reqnroll would be to create test project for each stack. Each stack would have each own tag that would tell: "hey I implemented that scenario, all steps should be available, run that test here".

The problem I face is that the only way to do that is to use 'dotnet test filter' with given tag. We use VisualStudio and Test Explorer to run tests locally. All scenarios, are available there right away, no matter if tagged or not. Or whether it is implemented or not. My preferred outcome would be that only 'tagged properly' scenarios are available there (even if not implemented).

I don't want to put extra effort on team to run that command to check if scenario passes. Also, I don't feel it's robust enough that many tests are skipped and I am not sure if that's good thing, when going through test explorer. We do use 'dotnet test' for running unit tests when we build application, but that's already when changes are merged to main branch, which is too late and not good practice.

I went through documentation, I tried to create plugins to solve it and finally reached repository code where (from what I've seen) it's not possible to do that at this moment.

Could anybody tell me if that's supported, or in plans to support?

From what I've seen best way would be to expose it in generator section of reqnroll.json, so tests are not generated at all if doesn't fit tag criteria. Reason why I say this is that TestExplorer is quite greedy when it comes to test discovery and doesn't have option of filtering (that could be configured from csproj project file level)

Thanks in advance!

[clrudolphi]

This might be possible as a plug-in. Take a look at the External Data plugin as an example. It overrides the default generator by modifying the Gherkin AST (it adds rows to Example Tables using data from external files).

In your situation you might be able to surgically remove scenario nodes from the AST before code generation gets underway.

[wgala-tomra]

@clrudolphi thanks for the reply!

It feels a bit odd to filter out gherkin scenarios and mess with ReqnrollDocument, but indeed it is possible to filter out tests that I don't want to generate. Thanks for the tip!

Maybe it maybe good feature request to put that into system for good? I can imagine other people looking for such functionality too, if they want to cover scenarios with UTs on different layers (for E2E it may not be the case).

I have bit trouble with parsing config though, as there's no way to extend the configuration. I saw that you were driving effort to put that into system, but it's very small issue that I can manage on my end.

Thanks again!

[clrudolphi]

As a short-cut to dealing with config, you may wish to take an approach similar to what I'm doing for a new feature (still under development). I'm adding a new node to the configuration file, but reading it and parsing it directly, outside of the existing Reqnroll config object model.
See this file for example.
Within the reqnroll.json file, I will have a root object called "formatters", which will have a set of string valued properties. I read those in to dictionary.

[wgala-tomra]

Yeah, that will be my approach as well. But thanks for the code snippet!