Directly provide ITestDiscoverer and ITestExecutor, instead of a host test framework

[nicojs]

Hi there 🙋‍♂️. Congrats on this fork 🍴🎉

I have been asking myself for a couple of years now why SpecFlow (and now Reqnroll) would use a test framework to 'piggyback' on. I think this makes for an overly complicated programming model:

• You must generate source code
• You must not forget to exclude the generated code from source control
• You must not forget to regenerate the source code whenever the input changes
• Plug into build system
• etc.

It must be possible to run spec files directly. Indeed, this is how the cucumber family of frameworks works as well.

After some searching, I am convinced that this is possible:

• dotnet test scans your project for *.TestAdapter.dll files: link
• There seem to be two interfaces you need to provide an implementation for: ITestDiscoverer and ITestExecutor
• XUnit example implementation

Would you be interested in such a feature?

[ajeckmans]

1. Is this something that can be supported alongside the existing implementation?
2. Would this also make it easier to integrate with libraries such as https://github.com/VerifyTests/Verify and https://github.com/fscheck/FsCheck ? Currently they are heavily tied to both the Specflow library as well as a test framework and the library itself. Ideally one would be able to choose any (sane) combination of versions.

[gasparnagy]

Is this something that can be supported alongside the existing implementation?

Yes, in principle. That would be an alternative of the NUnit/xUnit/MsTest integration. Like "SpecRun" did (see my comment below).

Would this also make it easier to integrate with libraries such as https://github.com/VerifyTests/Verify and https://github.com/fscheck/FsCheck ?

Hm. I don't see the connection yet. Do you have something concrete in your mind? (BTW, here is my old plugin that integrated SpecFlow with FsCheck - did not get a real attention: https://github.com/gasparnagy/SpecFlow.FsCheck)

Currently they are heavily tied to both the Specflow library as well as a test framework and the library itself. Ideally one would be able to choose any (sane) combination of versions.

Could you please explain a bit more in detail, what do you mean by "they"?

[ajeckmans]

Take your project. It is coupled to a specific Specflow version and a specific minimum FsCheck version. If Reqnroll would provide its own test adapter and its own extension points I imagine integrating FsCheck would be a lot simpler. Either through something like a stepbinding-like feature.

For Verify there's an integration as well, but that is even worse, as it currently mandates that you use xunit as the backing test framework for specflow as well. The MsTest integration needs customization of the test classes which is not possible using reqnroll/specflow. If reqnroll would have its own testadapter integration would be a lot simpler.

[gasparnagy]

@ajeckmans I see. Yes, for all needs that need dynamic data set generation, that could make the things simpler. This is an interesting use case, so we should keep it in mind for the #11 idea as well.

[gasparnagy]

@nicojs Welcome here.

You are absolutely right, but the situation is not that easy.

The SpecFlow+ Runner (aka SpecRun) was an implementation that did exactly this (unfortunately closed source), however it turned out, that

1. The ITestDiscoverer and ITestExecutor interfaces are (were) mostly undocumented and they changed even between Visual Studio versions without any notice. So you basically woke up on a average Tuesday and realize that no one can run the tests anymore and there is not even a changelog that would help you to figure out what has changed. But even besides these bloody days, it was hard to have a version that works with all commonly used versions of Visual Studio.
2. Many people uses ReSharper test runner that was not supporting these interfaces.
3. You cannot "debug" the feature files, i.e. cannot make breakpoints or will not see the feature file lines in the stack traces.
4. With this approach you cannot modify the behavior of tests with generator plugins (it's a question, whether this is really needed if we have more options runtime)
5. We would immediately lose some complex features of the test runners, like parallel execution, integration with code coverage (?), test parameters, etc.
6. People seemed to feel more safe with the existing test frameworks (maybe that is just a question of communication tough)

I don't know how the big frameworks, like xunit, etc. have overcome these problems. Maybe they had more capacity or were better informed. Anyhow, it was not easy at all. It felt at the end like completely rewrite xUnit.

An alternative approach I tried is to use the xUnit dynamic test registration framework that basically achieved the same. You can still find that among my GitHub repos and that is definitely a good showcase of how the dynamic execution could be done: https://github.com/gasparnagy/SpecFlow.xUnitAdapter. At the end it turned out that ReSharper does not support that interface and xUnit was lacking some features that are important for acceptance tests (like recording an attachment), so it was not picked up. (Also there were some strange stability issues.)

All these info are from 2018, so things might have changed and we can reconsider this again.

I cross-link this discussion with the Roslyn Source Generator that is basically trying to achieve a similar goal with a different approach. But some of the questions (e.g. the necessity of generator plugins) is the same.

[ajeckmans]

VSTest adapters have been supported in Resharper since 2021.1 as an opt-in setting, so that is not an issue anymore.

[nicojs]

1. The ITestDiscoverer and ITestExecutor interfaces are (were) mostly undocumented [...].

Maybe this improved over the years, since Microsoft is now more open source minded?

3. You cannot "debug" the feature files, i.e. cannot make breakpoints or will not see the feature file lines in the stack traces.

Well... maybe this can be provided with some sort of source-map (or .pdb) magic? Or it might not be needed; stepping into glue-code and having great error messages might be enough.. 🤷‍♀️

5. We would immediately lose some complex features of the test runners, like parallel execution, integration with code coverage (?), test parameters, etc.

This would need to be investigated. It should be possible to recreate these features.

The question remains: would you be interested in this feature? We might be able to create it in-house (either with help from students or by investing open-source time) and contribute it back here. If it is successful, we might be able to remove the integration with all those frameworks 😉.

[Code-Grump]

From everything mentioned here, this sounds like a considerable amount of work to achieve, even if we were to ignore the maintenance overheads of supporting TestDiscoverer and ITestExecutor.

It doesn't feel like a justified effort just to get rid of code-behind files. Could you provide a couple of additional reasons that might make the effort worth it?

[nicojs]

To summarize:

1. No code behind files needed.
2. Direct integration into IDEs that support the ITestDiscoverer family of interfaces. (Dubble-click would open the .feature file directly without the need for additional magic ?)
3. Performance improvement, as we don't need to run code generation at all
4. If this is a big success: no need to support code-behind to other frameworks.

One question I still have is if it is possible to have 2 test frameworks active in 1 test project. I think it should be possible if I understand the architecture correctly.

As mentioned, I'd be happy to take a swing at it and contribute it back here. Or open source it outside first and, if you're happy, maintain it inside the reqnroll org.

[gasparnagy]

@nicojs Here is my view on this after having slept on it.

The key question is not how cool it is (we all agree that it is cool and has a lots of benefits), but whether it can (at least mid term) replace the existing integration with the current test frameworks.

My answer for this question is that I don't think it would be able to fully replace the need for the integration with the current test frameworks, based on the following arguments:

1. Those frameworks have plenty of tricky features that people use. Re-implementing all of them is definitely beyond the scope of this project. (Just MsTest alone has an own team at MS and recently announced new improvements.)
2. Tricentis tried to promote SpecRun in all channels, but the popularity based on the NuGet stats never even get closer to the popularity of the MsTest/NUnit/xUnit integrations, although it was a free alternative with the benefits we also outlined here (it is essentially the same thing that is proposed here). Finally Tricentis even officially deprecated the tool.
3. There are some tools that have dedicated support for the main test frameworks (e.g. Azure DevOps have some features that only work with MsTest; some of the teams I work with have infrastructure built on the NUnit XML reports).
4. Maybe less relevant, but Cucumber Java, that traditionally had an own runner nowadays are used everywhere with the JUnit bundle.

Now if it would not replace MsTest/NUnit/xUnit integration trio, at the end we would not be able to simplify the current solution, but would just get additional maintenance burden. Which is something that we should not risk at the current state of the project.

Of course this does not mean that you cannot take a chance on this. This works well as a Reqnroll plugin (just like SpecRun was a SpecFlow plugin), but I suggest to create it as a separate project (so "outside"), so that Reqnroll does not have to make any commitment for that. We can still list & promote it in our docs of course. I hope you are fine with this.

[nicojs]

Sure, it sounds perfectly reasonable. Thanks for the pointers you gave me in this answer. 🙏