Revise Reqnroll "Actions" concept

[gasparnagy]

The concept of SpecFlow "Actions" were to provide a read-to-use solution for common automation targets, like Selenium.

The codebase has been ported to , and the packages have been released with the name Reqnroll.SpecFlowCompatibility.Actions.* (see https://github.com/orgs/reqnroll/discussions/27). The "SpecFlowCompatibility" prefix has been included, because the code needs to be rewised and maybe changed.

This topic is to discuss the long-term strategy for the Reqnroll "Actions".

I haven't used the SpecFlow actions myself, so I don't have first-hand experience. I generally like the concept, but I see some maintanance issues with this (none of them are blocking, but make sense to be aware of it):

• This codebase heavily depdens on external components (e.g. Selenium) that are keep changing, so this codebase needs a massive ongoing maintanance.
• As they are in a separate repo, it is harder to detect compatibility issues caused by Reqnroll changes (e.g. the configuration action was not updated to SpecFlow v4).
• The repo contains various different integrations that need different compile time dependencies (e.g. android), so it is not that easy to just "get and compile" all of them.
• There are internal dependencies between the plugins (e.g. almost all of them depends on the "Configuration") that makes the maintanance harder.
• Some of the plugins (especially "Configuration") is a quite deep an nasty hack of Renroll. Maybe this should be rather a feature of Reqnroll, so that the plugins remain simpler.
• There are some integration plugins for paid automation services. According to the Reqnroll Sustainability Strategy, we should provide services for non-free products if they sponsor Reqnroll.

I don't have a clear suggestion on how to move on. I am interested in your comments & feedback.

[gasparnagy]

(Side note: @tulavalle offered getting ideas about Selenium integration from the lib at https://github.com/tulavalle/SpecflowSelenium).

[SabotageAndi]

Some background info:

This codebase heavily depdens on external components (e.g. Selenium) that are keep changing, so this codebase needs a massive ongoing maintenance.

The idea of Actions wasn't to make whole wrapper around the APIs. It was to get easier started with Selenium/Playwright together with SpecFlow. So they provide a light opinionated way to integrate them in the SpecFlow lifecycle.
And for this, the APIs are normally quite stable within major versions of the external dependencies.

We had some helper methods for Selenium, but these are bare minimum. You are at their limit very fast.

As they are in a separate repo, it is harder to detect compatibility issues caused by Reqnroll changes (e.g. the configuration action was not updated to SpecFlow v4).

The reason for this was, to make contributing to it easier. The SpecFlow repo was too big and hard to navigate for newcomers.

There are internal dependencies between the plugins (e.g. almost all of them depends on the "Configuration") that makes the maintanance harder.

This was mainly done to give people who used SpecFlow+ Runner Targets a way to migrate away from it.
That is the big feature in the configuration Action. And as most other Action needed some configuration, they are using it.

There are some integration plugins for paid automation services. According to the Reqnroll Sustainability Strategy, we should provide services for non-free products if they sponsor Reqnroll.

Back in the days it was the other way around. We wanted to make it easier for users of this payed platforms to start with SpecFlow.

[gasparnagy]

Thx for the insights.

So they provide a light opinionated way to integrate them in the SpecFlow lifecycle. And for this, the APIs are normally quite stable within major versions of the external dependencies.

That idea makes sense, but in practice it does not really work. E.g. currently the code also depends on the browser divers of all major borwsers that change very frequently. Maybe the plugins itself do use only core API, but the examples & tests next to them heavily depend on changing stuff. But I also don't know how this could have been done better.

And as most other Action needed some configuration, they are using it.

I did not check the code in detail, but I would have expected that the plugins itself don't need to depend on the configuration, but if you want to use them with multiple browsers, you can use the configuration action in your project code (so composability over dependency).

I think the big question is how much this multi-target feature is a common need once the legacy of SF+ Runner erodated. Let's see. If it is, we can have it in the core Reqnroll and use the reqnroll.json config file to keep the things simpler.

We wanted to make it easier for users of this payed platforms to start with SpecFlow.

Absolutely fine, I'm not against this at all. But at the end it is mainly the interest of the commercial vendors, so we should concentrate our forces for those areas that do not have business backup. It's a different setup now.

[SabotageAndi]

I think the big question is how much this multi-target feature is a common need once the legacy of SF+ Runner erodated.

I always got the questions how to run your scenarios against different browser without duplicating scenarios. First the answer was, use SpecFlow+ Runner, then use SpecFlow.Actions.

[gasparnagy]

I would just add one more line to the CI to run the tests with a different config.
Running all test combinations locally is anyway overkill and anti-bdd.

[Code-Grump]

Sounds like we could solve with an example of multi-browser testing, if that's the goal being sought.

[tulavalle]

A point to consider in relation to actions (for e2e tests) is that it multiplies the feature by configured action.json, which is very good, but for simpler tests that do not involve a mass of data. I discontinued its use in some larger projects precisely because when running with the Chrome browser action, for example, the mass of data would be changed and the same test would fail in the Firefox browser, the tests run in parallel with xUniot, that is, the parallelism occurs per feature. We tried a strategy to clean up the mass in each scenario, but it didn't pass in terms of maintainability... But i continue to use actions for simpler tests of the application without involving editing or deleting records

[gasparnagy]

One more question, @SabotageAndi, just that I see it clearly.

Let's take the selenium action as an example. It depends on SpecFlow (Reqnroll) AND Selenium.

What was the intended release strategy of the action?

1. Release for every major version of SpecFlow (Reqnroll) AND Selenium
2. Release for every major version of Selenium
3. Release for every major version of SpecFlow (Reqnroll)

What is the intended usage for teams that plan to upgrade SpecFlow (Reqnroll) but not ready to upgrade Selenium or vica versa?

Can the users override the used Selenium package (let's say use Selenium 4, while the action was compiled with Selenium 3) in their client project (as long as the tiny bit we use from Selenium remains unchanged)?

I guess the same question applies for our DI integration packages as well. @mbhoek, what was your strategy with the MS DI plugin?

[gasparnagy]

Makes sense, thx.

[mbhoek]

For the MS DI plugin, I looked at the lowest .NET SDK version SpecFlow was targetting, and targeted the Microsoft.Extensions.DependencyInjection version that was included in that .NET version. This only works for the MEDI plugin I guess because it is part of the .NET SDK that SpecFlow is targetting (i.e. no independent version releases).

[tulavalle]

I agree, i would stay with Selenium 4. The cost of migrating from Selenium 3 to 4 is not great, it is well documented, I really don't see any reason to invest in 3. Resuming the selenium project in relation to the project I made available, as already discussed , I understand that it would be worth streamlining, particularly I would maintain the driver creation logic, and the PageFactory logic that was discontinued in Selenium 4 but is of great value. It would remove the driver options logic and go through parameters, considering the countless configuration possibilities, it would be unfeasible to parameterize everything. I would remove the other items, at least initially, and treat the evolution according to demand, avoiding something big without need. @gasparnagy , i liked the idea of ​​passing action parameters through reqnroll.json. I believe that some parameters will be inevitable, but I don't immediately see such a large number of them. And thinking further, wouldn't it be possible to pass these acton parameters from reqnroll.json to the context of the scenario, or something similar when the tests start running?

[gasparnagy]

@tulavalle yes, I was also thinking about that.

[DomZZ]

in my situation, I have forked specflow actions to have the opportunity to use the last versions of selenium/playwright packages.

[Code-Grump]

I'd like to suggest we approach this from a completely different angle.

Rather than trying to provide helper libraries to get people started with their particular application of Reqnroll, could it be better to supply scaffold projects that generate the code and structure necessary for the particular set of problems? We were chatting about this in the Discord, whether it would be good to help users with ASP.NET applications get up and running. It would give us an opportunity to not only solve the problem of "getting started" but also give some guidance on how to structure a non-trivial test-suite and examples of what good scenarios look like in their problem space.

This is essentially how Microsoft approached the problem of getting started with ASP.NET, and the result is every project I've ever worked on has a reasonable and predictable structure. I know what I should expect in Program.cs, I know that a "Controllers" folder is almost certain to contain their controller classes. There are projects that don't follow these conventions, but they're the vast minority, because most users simply don't have a good reason to move away from the default.

I would love the same to be true for Reqnroll, where an ASP.NET integration test suite correctly sets up its server and dependencies, correctly configures some mocks, invokes a Selenium driver to interact with pages via some kind of model, and these things are common across projects, rather than exceptional.

Is it possible for us to achieve this for the common integration scenarios we know of? Does anybody have knowledge of how we might scaffold projects in this fashion?