Reqnroll
Run feature test in parallel does not respect "maxParallelThreads" in xUnit #328
-
|
I'm migrating my specflow test project to reqnroll, I've been able to fix some issues I've found, but there's one issue I'm not able to fix. I used to run the feature files in parallel in specflow where the maxParallelThreads set the maximum number of feature files to run in parallel, so if set to 2 there where 2 drivers running all the scenarios from that feature file and no other feature file started until one was finished. But what currently happens is that at the start the max threats are respected and only that number of feature files are run, but as soon as one of the scenarios has finished (even though there are many more in that feature) another feature has started and it continues without respecting the maxParallelThreads. This is my xunit config file (which worked with specflow)
This is how it starts This is how continues This is my VS Test explorer with the feature files |
Beta Was this translation helpful? Give feedback.
Replies: 6 comments 11 replies
-
|
I have tried to reproduce this with our basic sample project: https://github.com/reqnroll/Reqnroll.ExploratoryTestProjects/tree/main/ParallelExecution/ReqnrollCalculator.Specs.XUnitFw, but could not. In that sample there are 3 feature files and if I set the "maxParallelThreads" to "2" from json (or from code) it completes the two features is starts with and only after it continues with the third one. Could you please try to reproduce it on a minimalistic sample project that you can share? In addition to that, you can try to change the config from code, instead of json to check if it has any impact (see sample here). Also you can try to make some logging to a file, like what I do here. This produces a log file where you can track the execution order. In my example it produced: |
Beta Was this translation helpful? Give feedback.
-
|
Hi @gasparnagy thanks for your answer, I tried to log the order of the test, in this case I run 3 feature files with the maxParallelThreads set to 2 also, the Auth has 5 tests scenario outlines for a total of tests, the Products has 7 tests scenario outlines for a total of 7 tests and the FileProcessor has 3 scenario for a total of 3 tests and also I have a [ScenarioDependencies] tag. Looking at the logs I see that it starts running 2 tests but then it continues with 3 and the after feature log is run at the end in all iterations I've tried.
|
Beta Was this translation helpful? Give feedback.
-
|
@llbonet14 Are you sure you have configured xUnit worker counts correcty? Because from your trace, it seems that there are 3 workers running in parallel (the number after the timestamp is the worker id): So it seems that worker 4 deals with Please try to set the config from code (see example) to make sure the problem is not with the xUnit JSON config. |
Beta Was this translation helpful? Give feedback.
-
|
I use this file
And this is the configuration I use with specflow, the strange thing is that worker 4 does not start until one test of other workers have finished. As you can see worker 2 and 3 start at the same time and after 48 seconds the other worker starts. Also the all the AfterFeature are run at the same time after all test from all featureFiles have finished, but some features have finished before |
Beta Was this translation helpful? Give feedback.
-
This is not true. If you check the beginning of your log, you can see that the first "AfterScenario" (so the first finished test) is at 1:16, while worker 4 starts already at 0:48.
This is a behavior that Reqnroll introduced. It delays the execution of the AfterFeature until a test from another feature has been scheduled to the same worker. So this is by design.
I don't know. From your log it seems that there are 3 workers running: for example at 1:00 of your execution time the scenarios
have all started and none of them finished yet. As you see my trace above, it worked for me, so there must be something special in your setup that causes this. You should somehow try to figure out what it is, so that we can help. This is why I suggested to test the code-based configuration, but maybe there is something else too. |
Beta Was this translation helpful? Give feedback.
-
|
I downloaded your test project |
Beta Was this translation helpful? Give feedback.
-
|
Could this be related to xunit/xunit#2003? |
Beta Was this translation helpful? Give feedback.
-
|
Hi @gasparnagy could you reproduce this behavior in your local project with the latest xunit versions? |
Beta Was this translation helpful? Give feedback.
-
|
I have migrated from xUnit to Nunit and using this assembly it works perfectly, so it's a xunit problem with the await async tests. |
Beta Was this translation helpful? Give feedback.
-
|
No it is a misunderstanding of what that setting does. MaxParallelThreads controls how many test threads are started. But it never was about what the test thread actually does. It can run multiple tasks consecutively from different tests. So if both test A and B have awaits, the following is a valid run scenario: With two threads it could become something like the following (though unlikely): See also: xunit/xunit#2670 You could very well end up with more started tests (and features) that there are test threads configured, but that is more of an edge case. |
Beta Was this translation helpful? Give feedback.
-
|
Since Xunit 2.8, Xunit has the However, this does not work with Reqnroll.Xunit. Because the Regnroll runner which is registered with an Assembly Attribute which Regnroll creates does not know about this, and does not switch the mode. I would assume because of backward compatibility from the Xunit side. I tested this in a case. We had Xunit conservative mode enabled in a project with Reqnroll, all tests would still span up in parallel (e.g. Then we removed the Reqnroll NuGet from the (and the feature files) from the Project. So that the Reqnroll Runner was not registered in the generated code (because no code was generated), then Xunit suddenly respected the We could easily see this, because each test created a browser instance (via Playwright). With Regnroll with had dozens of open browser (which leads to resource problems), without Reqnroll, it opens just 4 (our max threads) in To fix this Issue, the Reqnroll Xunit Runner should be adapted so that the underlying implementation respects the conservative mode, don't know what to do for this. It could be just instruct Xunit to do so, or implement the algorithm by itself (which i don't expect). |
Beta Was this translation helpful? Give feedback.
-
|
@DerAlbertCom This is a good info. Could you please create an issue for that at https://github.com/reqnroll/Reqnroll/issues/? If you can also help with the fix, that is also appreciated because you seem to have investigated this topic already pretty deeply. |
Beta Was this translation helpful? Give feedback.
-
|
I asked xunit/xunit#3176 for a hint, hope to get a good answer to Kickstart the effort. |
Beta Was this translation helpful? Give feedback.
-
|
So, for xUnit v2 i think i have now all Information, it's basically "reimplementing" the xUnit v2 is done and finished. To really fix this little flaw the For xUnit 3 i have not looked, but it's seems way different and needs to solve differently, and also we have the offer hat xunit v3 could improved on an API if we have to much work to integrate it. |
Beta Was this translation helpful? Give feedback.
-
|
Thx. I think this fine (take in the code). I have also commented the related issue: #473 |
Beta Was this translation helpful? Give feedback.
Since Xunit 2.8, Xunit has the
conservativemode for https://xunit.net/docs/configuration-files#parallelAlgorithm, which then ensures that only such much test are running concurrently which are configured as max threads. That works.However, this does not work with Reqnroll.Xunit. Because the Regnroll runner which is registered with an Assembly Attribute which Regnroll creates does not know about this, and does not switch the mode. I would assume because of backward compatibility from the Xunit side.
I tested this in a case. We had Xunit conservative mode enabled in a project with Reqnroll, all tests would still span up in parallel (e.g.
agressivemode), even the direct async Xunit tests.T…