Living Documentation

[Code-Grump]

Having seen a few mentions of the SpecFlow Living Docs plugin, it seems like there's a demand for an equivalent feature in Reqnroll.

What I'd like to understand is:

• Does it matter when this output is produced?
• Should it be the output of a build, or a test-run?
• Would running a stand-alone tool to extract this information be preferrable to something built into Visual Studio?
• What do we think should be the output?

I can think of many ways to approach this, so knowing the constraints is probably the easiest way to find a path forward.

[gasparnagy]

Thx for taking care of this.

I don't think there are single answers for these questions (there are so many situation and needs), but let me share my thoughts based the usages of SpecFlow+ LivingDoc I have seen.

Does it matter when this output is produced?

The most common usage is to produce the report at the same time of the test execution or directly after in CI (or locally). SpecFlow+ LivingDoc CLI enforced a separate step (generated the data during test execution and run the HTML gen after), but I know that many teams tried to merge this by calling the HTML gen from an AfterTestRun hook.

Should it be the output of a build, or a test-run?

I think the most common usage is to use it with test-run, but I have seen usages where they were only interested in the nice rendering of the feature file and not interested in the test result. So both, but more commonly related to test-run.

Would running a stand-alone tool to extract this information be preferrable to something built into Visual Studio?

Yes, definitely, as it is going to be used mostly as part of the CI. I would even say a "stand-alone command line tool" (no UI).

What do we think should be the output?

I think there are diverse answers for this. The most simple and common usage would require a single-file active HTML (maybe multi-file HTML is also acceptable, but more problematic to share). But there are teams who use this as part of their QA process, where they might need to produce other output (PDF, Word, ???). But I think this latter is more the special case.

When I say active HTML, I mainly mean that something where you can browse the folders and feature files, search, etc. within the offline HTML.

Besides these questions, there are plenty of "nice-to-haves" that I have heard of. These are not necessarily part of the primary goal (or a goal at all), but good to think about them when we set the design. (SpecFlow+ LivinDoc supported some of these.)

• Customizable HTML output (templating?)
• Have summary controls (charts) in the result
• Non-HTML output (PDF, Word, JSON)
• Seamless integration of the result document to common ALM systems, like Azure DevOps, Jira, GitHub
• Manage result history

[tulavalle]

Hi @gasparnagy! How is the live documentation part going? Publishing the HTML extension on Azure is a prerequisite for us to finally migrate from Specflow to Reqnroll and take advantage of all the improvements made. I must admit, I'm very excited about this.

I would like to understand the status of this request, if there is anything in progress, any planning, and if https://github.com/praticandobdd and I can actively contribute (if so, what contribution would add value), especially in terms of development and testing. We have a rich environment for testing this entire process, as well as experience to share with this community.

[gasparnagy]

@clrudolphi is currently working on the cucumber messages support that is a prerequisite of this (see #233), but we realized that it is a bigger task than we (or at least I) expected. Once this is done, we can work on the HTML generation.

In principle we could work on the HTML generation even before, because we can get sample Cucumber Messages reports from other Cucumber implementations. But to start with that, we would need some design concept at least. My preference would be to use React (like the old one did), so that we could reuse something from https://github.com/cucumber/react-components. But that's all I have and I'm not a React expert. If you (or anyone else) is fluent in React, you could start working on a prototype...

[blaisep]

Hi @gasparnagy ... (I just learned about this proiect from Lisa Crispin)
If there is support for python, then you can use the tooling like Sphinx or MKdocs that currently power https://readthedocs.io
Sphinx will output to many formats (PDF, Confluence, ePub, Man Pages, HTML, etc) and if you only need HTML then there are many more options.

The tricky thing about live docs is not so much the HTML design but rather the mapping of source code to abstractions like definitions (Glossary, API, etc) and other cross references. In this sense Sphinx, AsciiDoctor and Antora would be the tools to consider.

[gasparnagy]

@blaisep Thx for the comment. Yes, that's why we currently focus on generating cucumber messages that contain all these information. Once we have that we can find the right tooling to produce HTML. Do you know maybe any tool that can generate report from Cucumber Messages .ndjson file directly?

[a-korevaar]

Having a LivingDoc equivalent feature would allow us to migrate from SpecFlow to Reqnroll. I would gladly contribute to the development or testing of such a feature.

[Code-Grump]

Can you tell us what you need the output to be? Any technical constraints?

[a-korevaar]

We're currently using the SpecFlow Azure DevOps extension and the CLI to generate LivingDoc in HTML format and publish it externally. I have added some custom code to the build pipeline to include images (mainly screen mockups) as well.

For us, the integration in Azure DevOps is a nice to have, as developers are focused on the feature and step files in their local repositories and product owners and stakeholders use the published documentation.

The HTML format is ok, but it would be nice to have other options such as markdown or pdf as well.

I'm not familiar with Cucumber reports but it looks interesting. We are for the most part bound to on-premises solutions though.

[Code-Grump]

We are for the most part bound to on-premises solutions though.

This is good to know as a constraint. I'm certain it's a common one too.

[gasparnagy]

I agree. Test results are highly confidential in many environments I worked with and storing any test result data introduces a lot of complication (GDPR, etc.).

[adrichem]

I love this is getting attention! LivingDocs reports are the only useful source of test results for us. Especially the test runner mixes output from concurrently running scenarios.

1. We need to generate and store results in our CI/CD pipelines .My org needs flexibility in deciding where/when this work is done as we have different projects using different setups in their pipelines. Some are YAML, some are classic. Some are build- some are release. Some pipelines only run tests and have no 'real' branch associated with them. Having LivingDoc invoked by shell script ticks all our boxes w.r.t. integration into our devops.
2. Having HTML output is critical for us as we include the HTML into viewers configured in DevOps on the pipelines..
3. Its very nice LivingDocs contains the markdown from the scenarios. This helps communicate why tests do what they do.
4. When running locally we sometimes manually create a report (both for test results but also to show the content of scenario to someone in a nice format). So ability to execute multi platform is needed.
5. The Integration with Azure DevOps is not something we use yet. We are entering a situation where its going to be used. Even then not always.....for end-to-end tests there is no 1-to-1 correlation between test results and a specific branch.
6. We need LivingDoc 's ability to filter on outcomes.
7. The analysis on step-definitions for us adds no value as they are shared over multiple projects. Its normal for a step-definition to not be used in a test run.
8. We use the ISpecFlowOutputHelper to include OpenTelemetry TraceId's and links to logfiles (Blobstorage / S3 etc etc) into the LivingDoc report.

[gasparnagy]

Thx. The "filter on outcomes" is especially a good point that I did not think of. Also good to know your feedback about the analysis on step-definitions. I had the same feeling.

[Digbigpig]

Hi I'd like to chime in
In the past i've had a great experience using specflow and specrunner+. I'm now setting up a new test automation solution and realized we are without a reporting feature in reqnroll. Like others have mentioned, it could be a reason to delay reqnroll adoption over specflow.

The main feature we need is the ability to generate an html report that can be easily navigated and parsed. We'd like it to support navigation/ filtering between test categories. We'd like for easy screenshot integration into the report. Being able to see the step, the timing, and console output for each test is an expected feature.

I come from using spec runner and that was everything I needed. The livingdoc stuff looks cool and has nice features but some of it seems overkill for my general use case, however I never used it besides failing to integrate it with reqnroll :)

PS reqnroll sounds closer to rick-roll than rock n roll to me🕺

[gasparnagy]

There is no realistic chance that we will be able to redo the SpecFlow+ Runner. What would be good though is to make the livingdoc generator HTML "template" more extensible, so one could make a more "classic test report" kind of template for it (we can even put it to the product itself).

[brajpal-jangir]

Thanks for bringing this in discussion!

We used CLI command to generate reports. These are integrated in DevOps pipelines.

• In the active development phase, we generate living doc without results to let Product owner and other stakeholders review the test coverage. Living doc also have links to UserStory (from TFS) to validate the requirement.
• In times of GA, we generate living doc with results and archive them as test metrics as part of GA.

I also tried to use SpecFlow+LivingDoc DevOps extension, but it has some limitations and that’s where I switched to use CLI. Few limitations which I came across are

• We have multiple Specflow Projects in GitRepo and it’s not possible to publish multiple Living doc for single git branch. It will always override the previous uploaded document for same git branch.
• We also came across the issue when Living document HTML file size is more and it failed to publish document on DevOps. I reported this issue but am not able to find the link now.

[MathiasLopes]

I've found a way to generate LivingDoc HTML from Reqnroll unit tests. I've made a GitHub project to explain it, here's the link if you're interested:
https://github.com/MathiasLopes/ReqnrollTemporaryLivingDoc

[RajeshPoola]

They pulled the plug on Specflow https://support-hub.tricentis.com/open?id=post&number=NEW0001432

[tulavalle]

Yes @RajeshPoola. That's why I came to change this idea of ​​what you're doing to view the post-deploy report.

[RajeshPoola]

@tulavalle They are currently not allowing new signups or logins. Mine is still working as I signed up a long time ago. However, if my browser or Azure forces me to reauthenticate, it could become a problem for me too.

[rdvanbuuren-gasunie]

Did anyone manage to run livingdoc with the test-assembly option and .NET 9?
I'm getting the following error:

Tool 'specflow.plus.livingdoc.cli' (version '3.9.57') was successfully installed.
Framework: .NET 8.0.12
Warning: Unable to load one or more of the requested types.
Could not load file or assembly 'System.Runtime, Version=9.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'. The system cannot find the file specified.

I noticed that Framework: .NET 8.0.12 is being selected, however the project is .NET 9. I can reproduce this locally so I'm wondering if I can force the Framework somehow. According to the NuGet it should also support .NET 9 and in one pipeline run I've seen it work. Maybe someone knows how it picks the framework?

[gasparnagy]

I haven't tried, but you can try with setting the DOTNET_ROLL_FORWARD env variable to LatestMajor (or maybe Major) and run it again.

[ShanmukhaGajula]

Yes I do.. thanks for the input. One question how can i add base64 image to the reporter?

…

On Wed, Aug 14, 2024, 11:41 AM dominik-niebuhr ***@***.***> wrote: @MathiasLopes <https://github.com/MathiasLopes> , small correction here. I gave the command as livingdoc feature-folder "mypath to features folder" --output-type HTML -t bin/Debug/net8.0/TestExecution.json instead of livingdoc feature-folder . --output-type HTML -t bin/Debug/net8.0/TestExecution.json but after removing the path it worked. Thanks again for the workaround. Appreciate your inputs. Have you got the test results to show up? — Reply to this email directly, view it on GitHub <#196 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AFBWBLKU5SQDZMCEN5KKEYLZRLYHTAVCNFSM6AAAAABKFOZSCGVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTAMZTGM2DQNI> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[Pressacco]

Status Update

As I understand it, the current game-plan is:

• to be compatible with the Cucumber eco-system by supporting Cucumber messages
  • in order to integrate with tools like Pickles, Report Portal, etc.
• which is being tracked by Feature/Cucumber Messages #233

yes?

[clrudolphi]

Yes. Support for Cucumber Messages is the first step.