Connecting Requirements to tests

[i-interprete-on-the-fly]

Hey,
does StrictDoc also aim to enable traceability between the requirements and respective test cases? (and maybe also accommodate for test result display in some format)?

What I currently need would be comprehensive traceability system between Requirements, refined requirements, their source code implementation and also a linkage to test cases.

Edit - To provide more context: I want to introduce this tool for requirements management in the ASPICE relevant projects of my company. I think I could technically represent the test cases as [REQUIREMENT] elements with relations as the tests being the childs of the requirements but I'm not sure if this is the most elegant way and also if you may have another possible solution in mind.

Thank you for this nice tool!

[i-interprete-on-the-fly]

I'll have to adjust my question a bit here. Through taking a closer look at the grammar and the specific relations I saw that I can create [TESTCASE] elements without any hassle and also define a 'verifies' parent relationship between the test case and the requirement. For the part of linking test cases (maybe automatically in the context of CI) I could then use the file location to a test case result summary file with the verdict.
What would be interesting in this scenario is: Have some kind of dynamic field in the test case node which updates according to the linked file content, maybe through so simple condition syntax?

[stanislaw]

I'll have to adjust my question a bit here. Through taking a closer look at the grammar and the specific relations I saw that I can create [TESTCASE] elements without any hassle and also define a 'verifies' parent relationship between the test case and the requirement. For the part of linking test cases (maybe automatically in the context of CI) I could then use the file location to a test case result summary file with the verdict.

Coming back to this one after some time. We have implemented 80%+ of the work needed for the custom grammar elements to be editable in the UI. I say 80%+ because based on the experience the other 20% will be spent on handling the corner cases that were not obvious during the implementation. This means that from now on, the non-requirement nodes should be also editable in the web user interface. We will enable editing of the File links in the UI very soon too.

What would be interesting in this scenario is: Have some kind of dynamic field in the test case node which updates according to the linked file content, maybe through so simple condition syntax?

For this kind of dynamic fields updates, I could think of a command-line interface command or a UI button that scan over all requirements and performs a certain action. Could you give a more concrete example of what you would like see updated: which field should updated with which content?

[nogjam]

We are looking for something similar to this at my workplace. (Thanks @i-interprete-on-the-fly for starting the discussion!) I'm imagining something similar to TDD, but for requirements (call it RDD?):

1. Write a requirement, REQ-1.
2. Run some test script. Maybe it's part of the strictdoc CLI. It fails because REQ-1 is not covered by any tests.
3. Write a test that links to said requirement.
4. Run test script again. It still fails because although you have linked a test to REQ-1, the test does not pass.
5. Write code to make the test pass.
6. Run test script a final time. It passes because all your requirements are covered by passing tests.

At the end of the day when you have fully specified the system and all its requirements are covered by passing tests, you should be able to have confidence that your job is done. Fully specifying the system with requirements would be up to the human in the loop, whereas StrictDoc could handle linking requirements to tests and test results.

It would be super helpful to have a UI view similar to Traceability Matrix. Except instead of file details, it would focus on a minimal view involving just requirement titles or UIDs and a color/symbol/something to indicate the status of your requirements. This would make it easy to visualize progress on bringing a developing system to completion.

Here's an example diagram that show roughly what I'm thinking:

In this diagram, the overall status of "SYSTEM SPEC" is equivalent to the worst status of any of its children, PASS being best, FAIL being worse, and NONEXISTENT being the worst. Hopefully that makes sense. And maybe in the future it could be a little more granular so as to understand what percentage of tests are passing in the "FAIL" case, something like we already have on the Source Coverage UI view:

Maybe this could be associated with a project feature called "REQUIREMENT_DRIVEN_DEVELOPMENT" or similar.

Mostly just ideating here, and still familiarizing myself with all that StrictDoc has to offer. Let me know if there are current features I'm missing that it already has, or if this is something that would otherwise fit on the roadmap!

[nogjam]

For this kind of dynamic fields updates, I could think of a command-line interface command or a UI button that scan over all requirements and performs a certain action. Could you give a more concrete example of what you would like see updated: which field should updated with which content?

To answer this question @stanislaw , I think the desire would be a command like strictdoc rdd to scan all requirements. For each requirement, it would:

1. Find all test cases associated with it.
2. Find all results associated with each test case.
3. Determine the status of each test case based on its results: PASS or FAIL, and if FAIL, what percentage have passed?
4. Determine the status of the requirement: PASS, FAIL, or NONEXISTENT and the proportion of each.
5. Update metadata/cache that the UI needs to display this information.

[mplum]

Would the following abilities satisfy the use case of linking in automated test results?:

1. scan source_path files for '@relation' links OR
2. '@relation' links to support link roles, something like: @relation(REQ-1, scope=line, role=verifies/pass/fail)
3. Search / Query file links to locate tests and results
4. Optionally in the UI: configure node colors based on link roles (pass=green, fail=red, none=grey, ....)

Since automated test results do not exist initially, its not currently possible to link them as source files from requirements. If file links could be created like parent links in the source_path files, then test result files could be linked to requirements dynamically as they are created without modifying requirement nodes.

The workflow would look like:

1. Author requirements
2. Author automated tests with cases linked with: @relation(REQ-1, ..., role=verifies)
3. Execute tests, where results are produced with @relations in tact OR
   3.1) A custom test runner could produce links with 'pass/fail' roles based on actual status: @relation(REQ-1, ..., role=pass/fail)
4. Run Strictdoc to view and query requirements with and without test / test result links

A benefit of this method would be that The 'source_path' directories are already part of the project configuration, and this could be solved without a custom command.

[haxtibal]

Hi @mplum, for the reference, @relation(REQ-1, scope=line, role=verifies/pass/fail) is under development in #2104. Feel free to comment there.

[fkromer]

Hi @haxtibal, is the PR specific for Python test code? Would be great to have a generic concept which allows to trace low level C/C++/Rust tests as well. FYI @stanislaw

Hi @mplum, for the reference, @relation(REQ-1, scope=line, role=verifies/pass/fail) is under development in #2104. Feel free to comment there.

[haxtibal]

is the PR specific for Python test code? Would be great to have a generic concept

Hi @fkromer, the @relation(..., role=whatever) PR is generic and doesn't depend on language or framework. It's really just an additional attribute that can hold any user defined value.

However, #2107 (to parse test results back to sdoc), or cheats like #2107 (comment) (to enrich test reports with sdoc UIDs by pytest introspection) are language/framework specific.

#2107 is generic regarding the concept IIUC, but we have to write a dedicated internal adapter for every framework we want to support.

[stanislaw]

I would like to connect this thread with another thread: #2009. Just to make sure you are aware of both threads @i-interprete-on-the-fly @mplum @nogjam @johanenglund.

I have been busy doing something else with StrictDoc but I will come back to this topic within this week.

[stanislaw]

Just a heads up. I am looking into this with a medium priority, and this feature seems to be an important missing building block in the overall StrictDoc feature set. So far I see three proposals which are all similar but not the same:

1. What @johanenglund started prototyping in Feature: Requirement verification functionality #2009. The keywords: Custom TestCase grammar elements, JUnit XML, XLST.

2. What @nogjam proposed here and in the comment below. The keywords: rdd command that regenerates the verification statuses for requirements in a screen similar to Traceability Matrix.

3. What @mplum proposed here right above. The keywords: not using 'rdd' command, instead making test cases produce @relation(..., role=verifies/pass/fail).

From my side, I would like to use StrictDoc's own codebase to try the solution #1 to explore how usable it would be from my perspective. Also, I would like to learn more about the JUnit XML format and how easy it is to modify it.

The most natural flow from my point of view would be to make the test results link back to requirements and possibly the test cases. But we can of course consider the forward traceability when the test cases would link forward to the test reports.

[johanenglund]

Let me know if you want some help to test it out!

[mplum]

Sounds good, I like the idea of translating Junit to sdoc.

[stanislaw]

Just a heads up. I have been working on a few other features, and so I postponed testing your proof of concept. I feel like I need a good day or two to dive into this deeper but didn't have a chance recently.

@haxtibal is working on adding roles to the source file links which means we are anyway approaching having @relation(..., role=verifies/pass/fail) independently of what else will become The way to connect requirements and tests. My next work item is to support the integration of this work.

[stanislaw]

I have opened a separate issue for a dedicated JUnit XML -> SDoc generator script: #2107.

[fkromer]

@i-interprete-on-the-fly @stanislaw Have you already considered using/providing support for the test interchange format TestIF? ReqIF (for requirement interchange) and TestIF (for test interchange) should play nicely within strictdoc as well as Interface to other tools.