Skip to content

Commit 3c4cc8c

Browse files
authored
Merge pull request #2251 from haxtibal/tdmg/robot_test_report
Robot Framework part 2: Test report reader
2 parents 7364169 + d12f9c7 commit 3c4cc8c

File tree

11 files changed

+461
-22
lines changed

11 files changed

+461
-22
lines changed

docs/strictdoc_01_user_guide.sdoc

Lines changed: 46 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -3609,15 +3609,40 @@ This feature is not enabled by default because it has not received enough testin
36093609

36103610
[[/SECTION]]
36113611

3612+
[[SECTION]]
3613+
MID: 74291dc6cfae47d2bae33066024f3807
3614+
TITLE: Test report integration
3615+
3616+
[TEXT]
3617+
MID: 07e78a4847554fbd87c44d7b11133f68
3618+
STATEMENT: >>>
3619+
StrictDoc can read test report files from several testing tools as SDoc content which allows establishing traceability between requirements, test cases, and test results.
3620+
3621+
The project must have the following features activated for StrictDoc to provide language-aware parsing of C/C++, Python or Robot Framework. It is recommended to provide a dedicated folder for test report XML files to not mix human-written SDoc content and auto-generated test reports.
3622+
3623+
.. code-block::
3624+
3625+
[project]
3626+
3627+
features = [
3628+
"REQUIREMENT_TO_SOURCE_TRACEABILITY",
3629+
"SOURCE_FILE_LANGUAGE_PARSERS",
3630+
]
3631+
3632+
include_doc_paths = [
3633+
"docs/**",
3634+
# A dedicated folder where StrictDoc could find JUnit XML.
3635+
"reports/**",
3636+
]
3637+
<<<
3638+
36123639
[[SECTION]]
36133640
MID: e9c94abacb12425980aec8ed7ee752cb
3614-
TITLE: JUnit XML report integration
3641+
TITLE: JUnit XML
36153642

36163643
[TEXT]
36173644
MID: eba63ebd45624139a8773cf42fe7d670
36183645
STATEMENT: >>>
3619-
StrictDoc can read JUnit XML files as SDoc content which allows establishing traceability between requirements, test cases, and test results.
3620-
36213646
JUnit XML format is mostly identical between different tools that produce it but there are subtle differences that must be handled case by case. StrictDoc supports several JUnit XML flavors. To be discovered by StrictDoc, the XML files must have one of the following extensions:
36223647

36233648
.. list-table:: Supported JUnit XML formats
@@ -3640,28 +3665,13 @@ JUnit XML format is mostly identical between different tools that produce it but
36403665

36413666
After the existing JUnit XML flavors have been implemented, it should be easy to add more tools in case their output differs.
36423667

3643-
The project must have the following features activated for StrictDoc to provide language-aware parsing of C/C++ and Python. It is recommended to provide a dedicated folder for JUnit XML to not mix human-written SDoc content and auto-generated test reports.
3644-
3645-
.. code-block::
3646-
3647-
[project]
3648-
3649-
features = [
3650-
"REQUIREMENT_TO_SOURCE_TRACEABILITY",
3651-
"SOURCE_FILE_LANGUAGE_PARSERS",
3652-
]
3653-
3654-
include_doc_paths = [
3655-
"docs/**",
3656-
# A dedicated folder where StrictDoc could find JUnit XML.
3657-
"reports/**",
3658-
]
3659-
36603668
.. warning::
36613669

36623670
The JUnit XML feature's status is experimental. The functionality has been implemented and passes basic integration tests but it has not received enough testing by the users. StrictDoc's own documentation is already using this feature and the JUnit XML traceability will be part of StrictDoc's own qualification package. It is expected that the JUnit XML will become a stable feature by no late than 2025 Q4.
36633671
<<<
36643672

3673+
[[/SECTION]]
3674+
36653675
[[SECTION]]
36663676
MID: 06d9dbbeca204f5e8d0042e142b615bd
36673677
TITLE: LLVM Integrated Tester JUnit XML specifics
@@ -3689,6 +3699,22 @@ In this example, the expectation is that a user has generated a JUnit XML with L
36893699

36903700
[[/SECTION]]
36913701

3702+
[[SECTION]]
3703+
MID: c7f5451905bf4fdb9a3a7dc97bb0e853
3704+
TITLE: Robot Framework XML
3705+
3706+
[TEXT]
3707+
MID: 6c3d98ca65cc40d585953237693ff2c6
3708+
STATEMENT: >>>
3709+
Robot Framework has its own native XML test report schema, sometimes referred to as ``output.xml``. Files must end with ``.robot.xml`` to be recognized as such.
3710+
3711+
.. warning::
3712+
3713+
The Robot Framework test report feature's status is experimental.
3714+
<<<
3715+
3716+
[[/SECTION]]
3717+
36923718
[[/SECTION]]
36933719

36943720
[[SECTION]]
Lines changed: 232 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,232 @@
1+
from typing import Dict, Optional, Union
2+
3+
import robot.result
4+
from robot.api import ExecutionResult, ResultVisitor
5+
6+
from strictdoc.backend.sdoc.document_reference import DocumentReference
7+
from strictdoc.backend.sdoc.models.document import SDocDocument
8+
from strictdoc.backend.sdoc.models.document_grammar import DocumentGrammar
9+
from strictdoc.backend.sdoc.models.node import SDocNode, SDocNodeField
10+
from strictdoc.backend.sdoc.models.reference import FileReference
11+
from strictdoc.backend.sdoc.models.section import SDocSection
12+
from strictdoc.backend.sdoc.models.type_system import FileEntry, FileEntryFormat
13+
from strictdoc.core.file_tree import File
14+
from strictdoc.core.project_config import ProjectConfig
15+
from strictdoc.helpers.paths import path_to_posix_path
16+
17+
18+
class SdocVisitor(ResultVisitor):
19+
def __init__(self, project_config: ProjectConfig):
20+
self.project_config = project_config
21+
self.suites: Dict[str, Union[SDocDocument, SDocSection]] = {}
22+
self.document: Optional[SDocDocument] = None
23+
24+
def visit_suite(self, suite: robot.result.TestSuite) -> None:
25+
"""Create document for top level suite and sections for nested suites."""
26+
assert suite.full_name not in self.suites
27+
28+
if suite.parent is None:
29+
self.document = SDocDocument(
30+
mid=None,
31+
title=f"Test report: {suite.name}",
32+
config=None,
33+
view=None,
34+
grammar=None,
35+
section_contents=[],
36+
)
37+
self.document.ng_including_document_reference = DocumentReference()
38+
grammar = DocumentGrammar.create_for_test_report(self.document)
39+
self.document.grammar = grammar
40+
self.document.config.requirement_style = "Table"
41+
self.suites[suite.full_name] = self.document
42+
summary = self.summary_from_suite(suite, self.document)
43+
self.document.section_contents.append(summary)
44+
else:
45+
assert self.document
46+
assert suite.parent.full_name in self.suites, (
47+
"depth-first traversal expected"
48+
)
49+
parent_sdoc_node = self.suites[suite.parent.full_name]
50+
section = SDocSection(
51+
parent=parent_sdoc_node,
52+
mid=None,
53+
uid=None,
54+
custom_level=None,
55+
title=suite.name,
56+
requirement_prefix=None,
57+
section_contents=[],
58+
)
59+
section.ng_including_document_reference = DocumentReference()
60+
section.ng_document_reference = DocumentReference()
61+
section.ng_document_reference.set_document(self.document)
62+
self.suites[suite.full_name] = section
63+
parent_sdoc_node.section_contents.append(section)
64+
summary_sdoc_node = self.summary_from_suite(suite, section)
65+
section.section_contents.append(summary_sdoc_node)
66+
67+
super().visit_suite(suite)
68+
69+
def visit_test(self, test: robot.result.TestCase) -> None:
70+
"""Create TEST_RESULT node for each test case."""
71+
assert self.document
72+
assert test.parent and test.parent.full_name in self.suites, (
73+
"depth-first traversal expected"
74+
)
75+
parent_section = self.suites[test.parent.full_name]
76+
testcase_node = SDocNode(
77+
parent=parent_section,
78+
node_type="TEST_RESULT",
79+
fields=[],
80+
relations=[],
81+
)
82+
testcase_node.ng_document_reference = DocumentReference()
83+
testcase_node.ng_document_reference.set_document(self.document)
84+
testcase_node.ng_including_document_reference = DocumentReference()
85+
86+
# Only executed tests will become traceable.
87+
if not test.skipped:
88+
testcase_node.set_field_value(
89+
field_name="UID",
90+
form_field_index=0,
91+
value=SDocNodeField(
92+
parent=testcase_node,
93+
field_name="UID",
94+
parts=[test.full_name.upper()],
95+
multiline__=None,
96+
),
97+
)
98+
99+
# Absolute path will be replaced with relative path to strictdoc root
100+
# in FileTraceabilityIndex.
101+
abs_path_on_exec_machine = path_to_posix_path(str(test.source))
102+
if test.source is not None:
103+
testcase_node.set_field_value(
104+
field_name="TEST_PATH",
105+
form_field_index=0,
106+
value=SDocNodeField(
107+
parent=testcase_node,
108+
field_name="TEST_PATH",
109+
parts=[abs_path_on_exec_machine],
110+
multiline__=None,
111+
),
112+
)
113+
114+
testcase_node.set_field_value(
115+
field_name="TEST_FUNCTION",
116+
form_field_index=0,
117+
value=SDocNodeField(
118+
parent=testcase_node,
119+
field_name="TEST_FUNCTION",
120+
parts=[test.name],
121+
multiline__=None,
122+
),
123+
)
124+
125+
testcase_node.set_field_value(
126+
field_name="DURATION",
127+
form_field_index=0,
128+
value=SDocNodeField(
129+
parent=testcase_node,
130+
field_name="DURATION",
131+
parts=[str(test.elapsed_time.total_seconds())],
132+
multiline__=None,
133+
),
134+
)
135+
136+
testcase_node.set_field_value(
137+
field_name="STATUS",
138+
form_field_index=0,
139+
value=SDocNodeField(
140+
parent=testcase_node,
141+
field_name="STATUS",
142+
parts=[test.status],
143+
multiline__=None,
144+
),
145+
)
146+
147+
testcase_node.set_field_value(
148+
field_name="TITLE",
149+
form_field_index=0,
150+
value=SDocNodeField(
151+
parent=testcase_node,
152+
field_name="TITLE",
153+
parts=[test.name],
154+
multiline__=None,
155+
),
156+
)
157+
158+
if not test.skipped:
159+
# File path will be resolved in FileTraceabilityIndex.
160+
testcase_node.relations.append(
161+
FileReference(
162+
parent=testcase_node,
163+
g_file_entry=FileEntry(
164+
parent=None,
165+
g_file_format=FileEntryFormat.SOURCECODE,
166+
g_file_path="#FORWARD#",
167+
g_line_range=None,
168+
function=test.name,
169+
clazz=None,
170+
),
171+
)
172+
)
173+
174+
parent_section.section_contents.append(testcase_node)
175+
176+
def summary_from_suite(
177+
self,
178+
suite: robot.result.TestSuite,
179+
parent: Union[SDocDocument, SDocSection],
180+
) -> SDocNode:
181+
assert self.document
182+
summary_table = f"""\
183+
.. list-table:: Test suite summary
184+
:widths: 25 10
185+
:header-rows: 0
186+
187+
* - **Number of tests:**
188+
- {suite.statistics.total}
189+
* - **Number of successful tests:**
190+
- {suite.statistics.passed}
191+
* - **Number of failed tests:**
192+
- {suite.statistics.failed}
193+
* - **Number of skipped tests:**
194+
- {suite.statistics.skipped}
195+
"""
196+
summary_node = SDocNode(
197+
parent=parent,
198+
node_type="TEXT",
199+
fields=[],
200+
relations=[],
201+
)
202+
summary_node.ng_document_reference = DocumentReference()
203+
summary_node.ng_document_reference.set_document(self.document)
204+
summary_node.ng_including_document_reference = DocumentReference()
205+
summary_node.set_field_value(
206+
field_name="STATEMENT",
207+
form_field_index=0,
208+
value=SDocNodeField(
209+
parent=summary_node,
210+
field_name="STATEMENT",
211+
parts=[summary_table],
212+
multiline__="True",
213+
),
214+
)
215+
return summary_node
216+
217+
218+
class RobotOutputXMLReader:
219+
@classmethod
220+
def read_from_file(
221+
cls: "RobotOutputXMLReader",
222+
doc_file: File,
223+
project_config: ProjectConfig,
224+
) -> SDocDocument:
225+
execution_result = ExecutionResult(doc_file.get_full_path())
226+
sdoc_visitor = SdocVisitor(project_config)
227+
execution_result.visit(sdoc_visitor)
228+
if sdoc_visitor.document is None:
229+
raise RuntimeError(
230+
f"No test suite could be parsed from {doc_file.get_full_path()}"
231+
)
232+
return sdoc_visitor.document

strictdoc/core/document_finder.py

Lines changed: 15 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,9 @@
1313
from strictdoc.backend.sdoc_source_code.test_reports.junit_xml_reader import (
1414
JUnitXMLReader,
1515
)
16+
from strictdoc.backend.sdoc_source_code.test_reports.robot_xml_reader import (
17+
RobotOutputXMLReader,
18+
)
1619
from strictdoc.core.asset_manager import AssetManager
1720
from strictdoc.core.document_meta import DocumentMeta
1821
from strictdoc.core.document_tree import DocumentTree
@@ -95,6 +98,11 @@ def _process_worker_parse_document(
9598
doc_file, project_config
9699
)
97100
assert isinstance(document_or_grammar, SDocDocument)
101+
elif doc_full_path.endswith(".robot.xml"):
102+
robot_reader = RobotOutputXMLReader()
103+
document_or_grammar = robot_reader.read_from_file(
104+
doc_file, project_config
105+
)
98106
else:
99107
raise NotImplementedError # pragma: no cover
100108
drop_textx_meta(document_or_grammar)
@@ -269,7 +277,13 @@ def _build_file_tree(
269277
file_tree_structure = FileFinder.find_files_with_extensions(
270278
root_path=path_to_doc_root,
271279
ignored_dirs=[project_config.output_dir],
272-
extensions=[".sdoc", ".sgra", ".junit.xml", ".gcov.json"],
280+
extensions=[
281+
".sdoc",
282+
".sgra",
283+
".junit.xml",
284+
".gcov.json",
285+
".robot.xml",
286+
],
273287
include_paths=project_config.include_doc_paths,
274288
exclude_paths=project_config.exclude_doc_paths,
275289
)

strictdoc/export/html/generators/view_objects/helpers.py

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -57,7 +57,11 @@ def screen_should_display_file(
5757
assert isinstance(file, File), file
5858
assert traceability_index.document_tree is not None
5959

60-
if file.has_extension(".junit.xml") or file.has_extension(".gcov.json"):
60+
if (
61+
file.has_extension(".junit.xml")
62+
or file.has_extension(".gcov.json")
63+
or file.has_extension(".robot.xml")
64+
):
6165
return True
6266

6367
if file.has_extension(".sdoc"):
Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
[DOCUMENT]
2+
TITLE: Hello world doc
3+
4+
[REQUIREMENT]
5+
UID: REQ-A
6+
TITLE: Feature A
7+
STATEMENT: The system shall do A.
8+
9+
[REQUIREMENT]
10+
UID: REQ-B
11+
TITLE: Feature B
12+
STATEMENT: The system shall do B.

0 commit comments

Comments
 (0)