Skip to content

[junit] Mark test as failed when finding a fatal error when xctest is used directly #412

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

[junit] Mark test as failed when finding a fatal error when xctest is used directly #412

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion Sources/XcbeautifyLib/CaptureGroups.swift
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1751,7 +1751,7 @@ struct FatalErrorCaptureGroup: ErrorCaptureGroup {
/// Regular expression captured groups:
/// $1 = whole error.
/// it varies a lot, not sure if it makes sense to catch everything separately
static let regex = XCRegex(pattern: #"^(fatal error:.*)$"#)
static let regex = XCRegex(pattern: #"^((?:.* )?[fF]atal error:.*)$"#)
Copy link
Author

@marcelofabri marcelofabri May 20, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

let me know if you'd rather have a separate capture group for this

Copy link
Owner

@cpisciotta cpisciotta Oct 30, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking at the example test file... Do you expect the prefix to always be the file / line number? If so, I think we should make a separate group, so we can extract that information.


let wholeError: String

Expand Down
15 changes: 15 additions & 0 deletions Sources/XcbeautifyLib/JunitReporter.swift
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,17 @@ package final class JunitReporter {
case let group as ParallelTestCaseSkippedCaptureGroup:
let testCase = TestCase(classname: group.suite, name: group.testCase, time: group.time, skipped: .init(message: nil))
parallelComponents.append(.testCasePassed(testCase))
case let group as TestCaseStartedCaptureGroup:
components.append(.testCaseStart(suite: group.suite, testName: group.testCase))
case let group as FatalErrorCaptureGroup:
switch components.last {
case let .testCaseStart(suite, testName):
let testCase = TestCase(classname: suite, name: testName, time: nil, failure: .init(message: group.wholeError))
components.append(.failingTest(testCase))
default:
break
}

default:
// Not needed for generating a junit report
return
Expand Down Expand Up @@ -95,6 +106,9 @@ private final class JunitComponentParser {
let .testCasePassed(testCase),
let .skippedTest(testCase):
testCases.append(testCase)

case .testCaseStart:
break
}
}

Expand All @@ -120,6 +134,7 @@ private final class JunitComponentParser {

private enum JunitComponent {
case testSuiteStart(String)
case testCaseStart(suite: String, testName: String)
case failingTest(TestCase)
case testCasePassed(TestCase)
case skippedTest(TestCase)
Expand Down
32 changes: 32 additions & 0 deletions Tests/XcbeautifyLibTests/JunitReporterTests.swift
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -286,6 +286,22 @@ class JunitReporterTests: XCTestCase {
</testsuites>
"""

private let expectedXCTestCrashXml = """
<testsuites name="All tests" tests="7" failures="1">
<testsuite name="DateTests.CalendarDay__Tests" tests="7" failures="1">
<testcase classname="DateTests.CalendarDay__Tests" name="testCalendarDayRawValue" time="0.006" />
<testcase classname="DateTests.CalendarDay__Tests" name="testCreatingCalendarDay" time="0.000" />
<testcase classname="DateTests.CalendarDay__Tests" name="testDateConversion" time="0.000" />
<testcase classname="DateTests.CalendarDay__Tests" name="testDateWithTimeConversion" time="0.000" />
<testcase classname="DateTests.CalendarDay__Tests" name="testDaysSince" time="0.000" />
<testcase classname="DateTests.CalendarDay__Tests" name="testDaysSinceWithDaylightSavings" time="0.000" />
<testcase classname="DateTests.CalendarDay__Tests" name="testToday">
<failure message="DateTests/CalendarDayTests.swift:63: Fatal error: This test is not implemented yet." />
</testcase>
</testsuite>
</testsuites>
"""

func testParallelJunitReport() throws {
let url = try XCTUnwrap(Bundle.module.url(forResource: "ParallelTestLog", withExtension: "txt"))
let parser = Parser()
Expand All @@ -301,4 +317,20 @@ class JunitReporterTests: XCTestCase {
let expectedXml = expectedParallelXml
XCTAssertEqual(xml, expectedXml)
}

func testXCTestCrashJunitReport() throws {
let url = try XCTUnwrap(Bundle.module.url(forResource: "xctest_crash_log", withExtension: "txt"))
let parser = Parser()
let reporter = JunitReporter()

for line in try String(contentsOf: url).components(separatedBy: .newlines) {
if let captureGroup = parser.parse(line: line) {
reporter.add(captureGroup: captureGroup)
}
}
let data = try reporter.generateReport()
let xml = String(data: data, encoding: .utf8)!
let expectedXml = expectedXCTestCrashXml
XCTAssertEqual(xml, expectedXml)
}
}
23 changes: 23 additions & 0 deletions Tests/XcbeautifyLibTests/TestData/xctest_crash_log.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
exec ${PAGER:-/usr/bin/less} "$0" || exit 1
Executing tests from //Libs/Date:DateTests
-----------------------------------------------------------------------------
Existing simulator 'BAZEL_TEST_iPhone 11_18.4' (A75DA946-4A16-48A5-95F2-5E3AC12DF582) state is: booted
Test Suite 'All tests' started at 2025-05-19 18:43:25.150.
Test Suite 'DateTests.xctest' started at 2025-05-19 18:43:25.151.
Test Suite 'CalendarDay__Tests' started at 2025-05-19 18:43:25.151.
Test Case '-[DateTests.CalendarDay__Tests testCalendarDayRawValue]' started.
Test Case '-[DateTests.CalendarDay__Tests testCalendarDayRawValue]' passed (0.006 seconds).
Test Case '-[DateTests.CalendarDay__Tests testCreatingCalendarDay]' started.
Test Case '-[DateTests.CalendarDay__Tests testCreatingCalendarDay]' passed (0.000 seconds).
Test Case '-[DateTests.CalendarDay__Tests testDateConversion]' started.
Test Case '-[DateTests.CalendarDay__Tests testDateConversion]' passed (0.000 seconds).
Test Case '-[DateTests.CalendarDay__Tests testDateWithTimeConversion]' started.
Test Case '-[DateTests.CalendarDay__Tests testDateWithTimeConversion]' passed (0.000 seconds).
Test Case '-[DateTests.CalendarDay__Tests testDaysSince]' started.
Test Case '-[DateTests.CalendarDay__Tests testDaysSince]' passed (0.000 seconds).
Test Case '-[DateTests.CalendarDay__Tests testDaysSinceWithDaylightSavings]' started.
Test Case '-[DateTests.CalendarDay__Tests testDaysSinceWithDaylightSavings]' passed (0.000 seconds).
Test Case '-[DateTests.CalendarDay__Tests testToday]' started.
DateTests/CalendarDayTests.swift:63: Fatal error: This test is not implemented yet.
Child process terminated with signal 5: Trace/BPT trap
error: tests exited with '133'