Add Swift Testing (@Test) fixture and structural assertions - #417
Conversation
Adds SwiftTestingSuite.swift to the sample app (SampleAppUnitTests target) with a passing test, a failing #expect(false) test, and a test carrying a .tags() trait, so a Swift Testing suite is actually exercised in the .xcresult fixtures generated by prepareTestResults.sh. Wiring it in required bumping SampleAppUnitTests' IPHONEOS_DEPLOYMENT_TARGET from 12.0 to 13.0 (Swift Testing's macro expansion needs Actor/#isolation, both iOS 13+); the app and UI test targets are untouched. Investigation (see #393) confirmed Swift Testing results flow through HTML/JUnit/JSON correctly today: status, counts, and failure messages all render as expected, and the tool exits 0 (not degraded). The one gap is that @test display names and .tags() never surface, because this tool depends on xcresulttool's legacy schema, which predates Swift Testing. CoreTests.swift gains testSwiftTestingResultsRendered to lock in the working behavior, and testResultStatusCount's fixed counts are updated for the 3 new sample tests.
|
Warning Review limit reached
Next review available in: 9 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (3)
Comment |
…it (#423) Closes #398. FirstSuite, SecondSuite and ThirdSuite each called XCUIApplication().launch() in setUp with continueAfterFailure = false. When the simulator was slow, the launch failed and took the test with it before its body ran — FirstSuite.testOne is XCTAssert(true) plus an attachment and cannot fail on its own assertion, yet it was observed failing. No test in the sample app touches the app's UI. There are no taps, no element queries, no XCUIApplication use beyond the launch itself. The launch existed only to make these genuine UI tests, on the assumption that the screen recording depended on it. It does not. Verified by controlled experiment: removed the launch from SecondSuite alone and regenerated — the fixture still contained 9 distinct screen recordings, exactly as before. Had SecondSuite lost its recordings the count would have dropped by two. Xcode captures the simulator screen for UI test targets regardless of whether the app runs. So the launches were pure flake risk with no fixture value. Fixture generation also drops from 85.7s to 50.4s locally, a 41% saving, because three suites no longer launch and terminate an app per test method. Whether that carries to CI is for CI to measure — see #412 for why local timing is not evidence here. One honest cost: SanityResults.xcresult shrinks from ~51M to ~140K, because a recording of an idle screen is much smaller than one of a launching app. TestResults still carries 9 recordings and 16 .mp4 references, so .mp4 attachment coverage — the reason the launches were thought necessary — is intact. Verified: swift test passes 23 tests with 1 skipped; TestResults reports All(16) Passed(9) Skipped(1) Failed(6), consistent with the suite after the Swift Testing fixture in #417.
1 participant
Summary
SwiftTestingSuite.swiftto the sample app'sSampleAppUnitTeststarget usingimport Testing: a passing@Test, a failing one (#expect(false)), and one carrying a.tags()trait plus a custom display name — so a real Swift Testing suite is exercised in the.xcresultfixturesprepareTestResults.shgenerates for the whole test suite.SampleApp.xcodeprojrequired bumpingSampleAppUnitTests'IPHONEOS_DEPLOYMENT_TARGETfrom12.0to13.0— Swift Testing's macro expansion needsActor/#isolation, both iOS 13+. The app and UI test targets are untouched.CoreTests.swiftwithtestSwiftTestingResultsRendered, a SwiftSoup structural test (following the existing patterns in that file) asserting the suite renders with correct pass/fail status and failure message.testResultStatusCount's fixed counts (13 → 16 total, failed ≥5 → ≥6) to account for the 3 new sample tests.Why
Per #393, roughly half the README's advertised features have no fixture exercising them — Swift Testing support was flagged as the highest-priority gap since
import Testingappeared nowhere in the repo and the tool had never been run against a Swift Testing bundle.Investigation (posted as a comment on #393) found Swift Testing support works today for the tool's core job: pass/fail status, per-suite/per-test counts, and failure messages all render correctly and consistently in HTML, JUnit, and JSON, and the tool exits 0 (not degraded). The one confirmed gap —
@Testdisplay names and.tags()traits never surface, because this tool reads bundles throughxcresulttool's legacy schema, which predates Swift Testing — is out of scope for this PR; fixing it is a larger design change that needs maintainer input.Test plan
swift test— 23 tests, 1 skipped, 0 failures (was 22/1/0; the newtestSwiftTestingResultsRenderedis the addition)swiftformat --lint/swiftlint lintclean on changed files (two pre-existing warnings elsewhere inCoreTests.swiftare unrelated to this change, confirmed viagit stash)xcodebuild build-for-testing/xcodebuild testagainstMainSchemeon iPhone 17 Pro (iOS 26.2) to confirm the sample app target still builds and runs./.build/debug/xchtmlreportagainst a bundle containingSwiftTestingSuiteand confirmed HTML/JUnit/JSON all show correct status, counts, and failure text (see Close fixture coverage gaps #393 comment for full output)Not merging — per the task, this is left open for maintainer review.
🤖 Generated with Claude Code