Make beta runs measure reality: UIScene, a partial-bundle gate, a truncation fault and its canary, drift artifacts (refs #478) - #479

The triage on #478 falsified the working hypothesis. The Swift Testing tests were missing from both backends because they were not in the bundle at all: the sample app is pre-scene UIKit, the iOS 27 SDK promotes a missing scene manifest from a runtime warning to a launch trap, and the app died on launch — taking SampleAppUnitTests with it, since the unit tests are hosted in it. Twelve of twenty-one rows never existed; every downstream failure was arithmetic on that hole. XCResultKit is not implicated.

Four changes, one story: make beta runs measure reality. The beta leg has to measure this project rather than a bundle that was never produced, and nothing in the chain should have let a 10-of-21 bundle through in silence.

Updated after review. The review found a blocking counterexample against one of the two fault kinds originally proposed here. emptyPlannedTestable is withdrawn; systemFailure stays and gains a canary. Item 3 below is rewritten, and Changed after review records what moved and why.

1. The sample app adopts UIScene

A SceneDelegate, a scene manifest naming Main through UISceneStoryboardFile, and UIMainStoryboardFile retired since the manifest supersedes it. The delegate is deliberately empty — UIKit instantiates the storyboard and assigns the window itself, exactly as UIMainStoryboardFile did, so this changes which object receives the lifecycle callbacks, not what the app does. The five app-level lifecycle stubs go with it: scenes deliver those transitions to the scene delegate, so keeping them would leave five methods nothing calls, and they were empty Xcode-template comments.

Behaviour on stable is unchanged, measured rather than asserted. Fixtures regenerated on Xcode 26.2 and the whole suite run on both legs:

DifferentialTests is green on both legs. A direct smoke run of -only-testing:SampleAppUnitTests before and after tells the story most sharply: the console warning UIScene lifecycle will soon be required. Failure to adopt will result in an assert in the future. is gone, and all 12 rows the beta lost (7 XCTest + 5 @Test) run.

The app delegate also gains one deliberately inert branch — XCHR_TRAP_AT_LAUNCH, which nothing but fixture generation sets. Item 3 explains what it is for.

2. verify_fixtures.sh catches partials, not just stubs

totalTestCount > 0 passed a bundle missing 57% of the suite. It could not have done otherwise — ten is a perfectly healthy count for a bundle whose expected count is ten, so no global floor catches this. The floors are now per bundle, in a table at the top of the script with the source of each count documented (21 = 9 UI + 7 XCTest + 5 @Test; 1 and 4 from the -only-testing filters). Floors rather than equalities, so adding a sample test does not redden every branch until the table catches up — losing one still does.

A rejection names the bundle and prints observed-versus-expected, in a message distinct from the stub case, because a partial bundle looks healthy from the outside:

::error::partial fixture: TestResults.xcresult contains 1 tests, expected at least 21 — part of
the suite never ran (a host-app launch failure takes its whole target with it), so this bundle
must not be trusted or cached

Verified against a real partial bundle (rejected, exit 1) and against the healthy fixtures (accepted). pages.yml and pages-release.yml inherit this for free — #470 already routed them through the shared script, and the table keys on the bundle's basename, so their explicit paths hit it.

This is now where whole-target loss is caught. With emptyPlannedTestable withdrawn, these floors are the gate that stops a 10-of-21 bundle, and they catch it one step earlier than the tool would have: before swift test runs and before the bundle can be written to the fixture cache.

3. systemFailure, and a canary for the name it keys on

xchtmlreport ran against the truncated beta bundle, printed Report successfully created, and exited 0. That is exactly what the fault machinery exists to prevent. One new fault kind closes it:

• systemFailure — the group both formats file host-app and system failures under.

It is read off ParsedResult, the model both readers produce, so there is one implementation rather than one per reader — the same seam unresolvedLog uses. That alone is not a guarantee the two behave alike, which is the lesson of the withdrawn second kind; what makes this one agree is that the node it keys on was measured on both readers from the same bundle, and is re-measured on every toolchain.

The shapes were measured, not guessed. The crash is reproduced by trapping in the host app's didFinishLaunchingWithOptions, producing the same SampleApp (<pid>) encountered an error (Early unexpected exit, operation never finished bootstrapping…) the beta hit. Both readers were dumped:

legacy   ActionTestSummaryGroup  name='System Failures'  id='System Failures'
modern   Test Suite              name='System Failures'  id='System Failures'

They stop agreeing one level down — modern carries the crash row, XCResultKit drops it (the node is an ActionTestSummary, not one of the ActionTestMetadata entries subtests decodes). That asymmetry is why the bucket is the marker and its rows are not: keying on the row would have fired only on modern. Against that bundle:

The wording differs and that is not drift: each names the most specific node its own document holds.

The rule fails open, so it gets a canary. The bucket is a plain test suite on both surfaces, indistinguishable from a user's own suite by anything but its name, so detection cannot be made structural today. If Apple renames it, systemFailure silently stops firing and no healthy fixture notices. A recursive search of Xcode.app/Contents finds the literal in no binary and no .strings table — while the crash message Early unexpected exit is in XCTHarness, so the search demonstrably works — meaning the string is composed at run time. That is weak evidence against localization and none at all against a rename.

So prepareTestResults.sh now generates a fourth fixture, CrashResults.xcresult, with the toolchain in front of it: TEST_RUNNER_XCHR_TRAP_AT_LAUNCH=1 (xcodebuild's own channel to the launched process, which for a hosted unit test is the host app) makes the sample app trap at launch and take SampleAppUnitTests with it. That is the #478 failure reproduced, not simulated. SystemFailureCanaryTests then asserts:

• the modern reader still finds a group named System Failures in it;
• the legacy reader does too (skipped, not failed, once a toolchain drops the legacy commands — at that point there is no legacy reader to disarm);
• the tool records exactly one systemFailure naming SampleAppUnitTests on every backend this toolchain can run.

A failing assertion prints every group name the reader did produce, so the failure says what the bucket is called now. Verified by flipping ParsedGroup.systemFailureName to a wrong value: three tests, four assertions, red on both readers, with the found name in the message. The canary reaches a new Xcode before a release does, because toolchain-drift regenerates fixtures and runs the suite on the beta leg.

The cardinal rule holds. A bundle carrying no testable for a target is structural absence — which is precisely what -only-testing produces, and what SanityResults and RetryResults are — and is never a fault. testHealthyFixturesProduceNoTruncationFaultsOnEitherBackend asserts every healthy fixture through every runnable backend reports no fault, and CliTests would fail if a healthy render started exiting 3.

4. The drift workflow uploads its bundles on failure

This triage was done from job logs, because toolchain-drift.yml uploaded nothing and gh run download on the run reported no valid artifacts found. Bundles now upload on any failing signal, named per leg per the #470 convention (xcresult-fixtures-<channel> — v4+ rejects duplicate names, so a run where both legs fail keeps both), at 5 days retention since they are ~50 MB each and this workflow runs twice a week.

Placed before the report step, because that step exits 1 on the stable leg; gated on the explicit outcome list rather than failure(), because every signal above it is continue-on-error and so a bare failure() would never fire. if-no-files-found: ignore, since generation failing outright can leave no bundle at all and this must not add a red step of its own. The filed issue now points at the artifact by name — that pointer matters as much as the upload, since nobody knew there was nothing to download.

Changed after review

emptyPlannedTestable is withdrawn

The review produced a counterexample I reproduced here at a59640e before changing anything. On the unmodified sample app, with stock flags:

xcodebuild test-without-building -project SampleApp.xcodeproj -scheme MainScheme \
  -destination "id=$UDID" -derivedDataPath .derivedData \
  -skip-testing:SampleAppUITests/RetryTests \
  -skip-testing:SampleAppUnitTests/SampleAppUnitTests \
  -skip-testing:SampleAppUnitTests/SwiftTestingSuite \
  -resultBundlePath SkipAllUnit.xcresult

Every test that was planned to run, ran. Measured at a59640e:

Legacy keeps a zero-row Selected tests group for the still-selected unit target; the modern node tree drops the bundle node outright. One implementation, two inputs, two answers — so the kind was observable on one backend only, the one-backend-field shape the model doctrine exists to catch, and the divergence it produced was a false positive on the default reader.

It is not a contrived configuration: quarantining a flaky suite and sharding a target across CI jobs both produce it, and --lenient disables all fault detection, so a user had no way to suppress just this one.

Its residual value was near zero. The #478 partial bundle is caught by the per-bundle floors in item 2 — earlier in the chain, before the cache — and the launch-trap case by systemFailure, which the review confirmed fires on both readers. So the kind, its validate() arm, the now-unused ParsedTestable.allTestCases, its tests and its release-note line are all gone. testSelectedTargetWithNoTestRowsIsNotAFault pins the counterexample so the rule cannot return by accident.

After this change, the same bundle exits 0 on both readers.

The other findings

Kept as-is, all confirmed clean by the review: UIScene adoption, the count floors, systemFailure, the artifact upload.

Deliberately not in this PR

• The .auto preference is untouched. Ruled: legacy was not at fault, and flipping it would have produced the identical 10-row report.
• The modern file:line-on-27 question is not investigated. On Xcode 27 every failure row in a modern-backend report loses its source location, and whether XCTest stopped recording the prefix or xcresulttool stopped emitting it cannot be told apart from CI logs — the failureTitlePrefix allow-list rule strips it from both renders, so the differential is blind to it by construction. It needs a live Xcode 27 bundle, which item 4 above is what makes obtainable: the next scheduled beta run will now attach one. That is the direct linkage between this PR and the follow-up.

Gates

Re-verified on Xcode 26.2 / iPhone 17 Pro Max (iOS 26.2), fixtures regenerated from scratch by the updated script:

• 3 healthy fixtures × 3 readers (auto/legacy/modern) × 2 rendering modes (linking/inline) → exit 0, zero faults, 18/18.
• CrashResults.xcresult → exit 3 on both readers.
• The -skip-testing counterexample → exit 0 on both readers, where legacy gave exit 3 before.
• swift test on auto and on modern: 151 executed, 3 skipped, 0 failures on each.
• verify_fixtures.sh green on all four bundles.
• shellcheck, actionlint, zizmor --min-severity low (no findings), swiftformat --lint (0/88 require formatting), swiftlint (0 serious, no new warnings).

refs #478

🤖 Generated with Claude Code