Reject stub fixture bundles at the verify gate (fixes #454) - #470

…ation faults, drift artifacts (refs #478)

The Xcode 27 beta leg reported eight differential failures and a red suite. None
of it was a reader regression. The sample app is pre-scene UIKit, the iOS 27 SDK
promotes a missing scene manifest from a runtime warning to a launch trap, and
the app died on launch -- taking `SampleAppUnitTests` with it, since the unit
tests are hosted in it. Twelve of the fixture's twenty-one rows never existed,
and every downstream failure was arithmetic on that hole.

Four changes, because they are one story: the beta leg has to measure this
project rather than a bundle that was never produced, and nothing in the chain
should have let a 10-of-21 bundle through in silence.

**The sample app adopts UIScene.** A `SceneDelegate`, a scene manifest naming
`Main` through `UISceneStoryboardFile`, and `UIMainStoryboardFile` retired since
it is superseded. The five app-level lifecycle stubs go with it: scenes deliver
those transitions to the scene delegate, so keeping them would leave five
methods nothing calls, and they were empty template comments. Fixtures
regenerated on stable and the suite run on both legs: 21/1/4 test counts,
byte-for-byte the same shape as before the change, 150 tests green on `auto` and
on `modern`, `DifferentialTests` included.

**The verify gate learns each bundle's shape.** `totalTestCount > 0` passed a
bundle missing 57% of the suite -- ten is a healthy count for a bundle whose
expected count is ten, so no global floor can catch it. The floors are now per
bundle (21/1/4, with the sources for each documented at the table), a rejection
names the bundle and prints observed-versus-expected, and a bundle nobody
declared keeps the old `> 0` rule. `pages.yml` and `pages-release.yml` inherit
this for free; both already call the shared script.

**Two new faults, on both backends by construction.** `xchtmlreport` ran against
the truncated beta bundle, printed "Report successfully created", and exited 0.
Now `emptyPlannedTestable` fires on a test target present with no test rows
beneath it, and `systemFailure` on the `System Failures` group. Both are read
off `ParsedResult` -- the model both readers produce -- so there is one
implementation and it cannot drift between backends.

The shapes were measured, not guessed. Reproducing the crash on Xcode 26.2 by
trapping in the host app's `didFinishLaunchingWithOptions` yields a bundle both
readers agree on: a group named and identified `System Failures` sitting
directly under the testable. They stop agreeing one level down -- modern carries
the crash row, XCResultKit drops it (an `ActionTestSummary` is not one of the
`ActionTestMetadata` entries `subtests` decodes) -- which is exactly why the
bucket is the marker and its rows are not. Against that bundle the tool now
exits 3 on both readers, where it exited 0 on both before.

The cardinal rule holds: a bundle carrying no testable for a target is
structural absence, which is what `-only-testing` produces and what
`SanityResults` and `RetryResults` are, and is never a fault. Verified directly
-- every fixture, through every backend the toolchain runs, reports neither
fault.

**The drift workflow uploads its bundles.** This triage was done from job logs
because `toolchain-drift.yml` uploaded nothing and `gh run download` found no
artifacts, and the one open question -- whether XCTest stopped recording
`file:line` in failure titles on 27 or `xcresulttool` stopped emitting it --
cannot be answered without a bundle. Bundles now upload on any failing signal,
named per leg per the #470 convention, at five days' retention since they are
~50 MB and this runs twice a week. The filed issue points at them, which matters
as much as the upload: nobody knew there was nothing to download.

Not touched: the `.auto` preference, since legacy was not at fault and flipping
it would have produced the identical 10-row report.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

…ncation fault and its canary, drift artifacts (refs #478) (#479)

* Make beta runs measure reality: UIScene, a partial-bundle gate, truncation faults, drift artifacts (refs #478)

The Xcode 27 beta leg reported eight differential failures and a red suite. None
of it was a reader regression. The sample app is pre-scene UIKit, the iOS 27 SDK
promotes a missing scene manifest from a runtime warning to a launch trap, and
the app died on launch -- taking `SampleAppUnitTests` with it, since the unit
tests are hosted in it. Twelve of the fixture's twenty-one rows never existed,
and every downstream failure was arithmetic on that hole.

Four changes, because they are one story: the beta leg has to measure this
project rather than a bundle that was never produced, and nothing in the chain
should have let a 10-of-21 bundle through in silence.

**The sample app adopts UIScene.** A `SceneDelegate`, a scene manifest naming
`Main` through `UISceneStoryboardFile`, and `UIMainStoryboardFile` retired since
it is superseded. The five app-level lifecycle stubs go with it: scenes deliver
those transitions to the scene delegate, so keeping them would leave five
methods nothing calls, and they were empty template comments. Fixtures
regenerated on stable and the suite run on both legs: 21/1/4 test counts,
byte-for-byte the same shape as before the change, 150 tests green on `auto` and
on `modern`, `DifferentialTests` included.

**The verify gate learns each bundle's shape.** `totalTestCount > 0` passed a
bundle missing 57% of the suite -- ten is a healthy count for a bundle whose
expected count is ten, so no global floor can catch it. The floors are now per
bundle (21/1/4, with the sources for each documented at the table), a rejection
names the bundle and prints observed-versus-expected, and a bundle nobody
declared keeps the old `> 0` rule. `pages.yml` and `pages-release.yml` inherit
this for free; both already call the shared script.

**Two new faults, on both backends by construction.** `xchtmlreport` ran against
the truncated beta bundle, printed "Report successfully created", and exited 0.
Now `emptyPlannedTestable` fires on a test target present with no test rows
beneath it, and `systemFailure` on the `System Failures` group. Both are read
off `ParsedResult` -- the model both readers produce -- so there is one
implementation and it cannot drift between backends.

The shapes were measured, not guessed. Reproducing the crash on Xcode 26.2 by
trapping in the host app's `didFinishLaunchingWithOptions` yields a bundle both
readers agree on: a group named and identified `System Failures` sitting
directly under the testable. They stop agreeing one level down -- modern carries
the crash row, XCResultKit drops it (an `ActionTestSummary` is not one of the
`ActionTestMetadata` entries `subtests` decodes) -- which is exactly why the
bucket is the marker and its rows are not. Against that bundle the tool now
exits 3 on both readers, where it exited 0 on both before.

The cardinal rule holds: a bundle carrying no testable for a target is
structural absence, which is what `-only-testing` produces and what
`SanityResults` and `RetryResults` are, and is never a fault. Verified directly
-- every fixture, through every backend the toolchain runs, reports neither
fault.

**The drift workflow uploads its bundles.** This triage was done from job logs
because `toolchain-drift.yml` uploaded nothing and `gh run download` found no
artifacts, and the one open question -- whether XCTest stopped recording
`file:line` in failure titles on 27 or `xcresulttool` stopped emitting it --
cannot be answered without a bundle. Bundles now upload on any failing signal,
named per leg per the #470 convention, at five days' retention since they are
~50 MB and this runs twice a week. The filed issue points at them, which matters
as much as the upload: nobody knew there was nothing to download.

Not touched: the `.auto` preference, since legacy was not at fault and flipping
it would have produced the identical 10-row report.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

* Withdraw emptyPlannedTestable and give systemFailure a canary (refs #478)

Review of #479 produced a counterexample I reproduced here before touching
anything: on an unmodified sample app,

    xcodebuild test-without-building ... \
      -skip-testing:SampleAppUITests/RetryTests \
      -skip-testing:SampleAppUnitTests/SampleAppUnitTests \
      -skip-testing:SampleAppUnitTests/SwiftTestingSuite

produces a bundle where every test that was planned to run ran -- and at
a59640e `xchtmlreport` exits 3 on it under the legacy reader and 0 under
modern. Legacy keeps a zero-row `Selected tests` group for the still-selected
unit target; the modern node tree drops the bundle node entirely. One
implementation, two inputs, two answers.

**`emptyPlannedTestable` is withdrawn.** Being read off the shared model made
it one implementation, which is not the same as one behaviour: the structure
underneath differs between backends, so the rule was observable on legacy only
-- the one-backend field the model doctrine exists to catch -- and on the
default reader it faulted ordinary practice. Quarantining a flaky suite and
sharding a target across CI jobs both produce that shape, and `--lenient` is
all-or-nothing, so a user had no way to suppress just this one. Its residual
value was near zero: the #478 partial bundle is caught by the per-bundle floors
in `verify_fixtures.sh`, and the launch trap by `systemFailure`, which fires on
both readers. Gone with it: the kind, its `validate()` arm, the now-unused
`ParsedTestable.allTestCases`, its tests, and its release-note claim.

`testSelectedTargetWithNoTestRowsIsNotAFault` pins the counterexample so the
rule cannot come back by accident.

**`systemFailure` gets the canary the review asked for.** It keys on a display
name, and the review confirmed there is no structural alternative on either
surface: modern's node is `{"name": "System Failures", "nodeType": "Test
Suite"}`, legacy's an ordinary `ActionTestSummaryGroup`. So it fails open -- an
Xcode that renames the bucket disarms the fault, healthy fixtures keep passing,
and nothing reddens. A recursive search of `Xcode.app/Contents` finds the
literal in no binary and no `.strings` table (the crash message `Early
unexpected exit` is in `XCTHarness`, so the search does work), which says the
string is composed at run time -- weak evidence against localization, none at
all against a rename.

So the name is re-measured rather than trusted. `prepareTestResults.sh` now
generates a fourth fixture, `CrashResults.xcresult`, by setting
`TEST_RUNNER_XCHR_TRAP_AT_LAUNCH` -- xcodebuild's own channel to the launched
process, which for a hosted unit test is the host app -- and the sample app's
`AppDelegate` traps in `didFinishLaunchingWithOptions`. `SampleAppUnitTests` is
hosted in it and dies with it: the #478 failure reproduced, not simulated.
`SystemFailureCanaryTests` then asserts both readers still find the bucket by
name in that bundle and that the tool faults on both, and names every group it
did find when it fails, so the failure says what Apple renamed it to. Verified
by flipping the constant: three tests, four assertions, red on both readers.

The canary reaches a new toolchain before a release does, because
`toolchain-drift` regenerates fixtures and runs the suite on the beta leg.

**Also from the review.** The release note no longer says the two readers
"behave identically"; the rule is implemented once, and the two still word the
fault differently because each names the most specific node its own document
holds. The `AppDelegate` scene-configuration comment claimed a rename of
`UISceneConfigurationName` "stops compiling" -- the opposite of what happens:
the name is matched at run time, an unmatched configuration has no delegate
class and no storyboard, and the app comes up as a blank window. The drift
upload comment now says it reads the `failure` subset of the six signals
`Report drift` reads, not the same set. The floor table's comment now admits it
keys on basename alone.

Re-verified on Xcode 26.2 / iPhone 17 Pro Max (iOS 26.2), fixtures regenerated
from scratch by the updated script:

- 3 healthy fixtures x 3 readers x 2 rendering modes -> exit 0, zero faults,
  18/18.
- `CrashResults.xcresult` -> exit 3 on both readers (legacy names the bucket,
  modern the crash row).
- The `-skip-testing` counterexample -> exit 0 on both readers, where legacy
  gave exit 3 before.
- `swift test` on `auto` and on `modern`: 151 tests, 3 skipped, 0 failures.
- `verify_fixtures.sh` green on all four bundles; shellcheck, actionlint,
  zizmor `--min-severity low`, swiftformat `--lint` (0/88) and swiftlint
  (0 serious, no new warnings) all clean.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

---------

Co-authored-by: Claude Opus 5 (1M context) <[email protected]>