Build the sample app once instead of three times by tylervick · Pull Request #422 · XCTestHTMLReport/XCTestHTMLReport

tylervick · 2026-08-10T20:20:29Z

Toward #412. The test job now takes 15m33s, of which 790s (85%) is fixture generation.

The cost is per-invocation build overhead, not test execution

prepareTestResults.sh ran xcodebuild test three times, and that action builds before it runs — so the same target was compiled three times to produce three sets of results. CI timestamps from run 31372309618:

Invocation	What it runs	Duration
TestResults	full suite minus `RetryTests`	364s
Sanity	one single test	196s
Retry	4 tests, 2 iterations	~200s

The middle one is the tell: -only-testing:SampleAppUITests/FirstSuite/testOne runs one test and still costs over three minutes. That is build-and-install overhead.

This builds once with build-for-testing into a shared -derivedDataPath, then runs each pass with test-without-building.

Local timing is not offered as evidence

This machine's DerivedData is warm, so the old script pays almost no rebuild cost here — 85s for all three passes locally, versus 790s on CI. Measured both ways; the new version is 92s locally, i.e. indistinguishable. The saving exists only on a cold runner, so CI has to be the judge. That is the point of this PR being measurable rather than argued.

Verified locally for correctness, not speed

all three fixtures produced
swift test → 23 tests, 1 skipped, 0 failures
SanityResults.xcresult still contains exactly 1 test, 1 passed, so -only-testing is still honoured under test-without-building and SanityTests' assertions hold

An unrelated finding, worth recording on #398

While checking whether this change altered fixture content, SanityResults.xcresult came out at 152K on one run and 51M on another — a 340× swing. I re-ran the old, unmodified script and it also produced 51M, so this change is not the cause.

The difference is a ~30MB screen recording being attached or not. So the sample-suite flakiness (#398) does not merely move a test between pass and fail buckets — it changes whether large attachments exist in the fixture at all. Any future assertion about attachment presence would be flaky for the same reason.

🤖 Generated with Claude Code

Summary by CodeRabbit

Performance
- Improved test execution by building the app once and reusing the build artifacts across functional, sanity, and retry test passes.
Test Infrastructure
- Preserved existing test filters and result bundles for each test pass.
- Added support for reusing generated test data between runs.
Chores
- Excluded generated test data from version control.

Toward #412. Fixture generation is ~85% of the test job, which now takes 15m33s. The script ran `xcodebuild test` three times, and each of those builds before it runs — so the same target was compiled three times to produce three sets of results. CI timestamps show what that costs: the second invocation runs a single test (-only-testing FirstSuite/testOne) and still takes 196 seconds. That is per-invocation build and install overhead, not test execution. Now builds once with build-for-testing into a shared derivedDataPath, then runs each pass with test-without-building against it. Local timing cannot validate this and is not offered as evidence: this machine's DerivedData is already warm, so the old script pays almost no rebuild cost here (85s for all three passes, versus 790s on CI). The saving is real only on a cold runner, so CI has to measure it. Verified locally for correctness rather than speed: all three fixtures are produced, `swift test` passes 23 tests with 1 skipped, and SanityResults still contains exactly one test, so -only-testing is still honoured under test-without-building.

coderabbitai · 2026-08-10T20:20:49Z

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: cfe880ce-69a8-4ed9-98c1-964628b2d220

📥 Commits

Reviewing files that changed from the base of the PR and between 4e297a8 and 29d0928.

📒 Files selected for processing (2)

.gitignore
prepareTestResults.sh

📝 Walkthrough

Walkthrough

The test preparation script now builds the app once with xcodebuild build-for-testing. Functional, sanity, and retry passes reuse the shared .derived-data/ directory through xcodebuild test-without-building. Git ignores the generated directory.

Changes

Test Build Reuse

Layer / File(s)	Summary
Shared build and functional pass `.gitignore`, `prepareTestResults.sh`	The script creates one reusable derived-data directory, builds once, and runs the functional pass without rebuilding. Git ignores the generated directory.
Sanity and retry passes `prepareTestResults.sh`	The sanity and retry passes use the shared derived-data directory with `test-without-building`.

Estimated code review effort: 2 (Simple) | ~10 minutes

Sequence Diagram(s)

sequenceDiagram
  participant prepareTestResults.sh
  participant xcodebuild
  participant SharedDerivedData
  prepareTestResults.sh->>xcodebuild: build-for-testing
  xcodebuild->>SharedDerivedData: store test artifacts
  prepareTestResults.sh->>xcodebuild: run functional tests without building
  prepareTestResults.sh->>xcodebuild: run sanity tests without building
  prepareTestResults.sh->>xcodebuild: run retry tests without building
  xcodebuild->>SharedDerivedData: read shared test artifacts

Possibly related issues

Issue 412 — The change implements the issue objective of building once and reusing artifacts across the three test passes.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly and concisely describes the primary change: building the sample app once instead of three times.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

📝 Generate docstrings

Create stacked PR
Commit on current branch

🧪 Generate unit tests (beta)

Create PR with unit tests
Commit unit tests in branch tylervick/ci-latency

_{Comment @coderabbitai help to get the list of available commands.}

tylervick · 2026-08-10T20:48:18Z

Closing unmerged: measured, and it is not an improvement.

What the measurement showed

	`main`	this branch
Generate fixtures	790s	936s
Total	933s	1086s

main's fixture step varies 533–782s across recent runs, so a single comparison would not be conclusive on its own — but 936s sits above that entire range, and local measurement agreed in direction (85s → 92s).

Why the premise was wrong

The PR assumed the three xcodebuild test invocations were expensive because each rebuilds. Timestamps from this branch's own run disprove that:

20:20:54  build-for-testing starts
20:21:33  1st test pass starts     ← the build took 39 seconds
20:28:20  2nd test pass starts     ← 1st pass: 407s
20:33:13  3rd test pass starts     ← 2nd pass: 293s, for ONE test, with no building

Building is 39 seconds of a ~900 second job. The second pass runs a single test under test-without-building — no compilation at all — and still costs nearly five minutes.

So the per-invocation cost is simulator boot, app install, and test-runner setup/teardown. build-for-testing does not touch any of that; it only added 39s of extra work.

What this means for #412

The target is the number of xcodebuild invocations, not what each one builds. Three invocations each carrying ~200–400s of fixed simulator overhead is the entire cost. Recorded on #412.

tylervick mentioned this pull request Aug 10, 2026

Sample UI tests flake under CI load, making fixtures nondeterministic #398

Closed

tylervick closed this Aug 10, 2026

coderabbitai Bot mentioned this pull request Aug 12, 2026

ci: cache generated test fixtures keyed on toolchain and inputs #436

Merged

Navigation Menu

Build the sample app once instead of three times - #422

Build the sample app once instead of three times#422
tylervick wants to merge 1 commit into
mainfrom
tylervick/ci-latency

tylervick commented Aug 10, 2026 •

edited by coderabbitai Bot

Loading

Uh oh!

coderabbitai Bot commented Aug 10, 2026 •

edited

Loading

Walkthrough

Changes

Sequence Diagram(s)

Uh oh!

tylervick commented Aug 10, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

tylervick commented Aug 10, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

The cost is per-invocation build overhead, not test execution

Local timing is not offered as evidence

Verified locally for correctness, not speed

An unrelated finding, worth recording on #398

Summary by CodeRabbit

Uh oh!

coderabbitai Bot commented Aug 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Sequence Diagram(s)

Uh oh!

tylervick commented Aug 10, 2026

What the measurement showed

Why the premise was wrong

What this means for #412

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

tylervick commented Aug 10, 2026 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented Aug 10, 2026 •

edited

Loading