The test job takes about 10 minutes per run, on every push and every pull request. Measured from a successful run:
| Step |
Duration |
| Set up job |
1s |
actions/checkout@v4 |
3s |
| Setup Xcode version |
0s |
| Generate test fixtures |
519s |
| Run tests |
88s |
| Upload fixtures on failure |
0s |
| Total |
612s (10m12s) |
85% of the job is prepareTestResults.sh. The actual Swift test suite is 88 seconds.
Why it costs that much
prepareTestResults.sh runs three separate xcodebuild test invocations, each of which builds the sample app and drives a simulator:
TestResults.xcresult — the full suite minus RetryTestsSanityResults.xcresult — a single testRetryResults.xcresult — -test-iterations 2 -retry-tests-on-failure
The sample app is rebuilt for each, and simulator boot is paid repeatedly.
Why it matters
Ten minutes is long enough that contributors stop waiting for CI, and long enough that a flaky run (#398) is expensive to retry. This directly undercuts the point of making the suite runnable by outside contributors in the first place.
Options, roughly in order of leverage
- Cache the generated fixtures, keyed on a hash of
XCTestHTMLReportSampleApp/** plus the Xcode version. Most pull requests do not touch the sample app, so fixtures would restore in seconds. Keying on the Xcode version preserves the property that fixtures are regenerated when the toolchain changes — which is the reason they are generated at all.
- Build once, test three times. Replace the three
xcodebuild test calls with one build-for-testing followed by three test-without-building runs, so the sample app is compiled once rather than three times.
- Cache SwiftPM's
.build across runs for the swift test step.
- Boot the simulator once and reuse it across the three invocations rather than paying boot cost repeatedly.
(1) and (2) are independent and compose. (1) alone would take the common case close to the 88-second test time.
Constraint worth preserving
Whatever is done here must not reintroduce the problem that #379 fixed: fixtures must stay generatable by anyone, with no credentials, and must not silently go stale against a new Xcode. A cache keyed on the toolchain version satisfies both — a miss simply regenerates.
The
testjob takes about 10 minutes per run, on every push and every pull request. Measured from a successful run:actions/checkout@v485% of the job is
prepareTestResults.sh. The actual Swift test suite is 88 seconds.Why it costs that much
prepareTestResults.shruns three separatexcodebuild testinvocations, each of which builds the sample app and drives a simulator:TestResults.xcresult— the full suite minusRetryTestsSanityResults.xcresult— a single testRetryResults.xcresult—-test-iterations 2 -retry-tests-on-failureThe sample app is rebuilt for each, and simulator boot is paid repeatedly.
Why it matters
Ten minutes is long enough that contributors stop waiting for CI, and long enough that a flaky run (#398) is expensive to retry. This directly undercuts the point of making the suite runnable by outside contributors in the first place.
Options, roughly in order of leverage
XCTestHTMLReportSampleApp/**plus the Xcode version. Most pull requests do not touch the sample app, so fixtures would restore in seconds. Keying on the Xcode version preserves the property that fixtures are regenerated when the toolchain changes — which is the reason they are generated at all.xcodebuild testcalls with onebuild-for-testingfollowed by threetest-without-buildingruns, so the sample app is compiled once rather than three times..buildacross runs for theswift teststep.(1) and (2) are independent and compose. (1) alone would take the common case close to the 88-second test time.
Constraint worth preserving
Whatever is done here must not reintroduce the problem that #379 fixed: fixtures must stay generatable by anyone, with no credentials, and must not silently go stale against a new Xcode. A cache keyed on the toolchain version satisfies both — a miss simply regenerates.