Skip to content

Parameterized Swift Testing durations diverge across backends (legacy sums the argument executions, modern does not) #477

Description

@tylervick

A parameterized Swift Testing test case reports a different duration on each backend, and the difference is currently hidden by the durations allow-list entry.

Measured

On CI's TestResults.xcresult (downloaded from the failing run of #476 and reproduced locally), exactly one leaf test disagrees:

SwiftTestingSuite/parameterizedAddition(value:)   legacy 0.001268s   modern 0.000423s

Legacy is ~3× modern, and the test has three argument sets. Every other leaf — XCTest cases, the non-parameterized Swift Testing cases, both other fixtures — is bit-identical.

The same measurement on a faster machine gives 0.0 on both backends, which is why it did not appear during #476's local development and only surfaced in CI.

Root cause

This is the arguments form of a problem the migration design already solved for repetitions.

Design answer 8 (docs/superpowers/specs/2026-08-10-xcresulttool-legacy-migration-design.md):

8 | Test-case duration sums repetitions? | Yes — keep today's behaviour | The modern Test Case node reports its own duration, which is not the sum … Summing in the renderer makes both backends agree by construction.

TestCase.duration is iterations.reduce(0) { $0 + $1.duration }, so repetitions converge by construction. Answer 6 added ParsedTestCase.arguments at the same time and noted it was "unexercised by fixtures" — so nobody applied answer 8's treatment to it. It is exercised now: legacy merges a parameterized case's argument executions into iterations and therefore sums them; the modern reader reports the parameterized node's own durationInSeconds and does not sum its Arguments children.

Candidate fix

Have the modern reader sum the per-argument execution durations for a parameterized case, if xcresulttool's Arguments nodes expose them — the answer-8 move, converging the two backends by construction rather than by mask.

If the format does not expose per-argument durations, the alternative is the other direction the design prefers: hold legacy down to modern's capability, i.e. stop summing for the arguments case. Either way the divergence goes away instead of being masked.

Why it needs an issue rather than living under the mask

The spec is explicit that a mask is the worse of two outcomes:

Direction of the win: each answer either removes a field from the port or an entry from the differential allow-list. Deliberately holding the legacy backend down to the modern backend's capability makes the two agree, and an unmasked diff proves more than a masked one. The cost is one-way and honest … rather than a permanent asymmetry hidden behind a mask.

and differential-allowlist.json says an entry means "accepting a permanent difference between the backends — a design decision, not a way to quiet a failing test." No entry is being added here — this divergence already falls under the existing durations rule, and has since the parameterized fixture landed — but #476's summary header is the first place it aggregates into a single number, which is what made it visible. Filing it so the mask is a known debt with an owner rather than a rug.

When this is fixed

Two things come out, both named in the code:

  • the exclusion in DifferentialTests.testSummaryHeaderNumbersAgreeAcrossBackends, which today skips this one identifier from an otherwise exact per-leaf duration comparison;
  • the paragraph in Sources/XCTestHTMLReportCore/Classes/Models/RunSummary.swift explaining why the header's total is written parenthesised.

The header's duration can then be written in any shape, because it would no longer need the durations rule to cover it.

Found while implementing #439's summary header (#476).

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions