Skip to content

Commit 7ab80bc

Browse files
okwasniewskiokwasniewski
authored
fix(report)!: count interrupted tests apart from failures (tester-army#713)
* fix(report)!: count interrupted tests apart from failures * fix(report): tell a failed test from its last failed attempt, whatever the cut retry left
1 parent f7c0756 commit 7ab80bc

35 files changed

Lines changed: 519 additions & 100 deletions

‎.changeset/forced-interrupt-file-reporters.md‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
---
2+
'e2e': patch
3+
---
4+
5+
A forced interrupt (a second Ctrl-C, or a CI cancel that sends SIGINT and then SIGTERM) still writes `junit.xml` and `summary.md`. The runner used to abandon them along with every other reporter, which left the previous run's files beside the new `report.json`. Custom reporters are still abandoned on force.

‎.changeset/report-interrupted-count.md‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
---
2+
'e2e': minor
3+
'@e2e-dev/github': patch
4+
---
5+
6+
Breaking: an interrupted test is no longer counted as failed. `report.json` gains `run.summary.interrupted`, and `run.summary.failed` counts only failed and timed-out tests. `run.summary.skipped` now counts only selected tests, so `passed + failed + interrupted + flaky + skipped` equals `selected`; the tests a filter left out are `discovered - selected`. The `list` reporter prints `3 interrupted` in its own column, `summary.md` shows them with ⏹️ and gives them no failure block or page, and `junit.xml` writes each one as a `<skipped>` whose message starts `interrupted:`. A test that failed and whose retry an interrupt cut short stays failed, and its failure is the one reported. A `--repeat-each` run the interrupt stopped is listed as interrupted, not as a flake. The telemetry event gains `tests_interrupted`. In `@e2e-dev/github`, a test that was interrupted and then passed on a `--last-failed` rerun shows as passed, not flaky.

‎docs/ci.mdx‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -193,7 +193,7 @@ flaky, so you can identify flows that need investigation.
193193
`--max-failures <n>` stops the run once that many tests have failed, so a
194194
broken deploy does not spend a model call on every remaining agent step. The
195195
tests not started are skipped with cause `failure-limit`, the ones running
196-
end as `interrupted`, and the job exits 1.
196+
end as `interrupted`, counted apart from the failures, and the job exits 1.
197197

198198
## Exit codes
199199

‎docs/github.mdx‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -143,9 +143,11 @@ passes:
143143
144144
The comment covers the whole suite, not only the tests the rerun ran again.
145145
A test that failed and then passed shows as flaky, a test that failed twice
146-
stays failed, and the rest keep their first-pass results. The comment
146+
stays failed, one that was interrupted and then passed shows as passed, and
147+
the rest keep their first-pass results. The comment
147148
matches the job's status: it stays red when another filter kept a failed
148-
test out of the rerun.
149+
test out of the rerun, and shows ⏹️ when it kept out a test the first pass
150+
never finished.
149151
150152
## Matrix jobs
151153

‎docs/reference/cli.mdx‎

Lines changed: 16 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -479,8 +479,8 @@ bunx e2e run --reporter list,junit
479479
| --- | --- |
480480
| `list` | Terminal progress, failures with code frames, and a run summary. On a TTY, live progress includes the current agent step, model turns, tool calls, and reasoning excerpts. Untrusted fields are capped at 8 KiB and stripped of control characters. |
481481
| `json` | The report document to stdout and nothing else there, so `e2e run --reporter json \| jq` works. Same document as `<output>/report.json`, described by `schema/report-v1.schema.json` in the package. The shape changes only with the `schemaVersion` field (`report-1` today). Cannot be combined with `list`. |
482-
| `junit` | `junit.xml` beside the report. One `<testsuite>` per file, one `<testcase>` per test-target pair named `title > path [target]`. A test-class error is a `<failure>`, any other an `<error>`, a skip a `<skipped>` with its reason. Run-level errors form a `run` suite. |
483-
| `markdown` | `summary.md` beside the report with counts, cost, errors, and failure details. Failed and flaky tests also get a page under `failures/` there, with steps, model turns, and the screen at failure. Exploration runs show findings. [`@e2e-dev/github`](/github) uses this summary for pull request comments. |
482+
| `junit` | `junit.xml` beside the report. One `<testsuite>` per file, one `<testcase>` per test-target pair named `title > path [target]`. A test-class error is a `<failure>`, any other an `<error>`, a skip a `<skipped>` with its reason. An interrupted test reached no verdict, so it is a `<skipped>` whose message starts `interrupted:`. Run-level errors form a `run` suite. |
483+
| `markdown` | `summary.md` beside the report with counts, cost, errors, and failure details. Failed and flaky tests also get a page under `failures/` there, with steps, model turns, and the screen at failure. Interrupted tests are counted on their own and get no page. Exploration runs show findings. [`@e2e-dev/github`](/github) uses this summary for pull request comments. |
484484

485485
Every run that gets to its tests writes `report.json` under the [output directory](/reference/config#output)
486486
(`.e2e` by default) whatever the reporters are, and the reporters' files
@@ -490,6 +490,16 @@ fails to start, an interrupt) writes none of them and leaves the previous
490490
run's files in place; its errors are on the terminal and in the exit code. A `junit.xml` or `summary.md` that could not be written is a
491491
line on stderr, never a run error.
492492

493+
`run.summary` in the report counts the results. `discovered` is every
494+
test-target pair the runner collected and `selected` the ones this run
495+
chose; the selection's left-out pairs are in `results` as `selected: false`
496+
but in no other count. `passed`, `failed` (timed out included),
497+
`interrupted`, `flaky`, and `skipped` count the selected results and add up
498+
to `selected`. `interrupted` is a test the run stopped before its verdict
499+
(a signal, `--max-failures`, or a run-level error such as
500+
`MODEL_UNAVAILABLE`), never a failure. `executed` is the selected
501+
results that ran at least one attempt.
502+
493503
With `--ai-trace`, the run also writes `<output>/ai-trace.json`: one entry per
494504
model round trip with the exact prompt, tool definitions, response, usage,
495505
and provider metadata. Observations in it are already redacted. Inline image
@@ -510,7 +520,10 @@ Signals escalate.
510520
any test starts (during collection, an engine's `prepare`, or app
511521
startup) writes no report and leaves the previous one in place.
512522
2. A second signal forces. Workers dispose their engines at once and are
513-
killed when the cleanup budget is spent. The report is still written.
523+
killed when the cleanup budget is spent. The report is still written,
524+
and so are `junit.xml` and `summary.md` when the `junit` and
525+
`markdown` reporters are on; a custom reporter's
526+
`onRunFinished` is abandoned.
514527
3. A third signal exits on the spot, after killing the process groups of every
515528
app command the run started. A process attached through
516529
`reuseExisting` is left alone.

‎docs/reference/reporters.mdx‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -171,7 +171,7 @@ prints. All reporters run concurrently, each with a one-minute deadline.
171171
A timeout, exception, or invalid return value is reported on stderr without
172172
changing the run's status, exit code, or report.
173173

174-
One Ctrl-C still lets reporters finish. A second Ctrl-C stops waiting for them.
174+
One Ctrl-C still lets reporters finish. A second Ctrl-C stops waiting for them, except the built-in `junit` and `markdown`, which still write their files so none of the previous run's is left beside the new report.
175175

176176
The second argument is an `AbortSignal` that aborts when the budget runs out
177177
or the run is forced to stop. Hand it to every request and timer, or an

‎docs/telemetry.mdx‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -119,7 +119,7 @@ Versions:
119119
| `command` | `run` or `explore` |
120120
| `status`, `exit_code`, `duration_ms` | The run's outcome |
121121
| `flags` | The names of the flags given to the run command, never their values |
122-
| `tests_discovered`, `tests_selected`, `tests_executed`, `tests_passed`, `tests_failed`, `tests_flaky`, `tests_skipped` | The summary counts |
122+
| `tests_discovered`, `tests_selected`, `tests_executed`, `tests_passed`, `tests_failed`, `tests_interrupted`, `tests_flaky`, `tests_skipped` | The summary counts |
123123
| `attempts_total`, `tests_retried` | How many attempts the run made across tests and serial groups, and how many of them needed more than one |
124124
| `targets`, `platforms`, `engines` | How many targets, on which platforms, and with which engines and versions, as your config declares them: `web`, `ios`, `[email protected]`, or your own engine's name and version. A name that is not a plain token of letters, digits, dots, dashes, and underscores is reported as `other` |
125125
| `steps_total`, `steps_agent`, `steps_locator`, `steps_assertion`, `steps_screen`, `steps_app`, `steps_session`, `steps_resource` | Step counts by kind |

‎packages/e2e/schema/fixtures/report-v1.invalid.json‎

Lines changed: 11 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -81,6 +81,16 @@
8181
"targetId": "web"
8282
}
8383
}
84-
]
84+
],
85+
"summary": {
86+
"discovered": 1,
87+
"selected": 1,
88+
"executed": 1,
89+
"passed": 0,
90+
"failed": 0,
91+
"interrupted": -1,
92+
"flaky": 0,
93+
"skipped": 0
94+
}
8595
}
8696
}

‎packages/e2e/schema/fixtures/report-v1.valid.json‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -369,6 +369,7 @@
369369
"executed": 2,
370370
"passed": 1,
371371
"failed": 0,
372+
"interrupted": 0,
372373
"flaky": 0,
373374
"skipped": 1
374375
},

‎packages/e2e/schema/report-v1.schema.json‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -570,13 +570,14 @@
570570
"summary": {
571571
"type": "object",
572572
"additionalProperties": false,
573-
"required": ["discovered", "selected", "executed", "passed", "failed", "flaky", "skipped"],
573+
"required": ["discovered", "selected", "executed", "passed", "failed", "interrupted", "flaky", "skipped"],
574574
"properties": {
575575
"discovered": { "type": "integer", "minimum": 0 },
576576
"selected": { "type": "integer", "minimum": 0 },
577577
"executed": { "type": "integer", "minimum": 0 },
578578
"passed": { "type": "integer", "minimum": 0 },
579579
"failed": { "type": "integer", "minimum": 0 },
580+
"interrupted": { "type": "integer", "minimum": 0 },
580581
"flaky": { "type": "integer", "minimum": 0 },
581582
"skipped": { "type": "integer", "minimum": 0 }
582583
}

0 commit comments

Comments
 (0)