Skip to content

ci: make runtime matrix resilient to one-off flaky failures - #4977

Merged
mcollina merged 2 commits into
mainfrom
ci-runtime-flake-resilience
Jul 20, 2026
Merged

mcollina merged 2 commits into
mainfrom
ci-runtime-flake-resilience

Conversation

@mcollina

@mcollina mcollina commented Jul 19, 2026 •

Copy link
Copy Markdown
Member

Problem

The runtime test suite occasionally fails for reasons unrelated to the code under test. The latest instance (run 29703993963, job runtime - ubuntu-24.04 - 24 - main on PR #4976) was a fatal V8 crash in Node.js 24.18.0:

FATAL ERROR: v8::ToLocalChecked Empty MaybeLocal
 3: node::cjs_lexer::Parse(v8::FunctionCallbackInfo<v8::Value> const&)
...
1: cjsPreparseModuleExports (node:internal/modules/esm/translators:432:44)

The crash happened ~3 seconds into worker-scaler-1.test.js, while a runtime worker thread was preparsing CJS modules during app.start(). A fatal V8 error in any worker thread aborts the whole test process, so the file fails with no usable test output. This is nodejs/node#63323 — a race in the native CJS lexer introduced in Node 24.14.0 (when cjs-module-lexer WASM was replaced with the native implementation in nodejs/node#61456), triggered probabilistically by multiple worker threads concurrently doing ESM→CJS preparse. It was fixed by nodejs/node#63885 and released in Node 26.4.0, but the fix is not yet in any 24.x LTS release (24.18.0, which CI uses, was cut on 2026-06-23 from before the backport) — which is exactly why the Node 24 job crashed while 22 (no native lexer) and 26 (fix released) passed. Once the 24.x backport ships, CI picks it up automatically via node-version: 24.

It is not a defect in the test: the same file passed on Node 22/26 and on rerun, and the historical ELU-timing flakiness of this test was already addressed by #4806 and #4904 (recent main failures are in other suites).

The cost of one such crash is currently amplified by CI structure: the runtime matrix uses default fail-fast: true, so a single flaky job cancels the other 29 matrix jobs and the whole matrix has to be rerun manually.

Changes

  • fail-fast: false for the runtime matrix: a single flaky job no longer cancels the healthy ones, and gh run rerun --failed only repeats the job that actually failed.
  • Retry the test step once using the same nick-fields/retry action already used for pnpm install in every job. This automates the current practice of rerunning failed runtime jobs before investigating. Persistent failures still fail the job (and now do so twice, making a real failure easier to distinguish from a flake).
  • Job timeout raised from 30 to 55 minutes to fit a second attempt; each individual attempt is capped at 25 minutes, above the ~20 minutes the slowest (Windows main) suite currently takes.

No test code changes: the worker-scaler tests were already hardened against ELU-timing flakiness, and the V8 crash cannot be mitigated from test code since it kills the entire process.

The runtime test suite occasionally fails with failures unrelated to the
code under test, most recently a fatal V8 crash in Node.js 24.18.0
(v8::ToLocalChecked Empty MaybeLocal in node::cjs_lexer::Parse) while a
runtime worker thread was preparsing CJS modules during startup in
worker-scaler-1.test.js. Such a crash aborts the whole test process and,
with the default fail-fast behavior, cancels all the other 29 matrix
jobs, requiring a manual rerun of the entire matrix.

- Disable fail-fast for the runtime matrix so a single flaky job no
  longer cancels the healthy ones.
- Retry the test step once (same nick-fields/retry action already used
  for pnpm install), automating the current practice of rerunning
  failed jobs before investigating. Persistent failures still fail the
  job.
- Raise the job timeout to accommodate the second attempt; each attempt
  is individually capped at 25 minutes.

Assisted-by: Claude Code:claude-fable-5
Signed-off-by: Matteo Collina <[email protected]>
@socket-security

socket-security Bot commented Jul 19, 2026 •

Copy link
Copy Markdown

Dependency limit exceeded — report not shown.

This pull request scan exceeded the 10,000-dependency limit applied to this scan, so the results are incomplete and may be inaccurate. To avoid reporting false positives, Socket has not posted a report.

Upgrade your plan to raise the dependency limit and get complete reports, or view the partial scan in the dashboard.

Socket is always free for open source. If this is a non-commercial open source project, contact us to request a free Team account.

The default shell of nick-fields/retry on Windows is PowerShell 5.1,
which does not support the && statement separator.

Assisted-by: Claude Code:claude-fable-5
Signed-off-by: Matteo Collina <[email protected]>
@mcollina
mcollina merged commit 2725f87 into main Jul 20, 2026
229 checks passed
@mcollina
mcollina deleted the ci-runtime-flake-resilience branch July 20, 2026 06:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant