Bail mode leaves mutation test contribution unmeasurable in two packages #73

labels:
priority: p2
type: bug
area: testing
metadata:
packages: "omp/plugins/omp-claude-compat, packages/effect-daemon-spec"
first_observed_run: "31204982692"

Problem Statement

Two packages fail their mutation run even though they score a perfect 100 and leave no surviving mutants. The run does not fail on quality — it fails because Stryker was in bail mode, which stops each mutant at the first test that kills it, so the reporter can never see which test file did the killing and refuses to certify a measurement it did not take. The result is a red run that says nothing about the tests, and the only obvious ways to clear it are to turn the check off or to slow every run down. Neither is a decision anyone has made deliberately, so today the two packages sit permanently red.

Goal

pnpm --filter <pkg> mutation exits 0 for both omp/plugins/omp-claude-compat and packages/effect-daemon-spec, or exits non-zero naming at least one specific test file that killed no mutant nothing else killed — in neither case reporting that contribution could not be measured at all.

Evidence: CI run 31204982692

From the Mutation (packages/effect-daemon-spec) job log, and identically in Mutation (omp/plugins/omp-claude-compat):

INFO  MutationTestReportHelper Final mutation score of 100.00 is greater than
      or equal to break threshold 100
ERROR MutationTestReportHelper This run used Stryker's bail mode, which stops
      each mutant at its first killing test. A test file's contribution
      therefore cannot be measured on this evidence. Set `disableBail: true`
      to record every killing test, or remove `.workflow.property.test.ts,
      .policy.property.test.ts, .kernel.property.test.ts` from
      `requireTestContribution` (set it to `null` to disable the check)
Setting exit code to 1 (failure).

The machine verdict from the same run:

{"kind":"verdict","score":100,"thresholds":{"high":100,"low":100,"break":100},
 "counts":{"killed":23,"timeout":1,"survived":0,"noCoverage":0,
 "runtimeErrors":0,"compileErrors":23,"ignored":23,"pending":0},
 "testContribution":{"failed":true,"message":"This run used Stryker's bail mode…"}}

Orientation

The check lives in packages/stryker-js/core/src/reporters/test-contribution.ts and is defaulted on for .property.test.ts by requireTestContribution. Per-package settings are in each stryker.config.json. Root AGENTS.md describes the intended semantics: under bail only files that killed nothing at all are accused, disableBail: true buys the exact measure, and a run that credits no kill to any test file is reported as an unmeasurable run rather than as a package full of toothless tests. The 22 other packages in the same matrix run under the same defaults and report ✅, so whatever differs is package-local, not global — identifying that difference is the first useful step.

packages/stryker-js/core is a fork this repository owns and publishes, so the reporter itself is in scope, not only the package configs.

Definitions

• Unmeasurable run — the reporter credited no killing test to any test file, so it cannot distinguish a toothless file from an unobserved one. This is the state both packages are in.
• Accused file — a specific *.property.test.ts the reporter names as killing no mutant that nothing else kills. No file is currently accused; that is the defect.

Non-Counting Outcomes

• Setting requireTestContribution to null in either package purely to turn the red green. That is a visible opt-out the config supports, but using it without a stated package-specific reason converts an unanswered question into a permanent silence.
• Deleting or renaming the *.property.test.ts files so the check has nothing to inspect.
• Enabling disableBail: true without reporting the resulting runtime for those packages. It buys the exact measure at a cost, and the cost is the whole reason bail is on.
• Making the reporter treat an unmeasurable run as a pass. That removes the distinction between "measured and clean" and "never measured", which is the only thing the check contributes.
• Concluding the packages are fine because the score is 100 and no mutants survived. The score measures the mutant set; this check measures the test set, and the two are independent.
• Fixing only one of the two packages and reporting the class as resolved.

Acceptance Criteria

• (gatekeeper) pnpm --filter @systemfsoftware/effect-daemon-spec mutation exits 0, and pnpm --filter @systemfsoftware/omp-claude-compat mutation exits 0.
• Neither package's verdict JSON contains "testContribution":{"failed":true, with a message stating contribution could not be measured.
• A dispatched Mutation workflow run shows ✅ in the merged summary.md verdict column for both packages, with all 24 rows still merged.
• The reason those two packages behave differently from the other 22 is stated in the change, naming the package-local setting or test-file shape responsible.
• If requireTestContribution is disabled or narrowed for either package, the config carries the specific reason, and mutation score stays at 100 with zero survivors.
• If disableBail is enabled anywhere, the change reports the before and after wall time for that package's mutation run.