Verified Completions: benchmark suite with CI gate on false alarms #88

Why

A consistency checker that raises false alarms gets switched off within a week, and then the feature is dead. So CI must gate on the false-alarm rate, not just on how many planted contradictions we catch. The design package ships a 43-row benchmark (docs/internals/verified-completions/benchmarks/corrupted-prompt-benchmark.md) covering every rule family, including control rows that must NOT produce findings and declared gaps the system knowingly cannot catch yet.

What to do

• Turn each benchmark row into two fixtures: an engine-only test (hand-written facts, tests the rules) and an end-to-end test (raw conversation text, tests extraction plus rules)
• Add a mutation harness that programmatically corrupts clean conversations: flip a date, swap an entity, negate a claim, create an ordering cycle across turns
• Report three numbers in CI: detection rate per rule family over the flag rows, false-alarm rate over the control rows, and revision fidelity over the correction rows (a correction must retract the old value, not raise a contradiction)
• Block merge on any false-alarm regression. Detection-rate regressions warn; false-alarm regressions fail
• Keep the declared-gap rows in the suite as documentation of what we intentionally do not catch (counting-based contradictions needing aggregation beyond v1, unit conversion, cross-language, and similar)

Done when

CI runs the suite on every PR touching the rule pack, extraction prompts, or gateway ingestion, and prints the three metrics. A control row producing a finding fails the build.

Depends on

• Engine-only fixtures: rule pack validation (Phase 1)
• End-to-end fixtures: /v1/verify (M0)

Roadmap checkpoint — 2026-10-03

R09 should use deterministic facts and recorded extraction to gate detection, false alarms and revision fidelity without paid-model calls in routine verification. Reuse the current chat/turn contract rather than requiring the historical /v1/verify route. A planted control false alarm, stale correction or missing metric must fail the gate.