Skip to content

feat(metrics): add Recovery Step Ratio (RSR) for post-error recovery effort - #6

Merged
lostmartian merged 1 commit into
mainfrom
feat/recovery-step-ratio
Aug 22, 2026
Merged

lostmartian merged 1 commit into
mainfrom
feat/recovery-step-ratio

Conversation

@lostmartian

Copy link
Copy Markdown
Collaborator

What

Adds the Recovery Step Ratio (RSR) metric (roadmap B2): how many successful steps a run spends recovering after ERROR/RETRY/ABANDONED clusters, candidate vs baseline. DiffReport gains baseline_recovery_steps, candidate_recovery_steps, recovery_step_ratio; the gate is opt-in via --max-recovery-ratio / max_recovery_step_ratio.

Why

Roadmap item B2 — context/ROADMAP.md WS-B: "Recovery Step Ratio (steps to recover from error vs baseline)". WEI says a run failed; RSR says how expensive getting back on track was — a run can fail cheaply and recover fast, or burn many steps limping back to the golden path.

How

  • Semantics: a recovery window opens at each wasted-status step; successful steps inside it count; the window closes at the first step re-aligning with the other trace (MATCHED/MODIFIED — TDI's LCS set). Unrecovered tails count fully. Failed steps themselves stay WEI's territory.
  • Ratio: cand/base when baseline needed recovery; raw candidate count when the baseline is clean (nothing to be relative to); 0.0 when both clean.
  • Surfacing: summary(), terminal panel, PR markdown (gate row only when threshold provided → default output unchanged), and a new --explain finding category recovery.
  • Gates: opt-in only (None default) in assert_no_regressions, CLI flag, [cli] max_recovery_ratio, [assertions] max_recovery_step_ratio — zero behavior change for existing users.
  • Additive defaulted report fields → JSON contract stays backward compatible.

Testing

  • 22 new tests in tests/test_recovery_steps.py: metric edge cases (no errors, immediate retry, added-steps-then-rematch, trailing unrecovered, multi-cluster, both sides), ratio cases, comparator integration, JSON round-trip with new fields, summary block, opt-in gate behavior, explanations, config plumbing.
  • Full suite: 227 passed (was 205); make lint + format clean.
  • Manual e2e: CLI run shows Recovery Steps (base/cand): 0 / 2 (RSR 2.00), explanation finding, gate FAILED at --max-recovery-ratio 1.0.

Checklist

  • Exactly one logical change in this PR
  • make lint passes
  • uv run python -m pytest is green (227 passed)
  • Docs updated if user-facing (README metrics table)
  • CHANGELOG.md entry added under [Unreleased]
  • Conventional title

@lostmartian
lostmartian merged commit cd98e04 into main Aug 22, 2026
7 checks passed
@lostmartian
lostmartian deleted the feat/recovery-step-ratio branch August 22, 2026 18:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant