Skip to content

feat(engine): statistical envelopes — N-run baselines and topological equivalence - #36

Merged
lostmartian merged 3 commits into
mainfrom
feat/statistical-envelope
Aug 31, 2026
Merged

lostmartian merged 3 commits into
mainfrom
feat/statistical-envelope

Conversation

@lostmartian

Copy link
Copy Markdown
Collaborator

What

Pillar 1 of 0.5.0. Baselines can capture N ≥ 2 runs (record --runs 5) into a versioned agentdiff_baseline_envelope artifact (schema 2.0.0, additive — agent_trace.schema.json untouched per D2). Statistical compare judges a candidate against normal variance: min-TDI-of-N alignment (D1), step-count band (|drift| ≤ k·sigma), envelope-relative cost ceiling (max of the relative cap and the variance band), divergence ceiling. Also: topological equivalence — independent same-work reorders merge to matched_commutative with zero TDI penalty — and [scenario.<name>] config sections per the PRD's v0.5 TOML spec.

Why

The #1 adoption killer of v0.3/0.4 (PRD §1): a rigid LCS match against one static baseline.json fails builds on harmless variance — tool-order shifts, ±1 steps, jitter. Agents are non-deterministic; the gate must model that. This is the load-bearing pillar for the release's success metric (≥3 teams, zero false-positive flakes).

How

  • models/envelope.py: BaselineEnvelope + StatBand + compute_bands() (step count, tokens, cost, latency, per-tool tool:<name> bands); cached envelope block is recomputed from runs on load, never trusted
  • loader.load_baseline(): v1 bare traces wrap as envelopes with N=1, mode strict — full back-compat; N<2 statistical envelopes demote to strict
  • engine/comparator.compare_envelope(): aligns against every run, keeps the best explanation, evaluates band gates + reuses evaluate_gate for path gates and hard invariants
  • engine/aligner.mark_commutative_swaps(): pairs REMOVED/ADDED by signature — requires identical inputs/outcomes, a genuine positional swap (matched steps between), and data-independence in both parent graphs (property-guarded); otherwise untouched
  • config.py: [scenario.x] with nested hard_invariants/tolerances (ScenarioConfig); sole scenario needs no --scenario flag
  • CLI: envelope baselines take the statistical path; --update-baseline rotates a rolling window of sample_runs and refreshes bands; record --runs N writes envelopes
  • New status matched_commutative renders as "reordered" in terminal/JSON/markdown/PR/tree; excluded from culprit/first-divergence logic

Testing

  • 26 new tests (test_statistical_envelope.py): bands math, v1/v2 loading + demotion, band/ceiling/divergence/loop gates, min-TDI-of-N, commutative vs dependent swaps, arg-change guard, scenario config, CLI envelope flow + rotation + record --runs
  • Full suite 401 green; make lint clean; uv build green
  • Benchmarks (I2b): N=3/N=5 at 100/500/1000 steps — cost linear in N (N=5 @1000 ≈ 7.4s; KB-scale traces are ms)

Checklist

  • make lint + full suite green
  • CHANGELOG [Unreleased] entries
  • Schema additive + migration note updated (context/schema_migration.md)
  • No non-goal violations (deterministic, local-first, no LLM judges)

Links to context/ROADMAP.md Phase M / 0.5.0 (SPEC-0.5.0 Pillar 1). Stacked on #32.

…illar 2)

Severity-aware gate evaluation shared by CLI, assertions, and suites:
HARD violations block CI (exit 1); SOFT warnings render everywhere but
never flip the exit code. New cyclical-tool-loop invariant (identical
inputs + stagnant outputs, non-consecutive included, on by default) and
opt-in max-tool-repeats cap. Path drift renders as a non-blocking note.
Provenance (G7) and threshold flagging (G6) cover the new knobs.
…, commutative equivalence (Pillar 1)

Baselines capture N >= 2 runs into a versioned envelope artifact
(schema 2.0.0, additive — AgentTrace schema untouched). Statistical
compare judges a candidate against normal variance: min-TDI-of-N
alignment, step-count and cost bands (mean ± k·sigma, cost ceiling =
max of relative cap and variance band), divergence ceiling. Hard
invariants flow through. v1 single-trace baselines load as strict
envelopes — full back-compat.

Topological equivalence: independent same-work reorders ([A->B] vs
[B->A]) merge to matched_commutative with zero TDI penalty; dependent
swaps, changed args, and changed outcomes stay real divergence.

[scenario.*] config (PRD v0.5 spec), record --runs N, rolling-window
envelope rotation, N=3/5 benchmarks (cost linear in N).
@lostmartian
lostmartian merged commit 935e10c into main Aug 31, 2026
7 checks passed
@lostmartian
lostmartian deleted the feat/statistical-envelope branch August 31, 2026 18:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant