feat(engine): statistical envelopes — N-run baselines and topological equivalence - #36
Merged
Merged
Conversation
…illar 2) Severity-aware gate evaluation shared by CLI, assertions, and suites: HARD violations block CI (exit 1); SOFT warnings render everywhere but never flip the exit code. New cyclical-tool-loop invariant (identical inputs + stagnant outputs, non-consecutive included, on by default) and opt-in max-tool-repeats cap. Path drift renders as a non-blocking note. Provenance (G7) and threshold flagging (G6) cover the new knobs.
…, commutative equivalence (Pillar 1) Baselines capture N >= 2 runs into a versioned envelope artifact (schema 2.0.0, additive — AgentTrace schema untouched). Statistical compare judges a candidate against normal variance: min-TDI-of-N alignment, step-count and cost bands (mean ± k·sigma, cost ceiling = max of relative cap and variance band), divergence ceiling. Hard invariants flow through. v1 single-trace baselines load as strict envelopes — full back-compat. Topological equivalence: independent same-work reorders ([A->B] vs [B->A]) merge to matched_commutative with zero TDI penalty; dependent swaps, changed args, and changed outcomes stay real divergence. [scenario.*] config (PRD v0.5 spec), record --runs N, rolling-window envelope rotation, N=3/5 benchmarks (cost linear in N).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Pillar 1 of 0.5.0. Baselines can capture N ≥ 2 runs (
record --runs 5) into a versionedagentdiff_baseline_envelopeartifact (schema 2.0.0, additive —agent_trace.schema.jsonuntouched per D2). Statistical compare judges a candidate against normal variance: min-TDI-of-N alignment (D1), step-count band (|drift| ≤ k·sigma), envelope-relative cost ceiling (max of the relative cap and the variance band), divergence ceiling. Also: topological equivalence — independent same-work reorders merge tomatched_commutativewith zero TDI penalty — and[scenario.<name>]config sections per the PRD's v0.5 TOML spec.Why
The #1 adoption killer of v0.3/0.4 (PRD §1): a rigid LCS match against one static
baseline.jsonfails builds on harmless variance — tool-order shifts, ±1 steps, jitter. Agents are non-deterministic; the gate must model that. This is the load-bearing pillar for the release's success metric (≥3 teams, zero false-positive flakes).How
models/envelope.py:BaselineEnvelope+StatBand+compute_bands()(step count, tokens, cost, latency, per-tooltool:<name>bands); cachedenvelopeblock is recomputed from runs on load, never trustedloader.load_baseline(): v1 bare traces wrap as envelopes with N=1, modestrict— full back-compat; N<2 statistical envelopes demote to strictengine/comparator.compare_envelope(): aligns against every run, keeps the best explanation, evaluates band gates + reusesevaluate_gatefor path gates and hard invariantsengine/aligner.mark_commutative_swaps(): pairs REMOVED/ADDED by signature — requires identical inputs/outcomes, a genuine positional swap (matched steps between), and data-independence in both parent graphs (property-guarded); otherwise untouchedconfig.py:[scenario.x]with nestedhard_invariants/tolerances(ScenarioConfig); sole scenario needs no--scenarioflag--update-baselinerotates a rolling window ofsample_runsand refreshes bands;record --runs Nwrites envelopesmatched_commutativerenders as "reordered" in terminal/JSON/markdown/PR/tree; excluded from culprit/first-divergence logicTesting
test_statistical_envelope.py): bands math, v1/v2 loading + demotion, band/ceiling/divergence/loop gates, min-TDI-of-N, commutative vs dependent swaps, arg-change guard, scenario config, CLI envelope flow + rotation +record --runsmake lintclean;uv buildgreenChecklist
make lint+ full suite green[Unreleased]entriescontext/schema_migration.md)Links to
context/ROADMAP.mdPhase M / 0.5.0 (SPEC-0.5.0 Pillar 1). Stacked on #32.