Skip to content

feat(bench): add pytest-benchmark suite with make bench target - #15

Merged
lostmartian merged 1 commit into
mainfrom
feat/perf-benchmarks
Aug 22, 2026
Merged

lostmartian merged 1 commit into
mainfrom
feat/perf-benchmarks

Conversation

@lostmartian

Copy link
Copy Markdown
Collaborator

What

Implements I2 — performance benchmarks: a benchmarks/ directory (excluded from the regular test run via testpaths) covering alignment, end-to-end compare, loop detection (incl. pathological single-tool repeat), recovery metrics, OpenInference parsing at 1000 spans, and report serialization. Run via new make bench (pytest-benchmark, autosaved locally, .benchmarks/ gitignored).

Why

Roadmap WS-I2 and the evidence-based replacement for the descoped A6: measure where the engine actually strains instead of speculatively building streaming ingestion.

How

  • Self-contained synthetic trace factories in benchmarks/conftest.py (deterministic; no coupling to test fixtures).
  • Sizes parametrized at 100/500/1000 steps to expose scaling behavior.
  • make lint/format extended to cover benchmarks/; dev dependency added: pytest-benchmark>=4.0.

Baseline numbers (M2, py3.14) — the headline finding: LCS alignment dominates everything: 100 steps ≈ 16ms → 500 ≈ 390ms → 1000 ≈ 1.6s (quadratic-ish growth), while loop detection (~3ms @1k), metrics (<1ms), and adapter parsing (~17ms @1k spans) are noise by comparison. This hard-confirms the A6 descope rationale and gives us the regression baseline for future aligner work.

Testing

  • make bench: 13 benchmarks pass, results autosaved
  • Regular suite untouched and green: 300 passed; lint clean

Checklist

  • Exactly one logical change in this PR
  • make lint passes
  • uv run python -m pytest is green (300 passed)
  • Docs updated if user-facing — docs sync PR follows this one
  • CHANGELOG.md entry added under [Unreleased]
  • Conventional title

@lostmartian
lostmartian merged commit 4817ab3 into main Aug 22, 2026
@lostmartian
lostmartian deleted the feat/perf-benchmarks branch August 22, 2026 20:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant