Skip to content

Performance gate: quiet runner and fsync-bound metric calibration #198

Closed

Description

@jsam

Problem

The performance gate (#163, make perf-gate) cannot return PASS on the current shared development host, even when the approved baseline is compared with itself. Release checkpoint 1 ran it in a quiet window (all other builds paused), at 20 rounds with servers pinned (--server-cpus 4-31):

  • A/A of e732cd5: INCONCLUSIVE. 17 of 20 required metrics pass. Three straddle their budget, with these baseline round-to-round spreads:
    • insert_single.ack_us.p50: 101%
    • insert_batch.ack_us.p50: 69%
    • interference.delta_us.p99: 2317%, from rare multi-hundred-ms stalls
  • Candidate (release checkpoint 1 branch): INCONCLUSIVE with no FAIL. 14 of 20 pass, and every median ratio is at most 1.03.

Durable-ack p50 is dominated by fsync latency on the VM's virtual block device (ext4 on /dev/vda1), which is bimodal. The same metric's p99 (about 42 ms) is stable, at 0.7-2% spread.

The gate itself discriminates correctly. A 1 ms delay injected before every WS execute fails every cheap_query metric at 3.1-6.1x, with intervals well clear of the budget.

The calibration record is in perf-gate/baselines/e732cd52-aa.report.md and e732cd52-aa.run.json.gz.

Acceptance criteria

  • Provide a dedicated quiet runner, or document a pinned-host recipe (local NVMe or tmpfs-backed durable path, CPU isolation for servers and client), on which an A/A run of the approved baseline returns PASS on every required metric.
  • Calibrate the fsync-bound metrics (insert_*.ack_us.p50) and the interference.delta_us.p99 tail against that runner's A/A noise. Any change to the required set or budgets goes through the captain's D1 policy decision; nothing is loosened silently.
  • Re-record the approved baseline's A/A calibration on the chosen runner, and update perf-gate/README.md (runner requirements) to match.

Activity

  1. jsam commented on Oct 4, 2026

    @jsam
    ContributorAuthor

    Moved to the Engine production readiness milestone, the current focus (tracker #199). The quiet runner attests that engine latency does not regress; every readiness PR depends on it.

    Roadmap per the neurosymbolic decision layer plan: engine production readiness first, then gateway neural integration with Jev, CLM and Laya (#266).

  2. jsam commented on Oct 10, 2026

    @jsam
    ContributorAuthor

    The stall-aware gate with an A/A pass landed in https://github.com/inputlayer/inputlayer-symbolic/pull/83 and the measurement slot recipe in https://github.com/inputlayer/inputlayer-symbolic/pull/101. The approved baseline and policy still point at the old, inconclusive calibration. Remaining work continues in https://github.com/inputlayer/inputlayer-symbolic/issues/126.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    benchinfraInfrastructure, CI/CD, DevOps

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions