Repository navigation
feat(replication): synchronous shipping so acknowledged writes survive losing the primary - #367
Merged
Merged
Conversation
jsam
changed the title
jsam
changed the title
jsam
marked this pull request as ready for review
October 5, 2026 20:52
…e a follower applied it (EN-12 part 2) replication.mode = "sync" makes a primary reply to a request that changed state only once a follower acked durable apply of the newest event it appended. replication.sync_timeout_ms bounds the wait; on_follower_loss decides what follows: block (the reply fails with the new wire code replica_unconfirmed, the commit stands on the primary) or degrade (fall back to async with an alert, re-arm once a follower caught up). Requests note their LSNs through a task-local tracker handed to blocking pool threads, so every write path (programs, DDL, users, keys, ACLs) waits and reads never do. Async mode tracks nothing. Replication lag (events and age of the oldest unconfirmed event) and the sync state are on /metrics, /metrics/prometheus and /v1/replication/status.
…ttings and metrics; measured cost and RPO Measured on the perf gate host (release, primary and follower on one 32-core host, 5 interleaved rounds): 1 writer p50 1.46 ms async vs 2.69 ms sync; 4 writers 766 vs 770 commits/s. Async unchanged against 57ad2e9 (1.44 ms, 763/s). 10 kill-the-primary trials per mode: 0 acknowledged writes lost in sync mode.
…cument durable followers
jsam
changed the title
This was referenced Oct 5, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Intent
I want you to really hyper focus on making the symbolic engine, you know, crisp and really truly production ready. ... just ensure that both machines are propelry utilized now (Context: implement #284 exactly as written.)
Issue #284, "Engine: warm standby part 2, synchronous shipping with RPO 0 (EN-12)", part 2 of 4 of EN-12 (#264); part 1 (WAL streaming to a follower that applies and stays consistent, asynchronous shipping) is PR #283. Goal: synchronous shipping: a commit is acknowledged only after a follower has applied it durably, so the RPO is zero for every acknowledged write while a sync follower is attached. Scope: replication.mode = "async" | "sync" on the primary; in sync mode a commit reply waits until a follower acks its LSN (followers already ack each applied frame after one WAL fsync). replication.sync_timeout_ms and on_follower_loss = "block" | "degrade": block = writes wait, then fail with a timeout, so RPO stays 0 and availability drops; degrade = fall back to async with an alert, and re-arm once a follower catches up. The commit is already durable on the primary when the wait times out, so a timeout is reported as "not confirmed on a replica", never as a failed commit. Replication lag and the sync state on /metrics and /metrics/prometheus. Measure the commit latency cost of sync mode, and RPO in both modes, on the perf gate host. Done when: a kill-the-primary test in sync mode loses no acknowledged write; the latency cost is measured and documented; async mode latency is unchanged.
What Changed
replication.mode = "async" | "sync",sync_timeout_ms(default 5000) andon_follower_loss = "block" | "degrade"to the primary. In sync mode a write's reply waits until a follower acks an LSN at least as high as both the write's own LSN and the log head, so this also covers no-op writes. Withblock, a write that no follower confirms in time fails with the newreplica_unconfirmederror code: it is committed on the primary, just not confirmed on a replica. Withdegrade, the primary falls back to async with an alert and switches back to sync once a follower catches up. The new code is added to the ws-protocol, JS/Python clients, GUI types and AsyncAPI/error specs./metrics(the stats DTO gets areplicationfield) and on/metrics/prometheusasinputlayer_replication_*metrics. These include lag in events and ms, the sync state, waits, wait time, unconfirmed writes, degrades and rearms.tests/replication_tests.rs. One kills the primary in sync mode and checks that no acknowledged write is lost; another measures latency and RPO in both modes. The replication and configuration guides now document the new settings, the failure behaviour and the measured cost. For one writer, mean/p99 commit latency is 1.46 ms / 5.6 ms in async mode and 2.69 ms / 8.5 ms in sync mode; async stays at about the same latency as before.Risk Assessment
✅ Low: Sync shipping is opt-in (async stays the default and its path only adds a cheap staged/note lookup). Every client write entry point (execute_program_with_params, query_program, query_program_with_session, with nested meta commands tracked by the outer request) goes through the single Handler::replicated boundary. The confirmed-LSN, block/degrade and re-arm logic traced correctly against follower acks sent after its WAL fsync. Earlier rounds' decisions (head-of-log wait, durable-follower refusal, keeping the extra metrics) are implemented as decided.
Testing
I built release binaries of the change and of the base commit, ran the 3 sync-replication integration tests that start the real server (all passed), and drove primary/follower process pairs over WebSocket and HTTP. In 8 sync-mode SIGKILL trials (about 1,000–2,400 acknowledged writes in most) no acknowledged write was lost, even with the follower frozen. With the follower frozen, async mode lost 633–739 acknowledged writes per trial. Block mode returns
replica_unconfirmedafter the timeout (no-op writes included) and recovers when a follower returns. Degrade mode falls back to async and re-arms. Metrics report lag and sync state. Invalid configurations are refused. Async latency matches the base build, and sync costs about +1.3 ms per single-writer commit (median). Everything passed, and the temp data dirs, server processes and base build were removed.Evidence: Kill-the-primary trials (sync vs async, plain and frozen-follower)
sync plain x5 (plus trial 1): lost_acked 0 | sync with follower frozen x3: lost_acked 0 | async plain x3: lost 1, 0, 0 | async with follower frozen x3: lost_acked 739, 642, 633Evidence: Block mode, no follower: replica_unconfirmed incl. no-op writes
Evidence: Block mode recovers once a follower attaches; follower refuses writes
Evidence: Stalled-state /metrics and /metrics/prometheus
Evidence: Degrade then re-arm transcript
Evidence: Config guard refusals
Evidence: Commit latency: standalone / base async / new async / new sync
Evidence: Integration tests run
Pipeline
Updates from git push no-mistakes
✅ **intent** - passed
✅ No issues found.
🔧 **Rebase** - 1 issue found → auto-fixed ✅
src/storage_engine/mod.rs- merge conflict rebasing onto origin/main🔧 Fix applied.
✅ Re-checked - no issues remain.
✅ **Review** - passed
✅ No issues found.
✅ **Test** - passed
✅ No issues found.
cargo build --release --bin inputlayer-server(change) and the same build of base commit 32b56c62 in a /tmp copy, used only for the latency comparisoncargo test --release --test replication_tests -- in_sync_mode_killing_the_primary block_fails_writes degrade_falls_back: 3 passed (these tests start the real server binary)Manual: sync primary, block mode, no follower; over WebSocket: read, insert, read, re-insert of a present fact (no-op), delete of an absent fact (no-op) (block.mjs nofollower)Manual:/metrics/prometheusand/metricson the stalled primaryManual: started a follower, then wrote to the primary, read from the follower, tried a write on the follower, and read the primary's/v1/replication/statusManual: 15 kill-the-primary trials (killtrial.sh): 4 concurrent WebSocket writers, then SIGKILL of the primary; every acknowledged write checked against the follower's rows. Ran sync ×5, async ×3, and a frozen-follower (SIGSTOP) variant sync ×3 and async ×3Manual: sync primary in degrade mode; SIGSTOP the follower, write, SIGCONT it, write again; checked the status counters, log linesreplication_sync_degraded/replication_sync_rearmed, and Prometheussync_stateManual: config guards: follower withdurability_mode = "async"(andmode = "sync"in its section),mode = "semi",on_follower_loss = "ignore",sync_timeout_ms = 0Manual: latency (latpair.sh/latency.mjs): 600 commits from 1 writer, then 5 s of 4 writers; arms were standalone, base async pair, new async pair and new sync pair, each run twice✅ **Document** - passed
✅ No issues found.
✅ **Lint** - passed
✅ No issues found.
✅ **Push** - passed
✅ No issues found.