Skip to content

Engine: warm standby with WAL streaming, promotion and fenced writer epoch (EN-12) #264

Closed
2 of 3 issues completed
Closed
Engine: warm standby with WAL streaming, promotion and fenced writer epoch (EN-12)#264
2 of 3 issues completed
Labels
P1Required next for reliable reactive delivery and operationsbackendBackend/server relatedinfraInfrastructure, CI/CD, DevOps
Milestone
Engine production readiness

Description

@jsam
jsam
opened on Oct 4, 2026

Goal. A warm standby: WAL streaming to a follower, promotion on failover, and a fenced writer epoch so a deposed primary cannot commit.

Why. High availability with a measured recovery point and recovery time. Today the engine is a single writer with online and offline backups (#216) only.

Done when. Split into four PRs before starting: WAL streaming, follower apply, promotion with writer epoch, and failover in the Kubernetes deployment; RPO and RTO measured on the perf gate and documented.

Plan reference: EN-12 (large; split before work starts). Part of the Engine production readiness milestone (the current focus; tracker #199). Rule for every engine PR: millisecond query and update latency must not regress, and the perf gate runs before the PR opens.

Activity

  1. jsam
    added this to the Engine production readiness milestone on Oct 4, 2026
  2. jsam
    added
    backendBackend/server related
    infraInfrastructure, CI/CD, DevOps
    P1Required next for reliable reactive delivery and operations
    on Oct 4, 2026
  3. jsam
    added sub-issues on Oct 4, 2026

  4. jsam commented on Oct 4, 2026

    @jsam
    jsam
    on Oct 4, 2026
    ContributorAuthor

    Split into four PRs:

    1. WAL streaming to a read-only follower that applies the stream and stays consistent, with asynchronous shipping: feat(replication): stream committed changes to a read-only warm-standby follower #283 (in progress)
    2. Synchronous shipping (RPO 0 for acknowledged writes): Engine: warm standby part 2, synchronous shipping with RPO 0 (EN-12) #284
    3. Promotion with a fenced writer epoch: Engine: warm standby part 3, promotion with a fenced writer epoch (EN-12) #285
    4. Cell failover in the Kubernetes deployment, with RTO and RPO measured: inputlayer/inputlayer-platform#19

    Design summary:

    • The primary streams engine events in commit order, numbered by LSN under the lock that orders each change, over GET /v1/replication/stream. The events are commits in WAL framing, graph create/drop, relation drop, and index create/drop.
    • A follower that is behind the retained log, or that reconnects to a restarted primary, reconciles to a consistent checkpoint, then tails.
    • The follower applies events idempotently, so it saves its position after applying.

  5. jsam commented on Oct 10, 2026

    @jsam
    jsam
    on Oct 10, 2026
    ContributorAuthor

    Warm standby parts 1-3 were closed by https://github.com/inputlayer/inputlayer-symbolic/pull/16, with follow-ups in https://github.com/inputlayer/inputlayer-symbolic/pull/29, https://github.com/inputlayer/inputlayer-symbolic/pull/98, https://github.com/inputlayer/inputlayer-symbolic/pull/80, https://github.com/inputlayer/inputlayer-symbolic/pull/95. Failover RTO is unmeasured and writer confirmation is only safe for a two-server pair; part 4 moves with inputlayer/inputlayer-platform#19. Remaining work continues in https://github.com/inputlayer/inputlayer-symbolic/issues/128.

  6. jsam
    closed this as completedon Oct 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    P1Required next for reliable reactive delivery and operationsbackendBackend/server relatedinfraInfrastructure, CI/CD, DevOps

    Type

    No type

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions