Skip to content

Engine: warm standby part 3, promotion with a fenced writer epoch (EN-12) #285

Closed

Description

@jsam

Part 3 of 4 of EN-12 (#264). Builds on part 1 (#283) and part 2 (sync shipping).

Goal. Promote a follower to primary on failover, with a fenced writer epoch so a deposed primary can never commit again.

Scope.

  • A durable writer epoch per data directory (replication/epoch), carried on every stream frame, heartbeat and write reply.
  • POST /v1/replication/promote {epoch} (admin or the cell controller). The follower stops tailing, persists the epoch, raises its logical time above the newest primary revision it applied (so revisions keep increasing for clients), and becomes writable.
  • Lease-gated commits: a primary commits only while it holds an unexpired lease for its epoch. The controller promotes only after the lease expires plus a margin, so a partitioned old primary stops writing before a new one starts.
  • A node that sees a higher epoch anywhere (a follower hello, a heartbeat, a controller call) becomes read-only at once. Followers refuse streams from a lower epoch.
  • A returning old primary starts read-only and rejoins as a follower through resync. Its unshipped tail is discarded.
  • Subscriptions after failover: clients reconnect to the new primary through the service address. The existing epoch-scoped reconnect cursor is refused, so the client re-subscribes from a fresh snapshot, and nothing from the old primary's log is replayed.

Done when. Tests cover promote, a fenced old primary refusing writes, old-primary rejoin, and split-brain under partition (the old primary's lease expires before promotion).

Activity

  1. added
    backendBackend/server related
    infraInfrastructure, CI/CD, DevOps
    P1Required next for reliable reactive delivery and operations
    on Oct 4, 2026
  2. jsam commented on Oct 6, 2026

    @jsam
    ContributorAuthor

    Done: warm standby part 3 is merged: a durable writer epoch on every stream frame, heartbeat and write reply; promote with a fenced epoch; lease-gated commits (opt-in via replication.lease_required; split-brain protection needs it on); a deposed node rejoins through a resync and cannot be promoted until that resync completes; revisions keep increasing across a promotion. Known limits are listed in the change's description, including four review findings left open. Part 4 (#286) is not started.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    P1Required next for reliable reactive delivery and operationsbackendBackend/server relatedinfraInfrastructure, CI/CD, DevOps

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions