fix(schedule): warm followers before leadership handoff by ChiragAgg5k · Pull Request #13913 · appwrite/appwrite

ChiragAgg5k · 2026-09-25T19:52:48Z

Every scheduler replica now reconciles its source before electing, so a replacement loads its schedules while the old leader keeps dispatching. Adds Scheduler::isReady() and rebases a follower's discovery coverage on the predecessor's committed source view at takeover, so a handoff neither stalls dispatch nor replays delivered changes.

Ported from utopia-php/monorepo#312 after schedule moved to packages/schedule (#13887).

Check	Result
`bin/monorepo check schedule` (Pint, PHPStan, Rector)	passed
`bin/monorepo test schedule` unit	100 tests, 256 assertions
`bin/monorepo test schedule` e2e (Redis)	10 tests, 96 assertions
`bin/monorepo validate`	passed
`git diff --check`	clean

🤖 Generated with Claude Code

A replacement scheduler waited for leadership before loading its source, so a slow initial load stalled dispatch during a surge rollout. Every replica now reconciles before electing, exposes isReady(), and rebases discovery coverage on the predecessor's committed source view at takeover so delivered changes are not replayed. Ported from utopia-php/monorepo#312. Co-authored-by: loks0n <[email protected]>

github-actions · 2026-09-25T20:01:53Z

✨ Benchmark results

Comparing main (before) → fix/schedule-warm-followers (after).

Metric	Before	After	Change
🚀 Requests/sec	`186.68`	`191.8`	⚪ `+2.7%`
⏱️ Latency P50	`92.19 ms`	`90.12 ms`	⚪ `-2.2%`
⏱️ Latency P95	`216.7 ms`	`210.87 ms`	⚪ `-2.7%`

Per-scenario breakdown & investigation details

Metrics below reflect the current branch (after). Δ P95 compares against the base.

Scenario	P50 (ms)	P95 (ms)	Requests	RPS	Δ P95 (ms)
API total	`90.12`	`210.87`	`12,369`	`191.8`	`-5.83`
Account	`173.3`	`327.55`	`651`	`10.79`	`+3.18`
TablesDB	`87.49`	`166.46`	`6,727`	`107.71`	`+1.39`
Storage	`82.59`	`179.81`	`3,255`	`53.44`	`-2.81`
Functions	`129.5`	`253.4`	`1,736`	`29.01`	`-13.17`

Top API waits (after)

API request	Max wait (ms)
account.name.update	`466.18`
tablesdb.rows.decrement	`429.6`
account.prefs.update	`416.98`
functions.variables.update	`393.13`
storage.buckets.create	`389.77`

greptile-apps · 2026-09-25T20:09:36Z

Confidence Score: 3/5

[High risk] Adds readiness tracking and warm-takeover logic to the scheduler.

The PR is not safe to merge until takeover avoids replaying committed replacements and dispatching from a follower view older than the predecessor’s; the implementation-coupled test also needs revision under the repository requirement.

Findings

Committed replacements replay on takeover ▶
Takeover uses an older view ▶
Handoff test mirrors internal yields ▶

Fix with agent prompt

### Issue 1
packages/schedule/src/Scheduler.php:777
When a warm follower has loaded a replacement, its `replaced` flag stays true until it leads and commits. This rebase then starts coverage at the replacement’s `activeFrom`, even if the previous leader already delivered and committed that version. The new leader can deliver those occurrences again.

### Issue 2
packages/schedule/src/Scheduler.php:683
If the leader loads a new or updated schedule after the follower’s last read, then stops before the follower’s next sync, `syncIfDue()` does not refresh the follower at takeover. The follower can commit a window without the new schedule or dispatch its old definition. A later refresh cannot undo that dispatch.

### Issue 3
packages/schedule/tests/E2E/RedisStoreTest.php:349-350
These fixed `resume()` calls assume exactly when the follower’s `make()` callback and run loop suspend. A harmless change to that sequence can break the test without changing delivery behavior. The repository requires tests to assert observable behavior rather than mirror implementation steps; this requirement must be satisfied before merging.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Summary

The PR lets followers reconcile before election, exposes initial-load readiness, and rebases discovery coverage when a prepared follower takes leadership.

Warm handoff can replay an already-covered replacement or dispatch from a source view older than the predecessor’s.
The Redis handoff test relies on fixed internal fiber suspension counts.

_{Reviews (1) · Last reviewed commit: "fix(schedule): warm followers before lea..."}

greptile-apps · 2026-09-25T20:09:44Z

+        if ($syncedUntil !== null) {
+            $seen = $this->moment($syncedUntil);
+            foreach ($this->entries as &$entry) {
+                $entry['coverFrom'] = $this->coverFrom($entry['activeFrom'], $seen, $entry['replaced']);


Committed replacements replay on takeover When a warm follower has loaded a replacement, its replaced flag stays true until it leads and commits. This rebase then starts coverage at the replacement’s activeFrom, even if the previous leader already delivered and committed that version. The new leader can deliver those occurrences again.

Prompt To Fix With AI

This is a comment left during a code review. Path: packages/schedule/src/Scheduler.php Line: 777 Comment: **Committed replacements replay on takeover** When a warm follower has loaded a replacement, its `replaced` flag stays true until it leads and commits. This rebase then starts coverage at the replacement’s `activeFrom`, even if the previous leader already delivered and committed that version. The new leader can deliver those occurrences again. --- For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

greptile-apps · 2026-09-25T20:09:44Z

-                    continue;
-                }
-
                $this->syncIfDue();


Takeover uses an older view If the leader loads a new or updated schedule after the follower’s last read, then stops before the follower’s next sync, syncIfDue() does not refresh the follower at takeover. The follower can commit a window without the new schedule or dispatch its old definition. A later refresh cannot undo that dispatch.

Prompt To Fix With AI

This is a comment left during a code review. Path: packages/schedule/src/Scheduler.php Line: 683 Comment: **Takeover uses an older view** If the leader loads a new or updated schedule after the follower’s last read, then stops before the follower’s next sync, `syncIfDue()` does not refresh the follower at takeover. The follower can commit a window without the new schedule or dispatch its old definition. A later refresh cannot undo that dispatch. --- For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

greptile-apps · 2026-09-25T20:09:44Z

+            $b->resume();
+            $b->resume();


Handoff test mirrors internal yields These fixed resume() calls assume exactly when the follower’s make() callback and run loop suspend. A harmless change to that sequence can break the test without changing delivery behavior. The repository requires tests to assert observable behavior rather than mirror implementation steps; this requirement must be satisfied before merging.

Context Used: Call out and harshly judge implementation-coupled tests. We don't mirror source code, configuration, or version pins in assertions. We test observable behavior; use linters for syntax and schema checks. (source)

Prompt To Fix With AI

This is a comment left during a code review. Path: packages/schedule/tests/E2E/RedisStoreTest.php Line: 349-350 Comment: **Handoff test mirrors internal yields** These fixed `resume()` calls assume exactly when the follower’s `make()` callback and run loop suspend. A harmless change to that sequence can break the test without changing delivery behavior. The repository requires tests to assert observable behavior rather than mirror implementation steps; this requirement must be satisfied before merging. **Context Used:** Call out and harshly judge implementation-coupled tests. We don't mirror source code, configuration, or version pins in assertions. We test observable behavior; use linters for syntax and schema checks. ([source](https://app.greptile.com/review/custom-context?memory=instruction-0)) --- For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

ChiragAgg5k mentioned this pull request Sep 25, 2026

fix(schedule): warm followers before leadership handoff utopia-php/monorepo#312

Closed

greptile-apps Bot reviewed Sep 25, 2026

View reviewed changes

Navigation Menu

fix(schedule): warm followers before leadership handoff - #13913

fix(schedule): warm followers before leadership handoff#13913
ChiragAgg5k wants to merge 1 commit into
mainfrom
fix/schedule-warm-followers

ChiragAgg5k commented Sep 25, 2026

Uh oh!

github-actions Bot commented Sep 25, 2026

Uh oh!

greptile-apps Bot commented Sep 25, 2026 •

edited

Loading

Uh oh!

greptile-apps Bot Sep 25, 2026

Uh oh!

greptile-apps Bot Sep 25, 2026

Uh oh!

greptile-apps Bot Sep 25, 2026

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

ChiragAgg5k commented Sep 25, 2026

Uh oh!

github-actions Bot commented Sep 25, 2026

✨ Benchmark results

Uh oh!

greptile-apps Bot commented Sep 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Confidence Score: 3/5

Findings

Summary

Uh oh!

greptile-apps Bot Sep 25, 2026

Choose a reason for hiding this comment

Uh oh!

greptile-apps Bot Sep 25, 2026

Choose a reason for hiding this comment

Uh oh!

greptile-apps Bot Sep 25, 2026

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

greptile-apps Bot commented Sep 25, 2026 •

edited

Loading