Skip to content

feat: distributed run coordination (one instance per fire across a fleet)#549

Merged
merencia merged 5 commits into
mainfrom
feat/distributed-lock
Jun 18, 2026
Merged

feat: distributed run coordination (one instance per fire across a fleet)#549
merencia merged 5 commits into
mainfrom
feat/distributed-lock

Conversation

@merencia

@merencia merencia commented Jun 17, 2026

Copy link
Copy Markdown
Member

Phase 1 — the RunCoordinator provider. Opt-in distributed coordination so a task runs on a single instance per fire across a fleet (the #477 use case), without touching noOverlap.

The honest abstraction is "should this instance run this fire?" — a Redis lock is just one answer. The built-in default keys off an env var (a single designated runner, zero deps); a Redis-backed coordinator is the opt-in HA upgrade.

API

import cron, { setRunCoordinator } from 'node-cron';

// Out of the box: NODE_CRON_RUN=true on one instance, =false on the others.
cron.schedule('0 3 * * *', backup, { name: 'nightly-backup', distributed: true });

// HA upgrade: per-fire coordination across the fleet (any node can run, one wins).
setRunCoordinator(new RedisLockCoordinator(redis)); // you implement the interface
interface RunCoordinator {
  shouldRun(key: string, ttlMs: number): boolean | Promise<boolean>;
  onComplete?(key: string): void | Promise<void>;
}

Semantics

  • Opt-in per task (distributed: true) — setting a coordinator does not gate every task.
  • Key is ${name}:${fireTime} → requires a dev-defined name (the generated id is per-process and can't coordinate across instances). Throws if missing.
  • Default = EnvVarRunCoordinator (NODE_CRON_RUN, 'true'/'false'). No default value: a missing/invalid env throws at schedule time (startup), so a misconfigured fleet fails loudly instead of running everywhere (duplicates) or nowhere. It's a single designated runner — not HA.
  • Resolution: per-task runCoordinator → global setRunCoordinator → env-var default.
  • Skipping: a not-elected instance emits execution:skipped with context.reason: 'not-elected'. The instance that runs emits the normal execution:started/finished — no separate event needed.
  • Fail-closed: if shouldRun throws (e.g. Redis down), the run is skipped with reason: 'coordinator-error' (the one to alert on).
  • distributedTtl (default 30000) is a safety lease for lease-based coordinators; ignored by the env-var default. Must exceed the run time.
  • Guarantee: no concurrent execution across instances — effectively once when clocks are in sync, not a hard exactly-once. noOverlap (in-process) is a separate, unchanged concern.
  • Background tasks: supported. The coordinator lives in the parent; the daemon asks over IPC (IpcRunCoordinator), so the same shared backend coordinates the whole fleet.

Tests / coverage

Coordinator cases (elected run + onComplete + no skip, not-elected skipped, fail-closed coordinator-error, survives onComplete failure, custom distributedTtl, per-task override), the env-var default (eager throw, valid values, custom name), the IPC bridge unit, the parent IPC handler (shouldRun true/false/error, complete, no-coordinator), and the daemon wiring + skipped forwarding. src/coordinator 100% covered; overall coverage rises. Full suite green (373 tests).

A companion @node-cron/redis-coordinator package (implementing this interface) is fully specced for a separate build.

merencia added 2 commits June 17, 2026 17:05
Adds an opt-in `lock: true` per task, backed by a pluggable LockProvider set via
`cron.setLockProvider(...)`. node-cron ships no default provider — the guarantee
only holds with a real shared backend (e.g. Redis), so `lock` without a provider
throws.

- Key is `${name}:${fireTime}`, so it requires a dev-defined `name` (the
  generated id is per-process and can't coordinate). Throws if missing.
- The winner emits `execution:locked`, runs, releases, then emits
  `execution:unlocked`. A loser emits `execution:lockHeld` and skips.
- `acquire` failures are fail-closed (skip the run). `lockTtl` (default 30000)
  is a crash-safety expiry; it must exceed the run time.
- Guarantee: no concurrent execution across instances (effectively once with
  synced clocks) — not a hard exactly-once. `noOverlap` (in-process) is separate
  and unchanged.
- Not supported for background tasks (the provider can't cross the fork); throws.

Interface: `acquire(key, ttlMs): Promise<boolean>` + `release(key): Promise<void>`.
The assertion compared getMinutes() to now.getMinutes()+1, which is 60 (not 0)
during the :59 minute. Assert the next-minute-boundary invariant instead
(seconds/ms zeroed, within 60s of now), robust to the rollover.
@merencia
merencia force-pushed the feat/distributed-lock branch from b751caa to 5c1be38 Compare June 17, 2026 20:07
merencia added 2 commits June 17, 2026 18:21
…rovider

Background tasks run in a forked daemon, so the lock provider (set in the
parent via setLockProvider) cannot cross the fork. The daemon now uses an
IpcLockProvider bridge: it asks the parent for the lock over IPC, the parent
runs the real provider and replies. Cross-fleet coordination still happens in
the shared backend (e.g. Redis) held by each instance's parent; IPC only
bridges the child to its own parent. A provider error is reported back so the
daemon fails closed (skips the run), mirroring the inline path. The daemon now
forwards execution:locked/unlocked/lockHeld.

Also adds a per-task `lockProvider` option that overrides the global one
(inline and background). createTask no longer rejects background + lock; it
requires a name and a resolvable provider (per-task or global).
Replaces the Redis-presuming `LockProvider` with a `RunCoordinator`: the
honest abstraction is "should this instance run this fire?". The built-in
default (EnvVarRunCoordinator) keys off NODE_CRON_RUN for a single designated
runner — no Redis, works out of the box. A real coordinator (e.g. a Redis
lock) is the opt-in HA upgrade via setRunCoordinator. The env-var default has
no default value: a missing/invalid env throws at schedule time, so a
misconfigured fleet fails loudly at startup instead of running everywhere
(duplicates) or nowhere.

API rename (pre-merge, nothing released):
- LockProvider -> RunCoordinator (acquire/release -> shouldRun/onComplete)
- setLockProvider -> setRunCoordinator; lock -> distributed; lockTtl -> distributedTtl
- IpcLockProvider -> IpcRunCoordinator; src/lock -> src/coordinator
- events execution:locked/unlocked/lockHeld -> a single execution:skipped
  carrying context.reason ('not-elected' | 'coordinator-error'). The winner
  needs no event: execution:started/finished already fire only where it ran.
@merencia merencia changed the title feat: distributed lock (single instance per fire across a fleet) feat: distributed run coordination (one instance per fire across a fleet) Jun 17, 2026
Review follow-ups on the distributed coordination PR:

- Daemon exits on `process.on('disconnect')`. A forked daemon does not die
  with its parent — orphaned, it would keep running the schedule, and a
  distributed task's IPC shouldRun would hang forever with no parent to reply.
  Exiting on channel disconnect fixes both (the hang's root cause and a latent
  orphan bug), without a magic-number request timeout.
- Document that `maxExecutions` is counted per instance: with `distributed` and
  a per-fire coordinator, the fleet total can exceed it.
- Replace fixed sleeps in the daemon skipped-forwarding test with event-gated
  polling (waitFor) to remove real-timer flakiness.
@merencia
merencia merged commit 2ba12f1 into main Jun 18, 2026
6 checks passed
@merencia
merencia deleted the feat/distributed-lock branch June 18, 2026 01:12
@merencia merencia mentioned this pull request Jun 18, 2026
merencia added a commit that referenced this pull request Jun 18, 2026
Promote the Unreleased section to 4.4.0 and bump the version.

Highlights (all additive, backwards-compatible):
- Task introspection: getNextRuns, match, msToNext, isBusy, runsLeft, getPattern (#547)
- cron.parse and cron.validateDetailed (#548)
- Distributed run coordination: distributed:true + NODE_CRON_RUN default + RunCoordinator (#549)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant