feat: Add retry state for RETRY-spec backoff by jsonbailey · Pull Request #522 · launchdarkly/python-server-sdk

jsonbailey · 2026-09-14T15:08:45Z

Summary

Adds the retry state machine that the FDv1 data sources will use for RETRY-spec conformance. Pure addition — nothing imports it yet, so it can be reviewed against the spec without reading any data-source code.

RetryState answers one question: how long to wait before the next attempt. It serves all four data sources; streaming and polling differ only in their construction parameters and their reset policy.

The algorithm

Every failure is classified NORMAL or UNEXPECTED. 400, 408, 429 and all 5xx are normal; every other 4xx, including 401 and 403, is unexpected. Only an HTTP status can be unexpected — every network and TLS failure is normal.

A normal failure advances the delay on the current curve. An unexpected failure raises both bounds and keeps them raised until the reset condition is met, so a normal failure that follows cannot lower them.

streaming, normal:    1 -> 2 -> 4 -> 8 -> 16 -> 30 -> 30 ...
streaming, extended:  300 -> 600 -> 1200 -> 2400 -> 3600 -> 3600 ...
polling, normal:      poll_interval, flat — no escalation
polling, extended:    max(300, poll_interval) doubling to max(3600, poll_interval)

Each delay is then reduced by a jitter of up to half itself, and never falls below the caller's operating cadence — poll_interval for polling, absent for streaming.

Two things that are easy to get wrong

A backoff wait applies to a retry, not to every operation. record_success restores the operating cadence even while the retry state is still raised, so a recovered service is polled at its normal rate immediately. Another SDK shipped without this: after an outage its first successful poll still waited twenty minutes.

Clearing the retry state is separate from restoring the cadence. record_success does the latter; _reset_if_due does the former, and only when the reset policy is satisfied. Streaming's policy is 60 seconds of continuous healthy operation, polling's is two consecutive successes.

_reset_if_due is called from record_failure because nothing runs while a stream is healthy — the moment the 60-second threshold is crossed is otherwise unobservable, so the next failure is the only place it can be noticed.

Why TLS failures are normal

The narrow classification was built first and reverted. All SDK traffic is HTTPS, so a genuine certificate problem cannot be distinguished from an ordinary transient fault across platforms — a peer sending FIN mid-handshake surfaces differently from one sending RST, and the classification would depend on which an intermediary happened to send. A connection flapping faster than the reset threshold then ratchets to the hour ceiling with no way out.

Testing

make test: 1599 passed. make lint: clean across 226 files.

71 tests covering both delay ladders exactly, jitter bounds, ceiling stickiness, both reset policies, the cadence floor, and the classification table. No test sleeps: the clock and the jitter are patched at the module level, so the 60-second reset and a 20-cycle flapping scenario both run instantly.

Note

Overview
Introduces a standalone RetryState module (ldclient/impl/retry.py) that will drive FDv1 data-source reconnect/poll timing. Nothing in production imports it yet.

Failures are classified as normal vs unexpected (via classify_http_status: most 4xx except 400/408/429 are unexpected; 5xx and transport errors stay normal). Backoff doubles to a ceiling, subtracts up to half jitter, and never waits less than the operating cadence (zero for streaming, poll_interval for polling). An unexpected failure switches to an extended delay ladder (5–60 minutes) that stays sticky until reset.

Streaming uses for_streaming with exponential normal delays (default 1s → 30s cap) and resets after 60s of healthy operation. Polling uses for_polling with flat normal retries at the poll interval and resets after two consecutive successes. record_success restores the healthy cadence immediately even while extended state is still raised; clearing extended state is separate and gated by the reset policy.

Adds test_retry.py (delay tables, jitter bounds, reset policies, invalid config fallbacks) plus test_util helpers to patch retry’s time/random without sleeping.

^{Reviewed by Cursor Bugbot for commit 4d300c4. Bugbot is set up for automated code reviews on this repo. Configure here.}

tanderson-ld · 2026-09-15T15:33:58Z

+
+# The delay bounds of the extended regime, in seconds. A component enters the
+# extended regime after an unexpected failure.
+EXTENDED_INITIAL_DELAY = 5 * 60


Is it the norm in python that all time values / APIs are float seconds?

Yes. time.monotonic(), time.sleep(), threading.Event.wait() and asyncio.sleep() all take or return float seconds, so a float is what every call site here needs. Config already exposes poll_interval and initial_reconnect_delay as float seconds too, so this matches what callers already pass in.

tanderson-ld · 2026-09-15T15:53:22Z

+
+# An upper bound on the backoff exponent, so a long outage cannot overflow the
+# delay computation. Any real ceiling is reached long before this.
+_MAX_BACKOFF_EXPONENT = 30


This approach for calculation can remove the need for bounds protection logic.

https://github.com/launchdarkly/dotnet-eventsource/pull/126/changes#diff-16849bf43eb8418d2b4129594604eba65613dabf4f40abc7f6dae553239de81cR65

https://github.com/launchdarkly/dotnet-eventsource/pull/126/changes#diff-16849bf43eb8418d2b4129594604eba65613dabf4f40abc7f6dae553239de81cR86

We are limited by the float seconds data type that is used in the SDK so those bound protections won't work.

tanderson-ld · 2026-09-15T16:04:13Z

+        self._extended_initial_delay = extended_initial_delay
+        self._extended_ceiling = extended_ceiling
+        self._reset_policy = reset_policy
+        self._operating_cadence = operating_cadence


It seems like this operating_cadence is to help with the normal second poll after the first success? Can't this be calculated based on normal_initial_delay? It seems like redundant state? Maybe I'm not understanding the usage of operating_cadence and initial_delay set to different non-zero values.

You can think of operating cadence as the time between normal healthy operations and the normal_initial_delay as the starting point for delays between failed operations. For polling, these are the same, but for streaming the default cadence is 0s, but the failed starts at 1s or whatever the user configures so that a backoff can actually be calculated.

tanderson-ld · 2026-09-15T16:57:28Z

+        """
+        self._reset_policy.note_healthy()
+        self._reset_if_due()
+        self._next_delay = self._operating_cadence


This can be 0? I think this indicates a logical flaw in how the values are flowing through / overriding each other.

It can be 0, we can meet if it would be helpful to talk through this but 0 is intentional (for streaming).

…cess

tanderson-ld · 2026-09-15T18:31:08Z

+    log.warning(
+        "%s must be a positive, finite number of seconds; using the default of %ss"
+        % (name, default)
+    )


Are logger instances ever passed around in the code base? This static call stands out to me coming from our other code bases.

No, its is the standard in this repo to pull from the import. And it is different then most other languages.

…loor Streaming's operating cadence is zero rather than absent, so a healthy stream schedules no wait and a retry delay has no floor. Polling is the only data source that reads next_delay as a DelaySource, and its cadence floor keeps that from reaching zero. Make the seven observational properties private. No data source reads them; they are test instrumentation, and as public API they invite callers to attach logic to unsynchronized state.

The RETRY spec binds a normal ceiling and an extended ceiling. Use those words for the configured bounds so the parameters read as the spec does, and keep the _delay suffix the initial-delay names already use. The mutable maxDelay keeps its name: Requirement 1.3.2 calls it that.

…e delay The exponent driver was _n and a second counter held the name attempts, which is what Requirement 1.4.1 calls the exponent driver. A reviewer reading self.attempts against the spec was reading the wrong field. The second counter is gone. It had no reader outside tests, not even a logger, and every test that used it recorded only normal failures -- where the two counters are equal by construction.

🤖 I have created a release *beep* *boop* --- ## [9.18.0](9.17.0...9.18.0) (2026-09-25) ### Features * Add retry state for RETRY-spec backoff ([#522](#522)) ([e734558](e734558)) * Drive repeating tasks from a delay source ([9abe8cb](9abe8cb)) * Retry indefinitely after a data source failure instead of stopping permanently in FDv1 ([e17e173](e17e173)) * warn that payload filtering has no effect with FDv2 ([#518](#518)) ([296311f](296311f)) * Warn when the SDK is used in a forked process without postfork ([#512](#512)) ([75402e9](75402e9)) ### Bug Fixes * Add async modules to the readthedocs API reference ([#513](#513)) ([f40d3b2](f40d3b2)) * Log the cached-data evaluation warning only once per client ([#520](#520)) ([59ce998](59ce998)) * Read all items when a collection is empty ([#524](#524)) ([1087147](1087147)) * Repeating task wait duration is no longer reduced by a slow callback duration. ([9abe8cb](9abe8cb)) * Report a distinct User-Agent for the async client ([#516](#516)) ([514c467](514c467)) * Warn and use the documented default for an invalid poll interval or initial reconnect delay ([e17e173](e17e173)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).  --- > [!NOTE] > **Overview** > This PR **cuts release 9.18.0** by bumping the package version in `pyproject.toml`, `ldclient/version.py`, `.release-please-manifest.json`, and the provenance docs example, and by adding the **9.18.0** section to `CHANGELOG.md`. > > There is **no application code in the diff**—Release Please is packaging work that was already merged. The new changelog highlights **FDv1 data-source resilience** (indefinite retry after failures, RETRY-spec backoff state, repeating tasks driven by a delay source, and sane defaults/warnings for invalid poll/reconnect settings), **operational warnings** (forked process without `postfork`, FDv2 payload filtering), and **fixes** (empty-collection reads, repeating-task timing, one-time cached-data warnings, distinct async `User-Agent`, ReadTheDocs async API docs). > > <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit eef08a1. Bugbot is set up for automated code reviews on this repo. Configure [here](https://www.cursor.com/dashboard/bugbot).</sup>  --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Jason Bailey <[email protected]>

…off (#2045) ## Summary Adds a reusable retry controller to `@launchdarkly/js-sdk-common` that owns retry tracking, health checking, and delay calculation for long-running components. This is the foundation for RETRY-spec conformance in the Node server SDK (SDK-2790); the data-source wiring that consumes it arrives in a follow-up PR. **There are no consumers in this PR** — the additions are unused by product code and exercised only by tests. The design ports the controller pattern from the Python server SDK ([python-server-sdk#522](launchdarkly/python-server-sdk#522), wired in [#519](launchdarkly/python-server-sdk#519)), with deliberate differences noted below. ## What's where - **`src/datasource/retry/`** — the retry mechanics: the `RetryState` interface and its `createRetryState` factory (the controller), the `ResetPolicy` interface with its two implementations (`AfterHealthyFor` for streaming's healthy-duration reset, `AfterConsecutiveSuccesses` for polling's two-in-a-row reset), and the `forStreaming`/`forPolling` factories that bind the standard values (1s→30s normal and 5min→1hr extended regimes; 60s healthy window; two-success polling reset) with warn-and-default validation of the configured delay. - **`src/errors.ts`** — failure classification, placed beside the legacy helper it supersedes: `FailureKind` (`'normal' | 'unexpected'`), `classifyHttpStatus` (400/408/429 and 5xx and non-error statuses are normal; any other 4xx is unexpected), and `classifyTransportFailure` (always normal — in an all-HTTPS system certificate failures can't be reliably distinguished from transient faults). `isHttpRecoverable` now delegates to `classifyHttpStatus` — one table, no drift — and is documented as superseded; it stays undeprecated because the event-delivery pathway still legitimately consumes it until that pathway migrates. ## Design points for review - **`RetryState` is an interface, not a class.** `createRetryState(config)` returns it, backed by a closure over local state; `forStreaming`/`forPolling` return the interface. This keeps the publicly exposed surface an interface (per the repo's prefer-interfaces guideline) and lets future mutators be added additively. The `ResetPolicy` implementations stay classes, since the `ResetPolicy` interface already fronts them everywhere they are consumed. - **Clockless seams.** No method of `RetryState` or `ResetPolicy` takes a timestamp. Time lives in exactly one place: `AfterHealthyFor`'s constructor-injected clock, defaulting to a monotonic source (`performance.now()`, with a `Date.now` closure fallback for exotic runtimes). The controller itself holds no clock; its only injectable is `random`, for deterministic jitter tests. (The parameter is named `clock` rather than the codebase's `timeStamper` deliberately — it is not a timestamp source.) - **Ceiling-bounded backoff, no exponent constant.** The delay computation compares the base against the ceiling scaled *down* (`base >= max / 2**exponent`) rather than scaling the base up, so nothing can overflow the ceiling — the same compare-before-shift structure as the .NET implementation, expressed in lossless power-of-two float math. A zero base (legal: the spec forbids flooring server-directed retry values) short-circuits, closing a `0 × Infinity = NaN` edge otherwise reachable when a zero-valued server-directed retry is followed by very many failures. - **A configured delay above the normal ceiling clamps to it.** In the normal regime the ceiling wins, matching the literal spec (1.3.2 + 1.4.2) and the majority of the SDK fleet (Go, Java, .NET). The extended regime is the opposite: its bounds are floored at the configured delay, which spec requirement 1.5.4.1 mandates ("a delay or ceiling that applies after an `unexpected` failure MUST NOT be less than the component's initial delay"). - **`applyServerDirectedRetry(ms)`** — the SSE `retry:` entry point: sticky base that takes precedence over the regime's initial delay (including the extended regime's), doubling restarted, ceiling still applies, survives a healthy reset. Wire-level validation and the 1-hour cap live in the SSE library ([launchdarkly/js-eventsource#40](launchdarkly/js-eventsource#40)), not here. - **Post-success wait = operating cadence**, even while the retry state is raised — a recovering poller returns to schedule immediately rather than serving one more extended-regime wait. ## Testing 89 tests across three suites (584 package-wide, all green): - `RetryState.test.ts` (48) — exact delay ladders for both regimes under injected clock/random (including the 5m/10m/20m/40m/1h/1h extended ladder), regime transition/ratchet/re-arm, anchor-once healthy-stretch discrimination, reset-before-count ordering, fast-second-poll cadence, poll-interval wait floor under real jitter, jitter range with the maximal-draw boundary (the exact-`T/2` tie) and distinctness assertions, server-directed retry semantics (replace/clamp/persist/reject-invalid/later-wins, precedence over both regimes' initial delays, `retry: 0` staying finite through 1,100 failures), normal-ceiling clamp and regime-collapse cases, the extended-floor mandate under direct construction, flapping-never-ratchets, high-n robustness, and factory validation matrices. - `ResetPolicy.test.ts` (8) — the policy seam directly: threshold boundary, anchor-once under repeated healthy reports, failure-clears, consecutive-success counting, and both default-clock closures (monotonic and the `Date.now` fallback). - `errors.test.ts` (33) — the full classification matrix including boundary and non-error statuses, transport classification, and parity pins on the legacy `isHttpRecoverable` truth table through the delegation. The `src/datasource/retry` module is at 100% statement, branch, function, and line coverage. Coverage was cross-checked against the Python, Java, Go, and .NET RETRY test suites; the one technique deliberately not ported is .NET's `BigInteger`/randomized reference sweeps, which exist to exercise 64-bit integer shift surfaces that JS float arithmetic does not have.  --- > [!NOTE] > **Overview** > Introduces a **RETRY-spec-oriented retry controller** in `@launchdarkly/js-sdk-common` as shared library code only—**no data sources or SDKs call it yet**; follow-up PRs will wire streaming/polling. > > Adds `src/datasource/retry/` with **`RetryState`** (`createRetryState`, `forStreaming`, `forPolling`): exponential backoff with jitter, normal vs **extended** regimes after `unexpected` failures, pluggable **`ResetPolicy`** (healthy-for duration for streaming, consecutive successes for polling), and **`applyServerDirectedRetry`** for SSE `retry:` values. **`errors.ts`** gains **`FailureKind`** plus **`classifyHttpStatus`** / **`classifyTransportFailure`**; **`isHttpRecoverable`** now delegates to the same rules. > > New retry APIs are re-exported from the datasource barrel and package **`index`**. CI **package size limit** for common ESM rises **29 000 → 29 500** bytes. Coverage is **89 new tests** across retry and error classification. > > <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit 7f529cf. Bugbot is set up for automated code reviews on this repo. Configure [here](https://www.cursor.com/dashboard/bugbot).</sup>

feat: Add retry state for RETRY-spec backoff

ce18f18

jsonbailey marked this pull request as ready for review September 14, 2026 18:30

jsonbailey requested a review from a team as a code owner September 14, 2026 18:30

jsonbailey added 2 commits September 14, 2026 13:48

refactor: Report the next delay through the property, not a return value

b5e11a4

chore: Remove spec citations and passthrough test helpers

b94c022

tanderson-ld reviewed Sep 15, 2026

View reviewed changes

fix: Guard factory input and keep the next delay positive after a suc…

e0933d8

…cess

tanderson-ld reviewed Sep 15, 2026

View reviewed changes

jsonbailey added 3 commits September 16, 2026 14:54

jsonbailey requested a review from tanderson-ld September 16, 2026 22:00

tanderson-ld approved these changes Sep 17, 2026

View reviewed changes

jsonbailey merged commit e734558 into main Sep 17, 2026
15 checks passed

jsonbailey deleted the jb/sdk-2792/retry-state branch September 17, 2026 13:50

github-actions Bot mentioned this pull request Sep 17, 2026

chore(main): release 9.18.0 #515

Merged

jsonbailey mentioned this pull request Sep 17, 2026

feat: Remove permanent failure modes from FDv1 following RETRY spec #519

Merged

tanderson-ld mentioned this pull request Sep 24, 2026

feat: add a reusable retry state controller for RETRY-conformant backoff launchdarkly/js-core#2045

Merged

Navigation Menu

feat: Add retry state for RETRY-spec backoff - #522

feat: Add retry state for RETRY-spec backoff#522
jsonbailey merged 7 commits into
mainfrom
jb/sdk-2792/retry-state

jsonbailey commented Sep 14, 2026 •

edited by cursor Bot

Loading

Uh oh!

Uh oh!

tanderson-ld Sep 15, 2026

Uh oh!

jsonbailey Sep 16, 2026

Uh oh!

tanderson-ld Sep 15, 2026

Uh oh!

jsonbailey Sep 16, 2026

Uh oh!

Uh oh!

tanderson-ld Sep 15, 2026

Uh oh!

jsonbailey Sep 16, 2026

Uh oh!

tanderson-ld Sep 15, 2026

Uh oh!

jsonbailey Sep 16, 2026

Uh oh!

Uh oh!

Uh oh!

tanderson-ld Sep 15, 2026

Uh oh!

jsonbailey Sep 15, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

jsonbailey commented Sep 14, 2026 • edited by cursor Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

The algorithm

Two things that are easy to get wrong

Why TLS failures are normal

Testing

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

jsonbailey commented Sep 14, 2026 •

edited by cursor Bot

Loading