Repository navigation
fix(credentials): do not cool down keys for at-capacity errors - #6
Merged
Merged
Conversation
Any terminal upstream failure used to record a 60s credential cooldown, including capacity errors that arrive as statusless SSE error events and transient blips that are retried and eventually succeed. With a single key that benched the only credential, so every request for the next minute failed instantly with NoAvailableCommandCodeCredentialError. Only credential-scoped failures (401/402/403) may start a cooldown now. Provider-scoped failures -- statusless stream error events, 429/5xx responses, empty bodies, and network errors -- release the credential without marking it, while still being retried within the request up to the configured attempt budget. Cooldowns are treated as a preference rather than a gate: attempt zero falls back to an ignoreCooldown selection, so a cooling credential can still serve instead of failing the request. The provider client follows the same release policy. Cover the CommandCode client with regression cases for retry-then-success without a stale cooldown, statusless at-capacity errors, exhausted 429 retries, and serving through an injected cooldown.
Owner
|
Thanks for tracking this down and for the clear write-up — benching the only key for 60s on a transient at-capacity error was a real outage mode for single-key deployments. Verified on our side before merging: rebased onto current Merging as-is. We'll follow up on |
yelixir-dev
added a commit
that referenced
this pull request
Oct 2, 2026
Release PR #6 (do not cool down keys for provider-scoped failures): 429/5xx, empty bodies, stream errors and network errors now retry within the request without benching the credential, so a single-key deployment no longer fails every request for 60s after one at-capacity error. Update the README routing paragraph (en/ko/zh), KNOW_HOW, ARCHITECTURE and both deployment guides, which still described 429/5xx/timeout cooldowns, and record the release in PROCESS_LOG.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fix for the at-capacity cooldown incident on the single-key deployment. Any terminal upstream failure used to call
recordFailure, including at-capacity errors that arrive as statusless SSEerrorevents and transient failures that are retried and eventually succeed. With one credential, that benched the only key for 60s and every request failed in ~1ms withNoAvailableCommandCodeCredentialErroruntil the window expired.Changes
src/commandcode.ts(alpha/alpha/generatepath)isFatalCredFailure) record a failure and start a cooldown; HTTP non-ok 429/5xx responses, empty bodies, streamerrorevents and network errors now callfinalizeRelease()— releasing the in-flight slot without touchingdisabledUntil.errorevents are never recorded as credential faults. They are still retried while no visible event has been emitted, then forwarded to the caller as-is.finalizeCallerAborttofinalizeReleaseand routed the abort path through it, so caller aborts no longer leave a cooldown.NoAvailableCommandCodeCredentialError, selection is retried withignoreCooldown: trueso a cooling credential can still serve instead of failing the request.src/provider.ts(provider/provider/v1/chat/completionspath)finalizeReleaseand made both the!response.okbranch and the thrown-error branch record a failure only for fatal credential statuses (401/402/403); everything else releases.tests/commandcode.test.tsResulting behavior
An at-capacity response now retries within the request (default budget 5 attempts, ~3.75s of backoff worst case). If it still fails, or if visible output was already streamed, the upstream error is forwarded to the caller — an in-band SSE error frame (
commandcode_event_error) for streaming, HTTP 502 JSON for non-streaming — and no cooldown is recorded, so the next request is served normally.Tests
tsc --noEmit, eslint and the build pass.launchctlENOENT).