Skip to content

Retry transient server errors for theme API reads - #8894

Open
lizkenyon wants to merge 1 commit into
mainfrom
liz/retry-transient-theme-api
Open

lizkenyon wants to merge 1 commit into
mainfrom
liz/retry-transient-theme-api

Conversation

@lizkenyon

@lizkenyon lizkenyon commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

WHY are these changes introduced?

Theme reads fail on transient Admin API errors such as INTERNAL_SERVER_ERROR, even when the request is safe to repeat. These failures surface as unexpected CLI crashes.

Resiliency issue: Vault #88461 · GitHub mirror #96105

WHAT is this pull request doing?

  • Retry transient GraphQL and gateway errors for theme reads, with up to three retries and exponential backoff.
  • Only retry query-only documents. Mutations are never retried.
  • Report exhausted failures as handled errors with a recovery message and request ID when available.

How to manually test your changes?

With a local GraphQL fixture or proxy:

  1. Fail a theme read once with HTTP 200 and INTERNAL_SERVER_ERROR, then succeed. Confirm the command recovers.
  2. Fail every attempt. Confirm the final error asks the user to retry and includes the request ID without the raw response.
  3. Fail a mutation with the same error. Confirm it's sent only once.

Verification

  • 142 tests across four focused suites passed, including mixed throttling/transient failures.
  • Changed-file ESLint and CLI-kit type-checking passed.
  • Full E2E and live-store testing were not run.

Checklist

  • I've considered possible cross-platform impacts (Mac, Linux, Windows)
  • I've considered possible documentation changes
  • I've considered analytics changes to measure impact
  • The change is user-facing — I've identified the correct bump type (patch for bug fixes · minor for new features · major for breaking changes) and added a changeset with pnpm changeset add

@lizkenyon lizkenyon changed the title Retry transient server errors in theme API requests Retry transient server errors for theme API reads Oct 14, 2026
@github-actions github-actions Bot added the Area: @shopify/cli @shopify/cli package issues label Oct 14, 2026
@lizkenyon
lizkenyon force-pushed the liz/retry-transient-theme-api branch from 310d05f to e9d2565 Compare October 14, 2026 17:31
@lizkenyon
lizkenyon marked this pull request as ready for review October 14, 2026 17:32
@lizkenyon
lizkenyon requested review from a team as code owners October 14, 2026 17:32
Copilot AI balanced review requested due to automatic review settings October 14, 2026 17:32

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Two theme query paths still bypass the new retry policy and transient-error handling.

1 open finding
What changed in this PR

Adds safe retries and clearer errors for transient theme Admin API read failures.

Changes:

  • Retries query-only transient GraphQL/gateway failures with exponential backoff.
  • Converts exhausted theme failures into handled errors with request IDs.
  • Adds coverage and a patch changeset.
File Description
packages/​cli-kit/​src/​public/​node/​themes/​api.ts Enables theme retries and friendly errors.
packages/​cli-kit/​src/​public/​node/​themes/​api.test.ts Tests theme error handling.
packages/​cli-kit/​src/​public/​node/​api/​graphql.ts Restricts retries to query-only documents.
packages/​cli-kit/​src/​public/​node/​api/​graphql.test.ts Tests query and mutation behavior.
packages/​cli-kit/​src/​private/​node/​api.ts Implements transient-error retries and backoff.
packages/​cli-kit/​src/​private/​node/​api.test.ts Tests retry limits and classification.
packages/​cli-kit/​src/​private/​node/​analytics/​graphql-error-codes.ts Recognizes transient GraphQL codes.
packages/​cli-kit/​src/​private/​node/​analytics/​graphql-error-codes.test.ts Tests transient-code detection.
.changeset/​theme-retry-transient-server-errors.md Documents the user-facing patch.

🧠 Review effort: Balanced


💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

@lizkenyon
lizkenyon force-pushed the liz/retry-transient-theme-api branch from e9d2565 to fd13e7f Compare October 15, 2026 14:45
@graygilmore

Copy link
Copy Markdown
Contributor

@lizkenyon will this conflict with #8897?

@lizkenyon

Copy link
Copy Markdown
Contributor Author

@lizkenyon will this conflict with #8897?

Feel free to merge yours first andI can adjust! @graygilmore

Theme commands (`theme pull`, `theme dev`, ...) crashed on the first
Admin API response carrying `extensions.code: INTERNAL_SERVER_ERROR`
(returned on an HTTP 200), even though these are usually brief backend
failures, such as a database briefly running out of connections, that
succeed when retried. Because theme requests use `handleErrors: false`,
the raw `ClientError` was reported as an unexpected error, with the
whole GraphQL response as its message.

- Add an opt-in `retryTransientServerErrors` request behaviour. When set,
  `INTERNAL_SERVER_ERROR` and HTTP 502/503/504 responses are retried up
  to 3 times with exponential backoff (1s, 2s, 4s). This is a separate,
  smaller budget than rate-limit retries, so an overloaded backend
  isn't hit with many quick retries.
- GraphQL requests only honour the behaviour for documents that contain
  only queries. Mutations are never retried on these errors.
- Enable it for theme API requests.
- If a theme request still fails, throw a friendly AbortError that keeps
  the request ID instead of the raw response. `fetchTheme` no longer
  treats a temporary failure as "theme not found".

Co-authored-by: Elizabeth Kenyon <[email protected]>
@lizkenyon
lizkenyon force-pushed the liz/retry-transient-theme-api branch from fd13e7f to ca64b7b Compare November 3, 2026 14:23
@github-actions

github-actions Bot commented Nov 3, 2026

Copy link
Copy Markdown
Contributor

Differences in type declarations

We detected differences in the type declarations generated by Typescript for this branch compared to the baseline ('main' branch). Please, review them to ensure they are backward-compatible. Here are some important things to keep in mind:

  • Some seemingly private modules might be re-exported through public modules.
  • If the branch is behind main you might see odd diffs, rebase main into this branch.

New type declarations

We found no new type declarations in this PR

Existing type declarations

packages/cli-kit/dist/private/node/api.d.ts
@@ -1,14 +1,24 @@
 import { Headers } from 'form-data';
+import { ClientError } from 'graphql-request';
 export type API = 'admin' | 'storefront-renderer' | 'partners' | 'business-platform' | 'app-management';
 export declare const allAPIs: API[];
-export type NetworkRetryBehaviour = {
+interface CommonRetryBehaviour {
+    recordCommandRetries?: boolean;
+    /**
+     * Retry, with backoff, requests that fail with a transient server-side error: a GraphQL
+     * `INTERNAL_SERVER_ERROR` (usually on an HTTP 200) or an HTTP 502/503/504.
+     *
+     * Only enable this for requests that are safe to repeat. GraphQL requests only honour it for
+     * queries; mutations are never retried on these errors.
+     */
+    retryTransientServerErrors?: boolean;
+}
+export type NetworkRetryBehaviour = ({
     useNetworkLevelRetry: true;
     maxRetryTimeMs: number;
-    recordCommandRetries?: boolean;
-} | {
+} & CommonRetryBehaviour) | ({
     useNetworkLevelRetry: false;
-    recordCommandRetries?: boolean;
-};
+} & CommonRetryBehaviour);
 type RequestOptions<T> = {
     request: () => Promise<T>;
     url: string;
@@ -52,6 +62,15 @@ export declare function isNetworkError(error: unknown): boolean;
  * @returns A boolean indicating if the request was aborted.
  */
 export declare function isAbortedFetchError(error: unknown): boolean;
+/**
+ * Checks whether an error is a transient server-side failure that is likely to succeed if retried:
+ * a GraphQL `INTERNAL_SERVER_ERROR` (Shopify APIs usually return it on an HTTP 200), or an HTTP
+ * 502, 503 or 504.
+ *
+ * @param error - The error to check.
+ * @returns Whether the error is a transient server-side failure.
+ */
+export declare function isTransientServerError(error: unknown): error is ClientError;
 export declare function simpleRequestWithDebugLog<T extends {
     headers: Headers;
     status: number;
packages/cli-kit/dist/public/node/upgrade/types.d.ts
@@ -51,11 +51,11 @@ export declare const upgradeJsonOutputSchema: import("../json-output-schema.js")
     scope: zod.ZodEnum<["global", "local"]>;
 }, "strict", zod.ZodTypeAny, {
     status: "skipped";
-    scope: "local" | "global";
     reason: "development" | "local-autoupgrade" | "dependency-not-found";
+    scope: "local" | "global";
 }, {
     status: "skipped";
-    scope: "local" | "global";
     reason: "development" | "local-autoupgrade" | "dependency-not-found";
+    scope: "local" | "global";
 }>]>>;
 export type UpgradeResult = InferJsonOutputSchema<typeof upgradeJsonOutputSchema>;
\ No newline at end of file
packages/cli-kit/dist/private/node/analytics/graphql-error-codes.d.ts
@@ -36,4 +36,11 @@ export declare function isPermissionCode(code: string | undefined): boolean;
  * @param errors - The `errors` value from a GraphQL response.
  * @returns True when a rate-limit code is present in any error.
  */
-export declare function hasRateLimitCode(errors: unknown): boolean;
\ No newline at end of file
+export declare function hasRateLimitCode(errors: unknown): boolean;
+/**
+ * Whether any GraphQL error carries a transient server-side failure code.
+ *
+ * @param errors - The `errors` value from a GraphQL response.
+ * @returns True when a transient server code is present in any error.
+ */
+export declare function hasTransientServerCode(errors: unknown): boolean;
\ No newline at end of file

@EvilGenius13

Copy link
Copy Markdown
Contributor

Is this code meant solely for GraphQL requests? I don't think this will work for any fetch or shopifyFetch requests.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Area: @shopify/cli @shopify/cli package issues

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants