Stop app security prompts from passing unsafe code as safe - #9107

WHY are these changes introduced?

A team that runs shopify app security check in CI found four agent check prompts that can make an agent skip a real issue. Each prompt either checks the wrong signal or counts a control as protection when the check's shopify.dev page does not:

• MISSING_EMBEDDED_CSP decided whether an app is embedded from app_embed or theme_app_extension. app_embed is not a shopify.app.toml key. In the engine it is a capability for a theme extension with an app embed block, which is a storefront feature. So an embedded admin app with no theme extension was marked not applicable, and the prompt's example quoted app_embed = true.
• SSRF_REQUEST_FORGERY counted "URL scheme is restricted to https only", or an internal-IP blocklist alone, as protection. An https:// URL can still point at an internal host. The docs ask for a host allowlist that also checks the scheme, and for redirect handling.
• SCRIPT_TAG_URL_INJECTION counted "restricted to https" as protection. An attacker's own server serves https. The docs ask for trusted, versioned assets.
• THEME_EXTENSION_XSS exempted all | escape output and all textContent writes in its "Do not report" list. That contradicted step 2, which says HTML escaping is not enough in event handlers, srcdoc, or <script src>. Setting textContent on a <script> element sets the code it runs.

WHAT is this pull request doing?

• MISSING_EMBEDDED_CSP (v4) uses embedded = true from shopify.app*.toml, the signal the engine already reads as embedded_app. It also names @shopify/shopify-app-react-router beside the Remix SDK for addDocumentResponseHeaders.
• SSRF_REQUEST_FORGERY (v2) treats a call with user input as safe only with a host allowlist plus disabled or validated redirects. An https-only scheme or an IP blocklist alone is not enough.
• SCRIPT_TAG_URL_INJECTION (v3) drops the https bullet. It accepts an allowlist of domains the app controls, or a fixed set of trusted, versioned assets. A path on the app's own domain is safe only when no user-controlled content can be served from it.
• THEME_EXTENSION_XSS (v4) limits the escape exemption to HTML text and quoted ordinary attributes, and limits the textContent exemption to elements other than <script>.
• Regenerates checks/embedded.ts, updates the version expectations, and adds a regression test.

How to manually test your changes?

Run shopify app security check with agent checks on an embedded app (embedded = true) that has no theme app extension and does not call addDocumentResponseHeaders. The agent now evaluates MISSING_EMBEDDED_CSP instead of marking it not applicable.

Patch details

• Base: Shopify/cli main at c421dbe844122d551b26d1dacd4e187ae3ed18f8.
• Prerequisites: none.
• Apply: git am < CLISEC-21.patch.

Validation

• pnpm vitest run src/cli/services/app-security-engine/tests/ in packages/app: 22 files, 559 tests passed.
• pnpm vitest run src/cli/services/security src/cli/services/app-security in packages/app: 48 files, 1183 tests passed, 3 skipped.
• pnpm type-check in packages/app: passed.
• eslint and prettier --check on the changed TypeScript and changeset: passed.
• node embed-checks.mjs --check: embedded.ts is up to date.
• The patch applies with git am on the base.

Before and after agent runs

Each check ran against small fixture apps with the old prompt and with the new prompt, 5 times each, on claude-sonnet-latest and claude-haiku-latest. One isolated pass had the CLI's investigation and status rules, one check, and every fixture file inline. This approximates the real CLI flow, where an agent explores a repository. It does not replace it. The table counts runs that gave the correct result, out of 10 (5 per model).

What this shows:

• SCRIPT_TAG_URL_INJECTION: the old prompt missed an https-only check every time. The new prompt catches it every time.
• SSRF_REQUEST_FORGERY: the old prompt missed once. The new prompt never missed.
• MISSING_EMBEDDED_CSP: the reported miss did not reproduce. Both models read embedded = true even though the old prompt points to app_embed. For a non-embedded app with a theme extension, the old prompt gave unresolved or executed in 5 of 10 runs. The new prompt gave not_applicable every time.
• THEME_EXTENSION_XSS: no difference. The old prompt caught all three unsafe fixtures. The change removes the contradiction in the prompt text, but these runs show no change in behavior.
• No new prompt flagged a safe fixture.

Requested by Isaac Roldan [email protected]

Torrent ticket: CLISEC-21