Skip to content

Test every guardrail limit and stop flagging dead-letter queues - #267

Merged
matuscongrady merged 10 commits into
mainfrom
test/sweep2-t25
Oct 21, 2026
Merged

matuscongrady merged 10 commits into
mainfrom
test/sweep2-t25

Conversation

@matuscongrady

@matuscongrady matuscongrady commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

This is the second testing sweep for security inventories, guardrails and posture (slices S61, S62, S67). I broke the product code on purpose, one realistic fault at a time, and counted which faults a test noticed. The score goes from 34/52 to 52/52 across the three slices. The biggest gap was guardrails, at 2/14: stage, region, resource-type, container, engine, instance-size (read replicas), custom-domain, resource-count and dead-letter-queue limits could all be broken without any test failing. The 145-file CLI source suite also passed with every one of those faults applied.

The guardrail spec now has one table-driven scenario through the real ConfigManager. Each limit has to admit its boundary and the defaults normalization applies, block the first configuration past it, and pass the schema the Console uses when an admin saves the policy. The posture spec now includes an OpenSearch domain left at its default internet accessibility. The Trivy spec checks that a rejected download leaves no staging files behind. The MiniStack inventory scenario now deploys a fourth time, so choosing the oldest earlier inventory instead of the newest no longer goes unnoticed.

While writing the guardrail scenario I found a bug, which this PR fixes. The require-dead-letter-queue guardrail and the queue-without-dead-letter-queue posture rule also asked the dead-letter queue itself for a redrivePolicy. An organization with that guardrail could only deploy a queue with a dead-letter queue by making the two queues redrive into each other. The Console's suggested fix (add a dead-letter queue) also created a new finding for the queue it added. Queues that a stack queue names as its redrive target are now exempt from both rules. Both specs reproduced the bug before the fix.

I deleted the STACKTAPE_TRIVY_PATH unit test. The MiniStack inventory scenario runs through that override and fails when it is ignored. The Console side is stacktape/console-app#115. This PR does not move the Console pointer.

Checked locally (CI is off): cli-security-rules, the MiniStack inventory and deploy lanes, the CLI source suite (151 files) and smoke test, test:db --security and --insights, test:tools, and pnpm check:integrated up to turbo run test. That run stopped on two failures that also happen on clean origin/main, outside these slices: helper-lambda-sdk.spec.ts hits its 120 s timeout because packaging now compiles the code-swap benchmark three times, and fast-deploy.spec.ts counts 3 upload timeouts in the shared-layer scenario. I ran the gate's remaining steps on their own (build, tool types, pack, architecture, dead code, duplicates) and they pass. ministack-deploy passed except for that same fast-deploy.spec.ts case.

After review: the guardrail docs still said every queue needs a redrivePolicy, so they now describe the dead-letter queue exemption, and the generated LLM docs corpus is regenerated to match. The branch is merged with the latest main. pnpm check:integrated stops at the same helper-lambda-sdk.spec.ts timeout as before, which also happens on main. Everything before it passes, including docs and generation checks, and so do the later steps (build, tool types, pack, architecture, dead code, duplicates) run on their own.

matuscongrady and others added 10 commits October 21, 2026 19:13
…[skip ci]

The require-dead-letter-queue guardrail and the queue-without-dead-letter-queue
posture rule asked every queue for a redrivePolicy, including the queue another
queue sends its failures to. An organization with the guardrail could only deploy
a queue with a dead-letter queue by making the two queues redrive into each
other, and the Console's suggested fix (add a dead-letter queue) produced a new
finding for the queue it added. A queue named as a stack queue's redrive target
is now exempt from both rules.

The guardrail spec now covers every limit the CLI enforces: each one admits its
boundary and the defaults normalization applies, blocks the first configuration
past it, and goes through the schema the Console uses when an admin saves it.
Before this, stage, region, resource-type, memory/timeout, container, engine,
instance-size (read replicas), custom-domain, resource-count and DLQ guardrails
could be broken without any test failing. The posture spec now covers an
OpenSearch domain left at its default internet accessibility, and the Trivy
spec checks that a rejected download leaves no staging files behind.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
…Trivy test [skip ci]

The MiniStack inventory scenario only ever had one earlier inventory in the
bucket, so picking the oldest instead of the newest went unnoticed. It now
deploys once more with the third deployment's packages and requires that
inventory to be reused.

The STACKTAPE_TRIVY_PATH unit test is deleted: the MiniStack inventory scenario
runs through that override and fails when it is ignored.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
…olicy [skip ci]

The require-dead-letter-queue pages still said every queue needs a
redrivePolicy. Since the previous commit, a queue that another stack queue
names as its redrive target is exempt, which is also what the example on the
page relies on.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@matuscongrady
matuscongrady merged commit 883699b into main Oct 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant