ci: stage all workflows on isolated self-hosted runners - #2565
seonghobae wants to merge 33 commits into
Conversation
Retire the reverse-PR restack path and preserve the six dependency-review owner paths while recording protected main as the second parent.
|
Important Draft PR not reviewedDraft PRs are not automatically reviewed by default.
To automatically review draft PRs, update your CodeRabbit configuration: reviews:
auto_review:
drafts: true
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Historical circular-admission receipt — 2026-10-03
|
seonghobae
left a comment
There was a problem hiding this comment.
Exact-head review for cd49a60f1299fb8f7984de8d8ba223db673f7b84 (tree 5cbbaac086e19c94ac37abf2ca8a5914c88eac9b):
Blocking predecessor defect repaired: label-only cwlab-ci-isolated routing did not establish the runner access boundary. All 63 isolated selectors now require the dedicated CWL CI isolated group plus their existing labels; trusted-main groups and predicates are preserved. RED was reproduced before repair, focused contracts pass 42/42, all 38 workflows parse, git diff --check passes, and the complete suite passes 5,426 tests with 10 explicit skips and 40 subtests.
This is a COMMENT, not approval. Hosted jobs at this exact head fail before step execution (empty step lists and absent job logs) because verified group/capacity is still missing. Keep Draft/HOLD. Required next evidence is operator-created least-privilege group, disposable capacity and cleanup proof, canary, fresh exact-head required Checks, then eligible independent approval and ordinary protected merge.
seonghobae
left a comment
There was a problem hiding this comment.
Blocking single-writer/carryover finding on exact head cd2c3f9748e6bc50efd8bdfee9a385a74ec6e3ba: this runner-staging branch changes both canonical Dependency Review workflows without integrating the complete valid owner delta from #1725 exact 229027280e8bce6cc4f13e722b2b0c280a1ac17d. The reusable workflow again masks curl failure with || true and converts HTTP 403/404 into a successful unavailable/skip outcome; it also lacks the immutable base/head and repository-identity preflight. The bundled security-scan.yml likewise lacks #1725's pre-transport identity validation. This is a security regression, not only a stack-order concern.
Keep Draft / Proposed. Repair by ordinary non-force integration of the complete #1725 owner delta into this #2555-based stack, resolving only the runner selectors while preserving #1725 source, executable tests, ADR/doctoring, CHANGELOG, and Gap evidence. Then rerun the combined regression set and fresh exact-head hosted gates; no predecessor check evidence transfers.
Ordinarily integrate .github#1725 source, tests, ADR, doctoring, CHANGELOG, and Gap evidence into the isolated-runner stack while retaining the CWL CI isolated group selectors. Repair colliding ADR identifiers as ADR-0033/0034 and add a repository-wide uniqueness contract.
seonghobae
left a comment
There was a problem hiding this comment.
Exact-head review for 650bc6e35409d52d24fec890c2ab74063794e9b8 (tree 9522f0b9c96247b3412e147bc6f89fd2595c0776): ordinary two-parent integration preserves prior #2565 and complete #1725 ancestry. The reusable and bundled Dependency Review paths now fail closed on mutable/malformed identity, nonzero curl exit, and every non-200 result while all 63 isolated selectors retain the CWL CI isolated group and isolation labels. ADR identifier collisions discovered during review are repaired as ADR-0033/0034 with a repository-wide uniqueness regression. Focused contracts pass 53/53 in normal and GitHub Actions environments; 38 workflows parse; compile and diff checks pass; the warning-fatal suite passes 5,437 tests with 10 skips and 40 subtests. Independent final review reports no Critical/Important/Minor finding, and the carryover thread is resolved.
This is a COMMENT, not approval. Exact-head hosted jobs failed before executable steps because the isolated runner group/capacity prerequisite remains absent; CodeQL is Draft-skipped, and there is no qualifying approval. Keep Draft / Proposed / HOLD. Do not rerun blindly, widen runner access, relabel privileged capacity, bypass checks, auto-merge, close predecessors, or merge.
seonghobae
left a comment
There was a problem hiding this comment.
Exact-head provenance finding on 650bc6e35409d52d24fec890c2ab74063794e9b8: the security source/test carryover is present and the prior thread is resolved, but CHANGELOG names nonexistent commit ec7c1b58ba6a3c61a0d09e02d93ef470f12db98d as the ordinary merge. GitHub returns no commit for that object ID, while the published two-parent integration is current head 650bc6e35409d52d24fec890c2ab74063794e9b8 with parents cd2c3f9748e6bc50efd8bdfee9a385a74ec6e3ba and 229027280e8bce6cc4f13e722b2b0c280a1ac17d. Correct the canonical evidence and any bound Gap/doctoring references test-first or with an executable exact-commit/ancestry contract. Keep Draft/HOLD; no source-behavior rollback or predecessor evidence transfer.
seonghobae
left a comment
There was a problem hiding this comment.
Exact-head review for f0f7ed988fba5d94eda6513f35d171f23c4bd9a5 (tree 3a1407a8bd162566a291a2adef6c7b006a0884ad): RED 665321c169500029ede42112a2aa72b994f1ce16 adds an executable contract requiring CHANGELOG and the product/technical Gap baseline to name the same resolvable ancestor with exact parents cd2c3f9748e6bc50efd8bdfee9a385a74ec6e3ba and 229027280e8bce6cc4f13e722b2b0c280a1ac17d; GREEN corrects both records to published merge 650bc6e35409d52d24fec890c2ab74063794e9b8. Fresh GitHub commit evidence independently confirms that merge's tree 9522f0b9c96247b3412e147bc6f89fd2595c0776 and parent order. No blocking source/evidence finding remains. Non-blocking hardening: a future contract may parse and compare the documented tree SHA rather than leaving it as receipt text.
This is a COMMENT, not approval. Exact-head Security, SAST, Metadata, Trusted uv, and Cloudflare runs remain terminal pre-step failures; CodeQL is Draft-skipped, qualifying approvals are zero, and isolated runner group/capacity plus canary evidence remain absent. Keep Draft / Proposed / merge HOLD; no blind rerun, access expansion, bypass, predecessor retirement, or merge.
seonghobae
left a comment
There was a problem hiding this comment.
Exact-head COMMENT for ab0c865989012c88d2c10c717f6649a76ea49e27 / tree 9f5ec08e94e472be5fb0e61ab336322b27047adc.
RCA: caller-controlled matrix.config.os was interpolated inside a quoted JSON string, so quote-bearing input could introduce duplicate group/labels members. The minimal repair changes the format slot to an unquoted JSON value and supplies toJSON(matrix.config.os), retaining the original value as one data label.
RED on parent: 2 failed, 6 passed. GREEN: workflow contract 8/8 normally and with GITHUB_ACTIONS=true; workflow + Maturin integration 25/25; warning-fatal repository suite 5,439 passed, 10 skipped, 40 subtests; 18,830/18,830 statements and 7,684/7,684 branches; public-doc 100%; compileall/diff check GREEN.
Independent review found one Minor: hostile ubuntu- fixtures must expect the real linux platform branch. It was corrected and reverified. Critical/Important remain 0.
This COMMENT is not approval. PR remains Draft/HOLD pending operator runner-group/capacity/cleanup canary, fresh exact-head hosted checks, and qualifying independent approval.
Current exact-head R runner-label serialization receipt — 2026-10-03
ab0c865989012c88d2c10c717f6649a76ea49e27; tree9f5ec08e94e472be5fb0e61ab336322b27047adc; parentf0f7ed988fba5d94eda6513f35d171f23c4bd9a5.force=false; no rebase or Force Push.RCA → RED → GREEN
The reusable R selector inserted caller-controlled
matrix.config.osinside a quoted JSON string. A quote-bearing value could introduce duplicategroupandlabelsmembers; last-member replacement could replace the fixedCWL CI isolatedselector boundary (actual runner-group repository ACLs still apply).The durable RED requires the format slot to be an unquoted JSON value and the argument to use
toJSON(matrix.config.os). Exact parent failed 2 tests / 6 passed. The minimal source change serializes the value exactly once; focused GREEN is 8/8, including normal OS labels, duplicate-key text, backslashes, and newlines. Reviewer Minor corrected the hostileubuntu-fixtures to expect the reallinuxplatform branch.Exact-tree executable evidence
GITHUB_ACTIONS=trueworkflow contract: 8 passedgit diff --check: GREENFresh hosted exact-head evidence
At
ab0c8659, SAST37132539678, Metadata37132539640, Cloudflare37132539683, Security37132539747, and Trusted uv37132539746failed before executable steps; every failing job hassteps=nullandlogs_url=null. CodeQL37132539632is Draft-skipped. Both review threads are resolved, but all reviews are COMMENTED and qualifying approvals remain 0. This is fail-closed capacity/admission evidence, not a source-test failure or merge authority.Lifecycle
Draft / Proposed / merge HOLD. The dedicated runner group/capacity, repository access policy, disposable cleanup/canary proof, fresh exact-head hosted Checks, and qualifying independent approval remain required. No runner relabeling, access expansion, bypass, auto-merge, paid/model fallback, or source-neutral rerun is authorized.
Current exact-head provenance repair receipt — 2026-10-03
f0f7ed988fba5d94eda6513f35d171f23c4bd9a5; tree3a1407a8bd162566a291a2adef6c7b006a0884ad.665321c169500029ede42112a2aa72b994f1ce16adds the executable provenance contract; GREENf0f7ed988fba5d94eda6513f35d171f23c4bd9a5corrects both canonical receipts. Parent remains reviewed combined head650bc6e35409d52d24fec890c2ab74063794e9b8; Git Data publication usedforce=false. No rebase or Force Push occurred.ec7c1b58…. Both now name published two-parent merge650bc6e35409d52d24fec890c2ab74063794e9b8(tree9522f0b9c96247b3412e147bc6f89fd2595c0776) and its 53/53 focused / 5,437 passed, 10 skipped, 40 subtests historical receipt. The new contract requires both records to agree on a resolvable current-ancestry commit with exact parentscd2c3f97…and22902728….GITHUB_ACTIONS=true; Python compile,git diff --check, clean worktree, and exact tree match pass. The unavailable localruffexecutable was not treated as evidence.37120498641is Draft-skipped. Security37120498599, Metadata37120498646, SAST37120498634, Trusted uv37120498647, and Cloudflare37120498635failed before executable steps; every materialized failing job hassteps=nullandlogs_url=null. This confirms the absent isolated runner group/capacity admission blocker and does not authorize a blind rerun.Lifecycle: Draft / Proposed / merge HOLD. Missing
CWL CI isolatedgroup/capacity, disposable cleanup/canary proof, passing exact-head hosted Checks, and a qualifying independent approval still block ordinary merge. No runner relabeling, access expansion, bypass, auto-merge, source-neutral rerun, predecessor closure, or merge is authorized.Scope
User instruction: move this SDP workstream entirely to self-hosted runners. All 73 central job declarations retain dedicated control/CodeQL/OpenCode boundaries and replace every hosted fallback.
Blocking review finding and RCA
The predecessor exact head
d6ba56d04e22476b2ea41bfe7bb26e2cbc98f98fselected ordinary/non-main work with the mutablecwlab-ci-isolatedlabel alone, even though its rollout document correctly stated that a generic label does not attest separation. GitHub runner labels are routing metadata and GitHub does not validate that OS/architecture labels match the machine. A persistent or privileged runner with matching labels and repository access could therefore receive untrusted PR work.The repair scopes every affected direct, conditional, CodeQL and R-matrix selector through the dedicated
CWL CI isolatedrunner group and the existing platform/isolation labels. The group is the repository-access boundary; labels express required capability inside it. Trusted-main control/CodeQL/OpenCode groups and repository predicates are unchanged.Historical predecessor evidence —
cd49a60f1299fb8f7984de8d8ba223db673f7b84This section is retained for audit history and is superseded by the current exact-head receipt above.
test_isolated_label_is_scoped_to_dedicated_runner_groupfailed first at.github/workflows/actions-queue-health.ymlon the label-only selector.git diff --checkpassed.cd49a60f1299fb8f7984de8d8ba223db673f7b845cbbaac086e19c94ac37abf2ca8a5914c88eac9bd6ba56d04e22476b2ea41bfe7bb26e2cbc98f98f(ordinary one-parent fast-forward; no force push or rebase).Safety and rollout hold
DRAFT / HOLD. The retained organization inventory had no
cwlab-ci-isolatedcapacity and no verifiedCWL CI isolatedgroup/capacity record. Do not add a privileged runner to this group, relabel privileged runners, widen runner access, or merge until an authorized operator:CWL CI isolatedwith least-privilege repository access;Source configuration is not runtime activation. No credentials, production deployment, existing runner ACLs, protections, inference budgets or active model runs were changed. Model-backed workflows remain fixed to
orchestrator/free; no provider/model/group/paid fallback was added.Historical hosted evidence —
cd49a60f1299fb8f7984de8d8ba223db673f7b84This predecessor evidence is superseded by the fresh
cd2c3f9748e6bc50efd8bdfee9a385a74ec6e3barun receipt above.At
cd49a60f1299fb8f7984de8d8ba223db673f7b84, Cloudflare DNS, SAST Semgrep, Security Scan, Repository Metadata Reconcile and Trusted uv Materializer completed failure before any step existed; their job step lists are empty and job log blobs are absent. CodeQL PR was skipped. This was fail-closed pre-runner evidence consistent with the missing group/capacity prerequisite, not a passing runtime canary and not merge authorization. CodeRabbit status was success; no qualifying independent review or review threads existed.The PR must remain Draft/HOLD until capacity and fresh exact-head hosted evidence are available.