Skip to content

ci: stage all workflows on isolated self-hosted runners - #2565

Draft
seonghobae wants to merge 33 commits into
fix/sdp-ghas-analyses-pagination-20261002from
ci/sdp-all-self-hosted-20261003
Draft

seonghobae wants to merge 33 commits into
fix/sdp-ghas-analyses-pagination-20261002from
ci/sdp-all-self-hosted-20261003

Conversation

@seonghobae

@seonghobae seonghobae commented Oct 3, 2026 •

Copy link
Copy Markdown
Contributor

Current exact-head R runner-label serialization receipt — 2026-10-03

  • Exact head: ab0c865989012c88d2c10c717f6649a76ea49e27; tree 9f5ec08e94e472be5fb0e61ab336322b27047adc; parent f0f7ed988fba5d94eda6513f35d171f23c4bd9a5.
  • Publication: ordinary one-parent fast-forward via Git Data with force=false; no rebase or Force Push.

RCA → RED → GREEN

The reusable R selector inserted caller-controlled matrix.config.os inside a quoted JSON string. A quote-bearing value could introduce duplicate group and labels members; last-member replacement could replace the fixed CWL CI isolated selector boundary (actual runner-group repository ACLs still apply).

The durable RED requires the format slot to be an unquoted JSON value and the argument to use toJSON(matrix.config.os). Exact parent failed 2 tests / 6 passed. The minimal source change serializes the value exactly once; focused GREEN is 8/8, including normal OS labels, duplicate-key text, backslashes, and newlines. Reviewer Minor corrected the hostile ubuntu- fixtures to expect the real linux platform branch.

Exact-tree executable evidence

  • focused workflow + Maturin integration: 25 passed
  • GITHUB_ACTIONS=true workflow contract: 8 passed
  • warning-fatal full suite: 5,439 passed, 10 skipped, 40 subtests
  • production coverage: 18,830/18,830 statements, 7,684/7,684 branches
  • public-doc coverage: 100%; inherited Maturin helper/entry-point gaps were documented after the exact-parent gate exposed them
  • compileall / git diff --check: GREEN
  • independent review after correction: Critical/Important 0/0; the only Minor was repaired
  • local review evidence is not a qualifying GitHub approval

Fresh hosted exact-head evidence

At ab0c8659, SAST 37132539678, Metadata 37132539640, Cloudflare 37132539683, Security 37132539747, and Trusted uv 37132539746 failed before executable steps; every failing job has steps=null and logs_url=null. CodeQL 37132539632 is Draft-skipped. Both review threads are resolved, but all reviews are COMMENTED and qualifying approvals remain 0. This is fail-closed capacity/admission evidence, not a source-test failure or merge authority.

Lifecycle

Draft / Proposed / merge HOLD. The dedicated runner group/capacity, repository access policy, disposable cleanup/canary proof, fresh exact-head hosted Checks, and qualifying independent approval remain required. No runner relabeling, access expansion, bypass, auto-merge, paid/model fallback, or source-neutral rerun is authorized.


Current exact-head provenance repair receipt — 2026-10-03

  • Exact head: f0f7ed988fba5d94eda6513f35d171f23c4bd9a5; tree 3a1407a8bd162566a291a2adef6c7b006a0884ad.
  • Ordinary fast-forward ancestry: RED 665321c169500029ede42112a2aa72b994f1ce16 adds the executable provenance contract; GREEN f0f7ed988fba5d94eda6513f35d171f23c4bd9a5 corrects both canonical receipts. Parent remains reviewed combined head 650bc6e35409d52d24fec890c2ab74063794e9b8; Git Data publication used force=false. No rebase or Force Push occurred.
  • RCA / repair: Git Data publication changed the locally predicted merge object, but CHANGELOG and the Gap baseline retained nonexistent ec7c1b58…. Both now name published two-parent merge 650bc6e35409d52d24fec890c2ab74063794e9b8 (tree 9522f0b9c96247b3412e147bc6f89fd2595c0776) and its 53/53 focused / 5,437 passed, 10 skipped, 40 subtests historical receipt. The new contract requires both records to agree on a resolvable current-ancestry commit with exact parents cd2c3f97… and 22902728….
  • Exact-tree local verification: warning-fatal repository suite 5,439 passed, 10 skipped, 40 subtests passed; dependency-review, ADR-identity, and evidence contracts 24/24 normally and with GITHUB_ACTIONS=true; Python compile, git diff --check, clean worktree, and exact tree match pass. The unavailable local ruff executable was not treated as evidence.
  • Independent review: no blocking finding; provenance, topology, counts, and Proposed/HOLD wording verified. One optional future hardening remains: derive and compare the documented tree SHA rather than treating that tree string as receipt text.
  • Fresh hosted exact-head evidence: CodeQL 37120498641 is Draft-skipped. Security 37120498599, Metadata 37120498646, SAST 37120498634, Trusted uv 37120498647, and Cloudflare 37120498635 failed before executable steps; every materialized failing job has steps=null and logs_url=null. This confirms the absent isolated runner group/capacity admission blocker and does not authorize a blind rerun.
  • Review state: both threads are resolved/outdated; zero unresolved threads and zero qualifying approvals remain.

Lifecycle: Draft / Proposed / merge HOLD. Missing CWL CI isolated group/capacity, disposable cleanup/canary proof, passing exact-head hosted Checks, and a qualifying independent approval still block ordinary merge. No runner relabeling, access expansion, bypass, auto-merge, source-neutral rerun, predecessor closure, or merge is authorized.

Scope

User instruction: move this SDP workstream entirely to self-hosted runners. All 73 central job declarations retain dedicated control/CodeQL/OpenCode boundaries and replace every hosted fallback.

Blocking review finding and RCA

The predecessor exact head d6ba56d04e22476b2ea41bfe7bb26e2cbc98f98f selected ordinary/non-main work with the mutable cwlab-ci-isolated label alone, even though its rollout document correctly stated that a generic label does not attest separation. GitHub runner labels are routing metadata and GitHub does not validate that OS/architecture labels match the machine. A persistent or privileged runner with matching labels and repository access could therefore receive untrusted PR work.

The repair scopes every affected direct, conditional, CodeQL and R-matrix selector through the dedicated CWL CI isolated runner group and the existing platform/isolation labels. The group is the repository-access boundary; labels express required capability inside it. Trusted-main control/CodeQL/OpenCode groups and repository predicates are unchanged.

Historical predecessor evidence — cd49a60f1299fb8f7984de8d8ba223db673f7b84

This section is retained for audit history and is superseded by the current exact-head receipt above.

  • RED: test_isolated_label_is_scoped_to_dedicated_runner_group failed first at .github/workflows/actions-queue-health.yml on the label-only selector.
  • Repair: 63 affected selectors updated; durable all-workflow and workflow-specific contracts updated rather than removed.
  • Focused: 42 passed.
  • Complete repository verification at that predecessor: 5,426 passed, 10 skipped, 40 subtests, exit 0.
  • All 38 workflow YAML files parsed successfully; git diff --check passed.
  • Historical predecessor head: cd49a60f1299fb8f7984de8d8ba223db673f7b84
  • Historical tree: 5cbbaac086e19c94ac37abf2ca8a5914c88eac9b
  • Parent: d6ba56d04e22476b2ea41bfe7bb26e2cbc98f98f (ordinary one-parent fast-forward; no force push or rebase).

Safety and rollout hold

DRAFT / HOLD. The retained organization inventory had no cwlab-ci-isolated capacity and no verified CWL CI isolated group/capacity record. Do not add a privileged runner to this group, relabel privileged runners, widen runner access, or merge until an authorized operator:

  1. creates CWL CI isolated with least-privilege repository access;
  2. provisions disposable isolated Linux/Windows/macOS capacity required by the matrices;
  3. proves cleanup, tool availability, absence of privileged sockets/credentials/internal-network access, and separation from trusted control jobs;
  4. returns a successful canary plus every exact-head required Check.

Source configuration is not runtime activation. No credentials, production deployment, existing runner ACLs, protections, inference budgets or active model runs were changed. Model-backed workflows remain fixed to orchestrator/free; no provider/model/group/paid fallback was added.

Historical hosted evidence — cd49a60f1299fb8f7984de8d8ba223db673f7b84

This predecessor evidence is superseded by the fresh cd2c3f9748e6bc50efd8bdfee9a385a74ec6e3ba run receipt above.

At cd49a60f1299fb8f7984de8d8ba223db673f7b84, Cloudflare DNS, SAST Semgrep, Security Scan, Repository Metadata Reconcile and Trusted uv Materializer completed failure before any step existed; their job step lists are empty and job log blobs are absent. CodeQL PR was skipped. This was fail-closed pre-runner evidence consistent with the missing group/capacity prerequisite, not a passing runtime canary and not merge authorization. CodeRabbit status was success; no qualifying independent review or review threads existed.

The PR must remain Draft/HOLD until capacity and fresh exact-head hosted evidence are available.

Retire the reverse-PR restack path and preserve the six dependency-review owner paths while recording protected main as the second parent.
@coderabbitai

coderabbitai Bot commented Oct 3, 2026

Copy link
Copy Markdown
Contributor

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true
  • Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

seonghobae commented Oct 3, 2026 •

Copy link
Copy Markdown
Contributor Author

Historical circular-admission receipt — 2026-10-03

Superseded: this receipt describes predecessor head d6ba56d04e22476b2ea41bfe7bb26e2cbc98f98f. The authoritative current-head receipt is in the PR body for cd2c3f9748e6bc50efd8bdfee9a385a74ec6e3ba.

  • At that historical head, SAST 37094344652, Trusted uv 37094344666, Repository Metadata 37094344675, Security 37094344655, and Cloudflare DNS 37094344681 remained queued after every non-privileged route was pointed at the unprovisioned cwlab-ci-isolated label. Draft CodeQL 37094344661 was skipped.
  • Canonical prerequisite owners remain linux-cluster-ops#326 for runner registration, repository access and operator evidence, and quarantine-sandbox-runtime#136 / #137 for host-side/LAN denial attestation.
  • Keep Draft. Do not relabel a privileged central/control/OpenCode runner, substitute an arbitrary endpoint, blind-rerun, create a no-op event generation, or mark Ready before the owner prerequisites and canary are real.

@seonghobae seonghobae left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Exact-head review for cd49a60f1299fb8f7984de8d8ba223db673f7b84 (tree 5cbbaac086e19c94ac37abf2ca8a5914c88eac9b):

Blocking predecessor defect repaired: label-only cwlab-ci-isolated routing did not establish the runner access boundary. All 63 isolated selectors now require the dedicated CWL CI isolated group plus their existing labels; trusted-main groups and predicates are preserved. RED was reproduced before repair, focused contracts pass 42/42, all 38 workflows parse, git diff --check passes, and the complete suite passes 5,426 tests with 10 explicit skips and 40 subtests.

This is a COMMENT, not approval. Hosted jobs at this exact head fail before step execution (empty step lists and absent job logs) because verified group/capacity is still missing. Keep Draft/HOLD. Required next evidence is operator-created least-privilege group, disposable capacity and cleanup proof, canary, fresh exact-head required Checks, then eligible independent approval and ordinary protected merge.

@seonghobae seonghobae left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking single-writer/carryover finding on exact head cd2c3f9748e6bc50efd8bdfee9a385a74ec6e3ba: this runner-staging branch changes both canonical Dependency Review workflows without integrating the complete valid owner delta from #1725 exact 229027280e8bce6cc4f13e722b2b0c280a1ac17d. The reusable workflow again masks curl failure with || true and converts HTTP 403/404 into a successful unavailable/skip outcome; it also lacks the immutable base/head and repository-identity preflight. The bundled security-scan.yml likewise lacks #1725's pre-transport identity validation. This is a security regression, not only a stack-order concern.

Keep Draft / Proposed. Repair by ordinary non-force integration of the complete #1725 owner delta into this #2555-based stack, resolving only the runner selectors while preserving #1725 source, executable tests, ADR/doctoring, CHANGELOG, and Gap evidence. Then rerun the combined regression set and fresh exact-head hosted gates; no predecessor check evidence transfers.

Comment thread .github/workflows/dependency-review.yml
Ordinarily integrate .github#1725 source, tests, ADR, doctoring,
CHANGELOG, and Gap evidence into the isolated-runner stack while
retaining the CWL CI isolated group selectors. Repair colliding ADR
identifiers as ADR-0033/0034 and add a repository-wide uniqueness
contract.

@seonghobae seonghobae left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Exact-head review for 650bc6e35409d52d24fec890c2ab74063794e9b8 (tree 9522f0b9c96247b3412e147bc6f89fd2595c0776): ordinary two-parent integration preserves prior #2565 and complete #1725 ancestry. The reusable and bundled Dependency Review paths now fail closed on mutable/malformed identity, nonzero curl exit, and every non-200 result while all 63 isolated selectors retain the CWL CI isolated group and isolation labels. ADR identifier collisions discovered during review are repaired as ADR-0033/0034 with a repository-wide uniqueness regression. Focused contracts pass 53/53 in normal and GitHub Actions environments; 38 workflows parse; compile and diff checks pass; the warning-fatal suite passes 5,437 tests with 10 skips and 40 subtests. Independent final review reports no Critical/Important/Minor finding, and the carryover thread is resolved.

This is a COMMENT, not approval. Exact-head hosted jobs failed before executable steps because the isolated runner group/capacity prerequisite remains absent; CodeQL is Draft-skipped, and there is no qualifying approval. Keep Draft / Proposed / HOLD. Do not rerun blindly, widen runner access, relabel privileged capacity, bypass checks, auto-merge, close predecessors, or merge.

@seonghobae seonghobae left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Exact-head provenance finding on 650bc6e35409d52d24fec890c2ab74063794e9b8: the security source/test carryover is present and the prior thread is resolved, but CHANGELOG names nonexistent commit ec7c1b58ba6a3c61a0d09e02d93ef470f12db98d as the ordinary merge. GitHub returns no commit for that object ID, while the published two-parent integration is current head 650bc6e35409d52d24fec890c2ab74063794e9b8 with parents cd2c3f9748e6bc50efd8bdfee9a385a74ec6e3ba and 229027280e8bce6cc4f13e722b2b0c280a1ac17d. Correct the canonical evidence and any bound Gap/doctoring references test-first or with an executable exact-commit/ancestry contract. Keep Draft/HOLD; no source-behavior rollback or predecessor evidence transfer.

Comment thread CHANGELOG.md Outdated

@seonghobae seonghobae left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Exact-head review for f0f7ed988fba5d94eda6513f35d171f23c4bd9a5 (tree 3a1407a8bd162566a291a2adef6c7b006a0884ad): RED 665321c169500029ede42112a2aa72b994f1ce16 adds an executable contract requiring CHANGELOG and the product/technical Gap baseline to name the same resolvable ancestor with exact parents cd2c3f9748e6bc50efd8bdfee9a385a74ec6e3ba and 229027280e8bce6cc4f13e722b2b0c280a1ac17d; GREEN corrects both records to published merge 650bc6e35409d52d24fec890c2ab74063794e9b8. Fresh GitHub commit evidence independently confirms that merge's tree 9522f0b9c96247b3412e147bc6f89fd2595c0776 and parent order. No blocking source/evidence finding remains. Non-blocking hardening: a future contract may parse and compare the documented tree SHA rather than leaving it as receipt text.

This is a COMMENT, not approval. Exact-head Security, SAST, Metadata, Trusted uv, and Cloudflare runs remain terminal pre-step failures; CodeQL is Draft-skipped, qualifying approvals are zero, and isolated runner group/capacity plus canary evidence remain absent. Keep Draft / Proposed / merge HOLD; no blind rerun, access expansion, bypass, predecessor retirement, or merge.

@seonghobae seonghobae left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Exact-head COMMENT for ab0c865989012c88d2c10c717f6649a76ea49e27 / tree 9f5ec08e94e472be5fb0e61ab336322b27047adc.

RCA: caller-controlled matrix.config.os was interpolated inside a quoted JSON string, so quote-bearing input could introduce duplicate group/labels members. The minimal repair changes the format slot to an unquoted JSON value and supplies toJSON(matrix.config.os), retaining the original value as one data label.

RED on parent: 2 failed, 6 passed. GREEN: workflow contract 8/8 normally and with GITHUB_ACTIONS=true; workflow + Maturin integration 25/25; warning-fatal repository suite 5,439 passed, 10 skipped, 40 subtests; 18,830/18,830 statements and 7,684/7,684 branches; public-doc 100%; compileall/diff check GREEN.

Independent review found one Minor: hostile ubuntu- fixtures must expect the real linux platform branch. It was corrected and reverified. Critical/Important remain 0.

This COMMENT is not approval. PR remains Draft/HOLD pending operator runner-group/capacity/cleanup canary, fresh exact-head hosted checks, and qualifying independent approval.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Todo

Development

Successfully merging this pull request may close these issues.

1 participant