-
Notifications
You must be signed in to change notification settings - Fork 0
test(hosted): production e2e for scan --mode hosted across every ecosystem and package manager
#141
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Mikola Lysenko (mikolalysenko)
merged 4 commits into
main
from
test/hosted-mode-production-e2e
Jul 31, 2026
Merged
Changes from all commits
Commits
Show all changes
4 commits
Select commit
Hold shift + click to select a range
13e5442
test(hosted): production e2e for `scan --mode hosted` across every ec…
mikolalysenko f201102
fix(ci): unbreak the hosted-e2e job on GitHub runners
mikolalysenko 6f1a1ef
fix(ci): let corepack own pnpm in the hosted-e2e job
mikolalysenko a929b96
fix(ci): run hosted-e2e on Node 24 so pnpm 10 works
mikolalysenko File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -4,6 +4,15 @@ on: | |
| push: | ||
| branches: [main] | ||
| pull_request: | ||
| workflow_dispatch: | ||
| inputs: | ||
| hosted_e2e: | ||
| # Underscored on purpose: `inputs.hosted-e2e` is not valid expression | ||
| # syntax (a hyphenated name needs `inputs['hosted-e2e']`). | ||
| description: 'hosted-e2e: auto (obey vars.HOSTED_E2E_DISABLED) | force | skip' | ||
| type: choice | ||
| default: auto | ||
| options: [auto, force, skip] | ||
|
|
||
| permissions: | ||
| contents: read | ||
|
|
@@ -755,4 +764,194 @@ jobs: | |
| with: | ||
| name: setup-matrix-${{ matrix.ecosystem }} | ||
| path: report-${{ matrix.ecosystem }}.json | ||
|
|
||
| # ---------------------------------------------------------------------- | ||
| # Hosted-mode production e2e — REQUIRED status check, with a kill switch. | ||
| # | ||
| # Drives `scan --mode hosted` against the REAL production endpoints | ||
| # (patches-api.socket.dev + patch.socket.dev) and the REAL upstream | ||
| # registries, using patches that are actually published on production. | ||
| # Nothing is mocked. Every other hosted-mode capstone in this repo | ||
| # (e2e_redirect_*) points at a wiremock stand-in, so this job is the only | ||
| # thing that would notice production drifting away from the CLI. | ||
| # | ||
| # The suite itself is `#[ignore]`-gated, so it stays OUT of the `test` and | ||
| # `e2e` jobs and only runs where it is explicitly asked for — here. | ||
| # | ||
| # INVARIANTS (this job is registered in branch protection as a required | ||
| # check named exactly `hosted-e2e`): | ||
| # * NO job-level `if:` — a *skipped* required check is ambiguous to branch | ||
| # protection and can wedge a PR at "Expected — waiting for status". | ||
| # The kill switch gates the STEPS, never the job. | ||
| # * NO `needs:` — an upstream failure would skip this job, same wedge. | ||
| # * NO matrix and NO rename — the check name must stay `hosted-e2e`. | ||
| # * NO `continue-on-error` — a bypass must be visible, not invisible. | ||
| # The job ALWAYS runs and ALWAYS reaches success or failure. | ||
| # | ||
| # ESCAPE HATCH — when production is down and this is blocking merges: | ||
| # Settings -> Secrets and variables -> Actions -> Variables -> | ||
| # HOSTED_E2E_DISABLED = true | ||
| # then "Re-run failed jobs" on any blocked PR. `vars` is read at job-run | ||
| # time, so no commit and no push is needed; the job goes green with a loud | ||
| # ::warning:: and a BYPASSED banner in the job summary. DELETE the variable | ||
| # to re-arm. For a one-off: Actions -> CI -> Run workflow -> | ||
| # hosted_e2e = force (ignore the variable) | skip (bypass this run). | ||
| # ---------------------------------------------------------------------- | ||
| hosted-e2e: | ||
| name: hosted-e2e # registered in branch protection; do not rename | ||
| runs-on: ubuntu-latest | ||
| permissions: | ||
| contents: read | ||
| timeout-minutes: 30 | ||
| concurrency: | ||
| # These are real requests against a real production service — keep it to | ||
| # one run per ref rather than one per push. | ||
| group: hosted-e2e-${{ github.ref }} | ||
| cancel-in-progress: ${{ github.event_name == 'pull_request' }} | ||
| env: | ||
| HOSTED_E2E_DISABLED: ${{ vars.HOSTED_E2E_DISABLED }} | ||
| # The `inputs` context is empty on push/pull_request, so default to auto. | ||
| HOSTED_E2E_MODE: ${{ (github.event_name == 'workflow_dispatch' && inputs.hosted_e2e) || 'auto' }} | ||
| steps: | ||
| - name: Resolve the kill switch | ||
| id: gate | ||
| run: | | ||
| set -eu | ||
| run=true; reason='' | ||
| case "$HOSTED_E2E_MODE" in | ||
| force) reason='workflow_dispatch hosted_e2e=force (kill switch ignored)' ;; | ||
| skip) run=false; reason='workflow_dispatch hosted_e2e=skip' ;; | ||
| *) if [ "${HOSTED_E2E_DISABLED:-}" = 'true' ]; then | ||
| run=false | ||
| reason='repository variable HOSTED_E2E_DISABLED=true' | ||
| fi ;; | ||
| esac | ||
| echo "run=$run" >> "$GITHUB_OUTPUT" | ||
| if [ "$run" = 'false' ]; then | ||
| echo "::warning title=hosted-e2e BYPASSED::$reason" | ||
| { | ||
| echo '## :warning: hosted-e2e BYPASSED — no production coverage in this run' | ||
| echo | ||
| echo "Reason: $reason" | ||
| echo | ||
| echo 'Re-arm by deleting the HOSTED_E2E_DISABLED repository variable' | ||
| echo '(Settings -> Secrets and variables -> Actions -> Variables).' | ||
| } >> "$GITHUB_STEP_SUMMARY" | ||
| fi | ||
|
|
||
| - name: Checkout | ||
| if: steps.gate.outputs.run == 'true' | ||
| uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 | ||
| with: | ||
| persist-credentials: false | ||
|
|
||
| - name: Install Rust | ||
| if: steps.gate.outputs.run == 'true' | ||
| run: rustup show | ||
|
|
||
| - name: Cache cargo | ||
| if: steps.gate.outputs.run == 'true' | ||
| uses: Swatinem/rust-cache@c19371144df3bb44fab255c43d04cbc2ab54d1c4 # v2.9.1 | ||
| with: | ||
| key: hosted-e2e | ||
| save-if: ${{ github.ref == 'refs/heads/main' }} | ||
|
|
||
| - name: Setup Node.js | ||
| if: steps.gate.outputs.run == 'true' | ||
| uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4 | ||
| with: | ||
| # Node 24, NOT the 20.20.2 the other jobs pin. pnpm 10 imports | ||
| # `node:sqlite` for its store index, which does not exist before | ||
| # Node 22 (and is only stable in 24) — on 20 every `pnpm install` | ||
| # dies with ERR_UNKNOWN_BUILTIN_MODULE. This suite drives real, | ||
| # current package managers against production, so it tracks what | ||
| # users actually run rather than the pin the offline suites need. | ||
| node-version: '24.x' | ||
|
|
||
| - name: Setup npm-family package managers | ||
| if: steps.gate.outputs.run == 'true' | ||
| env: | ||
| # Every corepack shim invocation — including the suite's own | ||
| # `has_command` probes — must be non-interactive, or the probe hangs | ||
| # or exits non-zero and STRICT turns that into a failed leg. | ||
| COREPACK_ENABLE_DOWNLOAD_PROMPT: '0' | ||
| # corepack owns pnpm and both yarn flavors. `corepack enable` alone is | ||
| # not enough: it installs the shims, but `pnpm --version` still fails | ||
| # for a project with no `packageManager` field — exactly the pnpm | ||
| # fixture's shape — because the shim has no version to resolve. The | ||
| # `prepare … --activate` lines set that global default AND pre-download | ||
| # each version, so the first invocation inside a test is not also a | ||
| # network fetch. | ||
| # | ||
| # pnpm must NOT come from `npm install -g` here: corepack has already | ||
| # created its shim at the same path, and npm refuses with EEXIST. Only | ||
| # bun, which corepack does not manage, comes from npm. | ||
| run: | | ||
| set -eu | ||
| corepack enable | ||
| corepack prepare pnpm@10 --activate | ||
| corepack prepare [email protected] --activate | ||
| corepack prepare [email protected] --activate | ||
| npm install -g bun@1 | ||
| node --version | ||
| npm --version | ||
| pnpm --version | ||
| bun --version | ||
|
|
||
| - name: Setup Python + uv | ||
| if: steps.gate.outputs.run == 'true' | ||
| uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5 | ||
| with: | ||
| python-version: '3.12.x' | ||
|
|
||
| - name: Install uv | ||
| if: steps.gate.outputs.run == 'true' | ||
| run: python -m pip install --disable-pip-version-check uv && uv --version | ||
|
|
||
| - name: Setup Ruby | ||
| if: steps.gate.outputs.run == 'true' | ||
| uses: ruby/setup-ruby@319994f95fa847cf3fb3cd3dbe89f6dcde9f178f # v1.295.0 | ||
| with: | ||
| ruby-version: '3.2.10' | ||
| # The gem hosted rewrite pins into the Gemfile.lock CHECKSUMS | ||
| # section, which `bundle lock --add-checksums` only emits on >= 2.6. | ||
| bundler: '2.6' | ||
| bundler-cache: false | ||
|
|
||
| - name: Setup Go | ||
| if: steps.gate.outputs.run == 'true' | ||
| uses: actions/setup-go@4a3601121dd01d1626a1e23e37211e3254c1c06c # v6.4.0 | ||
| with: | ||
| go-version: '1.24' | ||
| cache: false | ||
|
|
||
| - name: Run hosted-mode production e2e | ||
| if: steps.gate.outputs.run == 'true' | ||
| env: | ||
| # A required check must never report green on an unexercised leg: | ||
| # STRICT turns the suite's local "toolchain missing" soft-skips into | ||
| # hard failures. Every toolchain it needs is installed above. | ||
| SOCKET_PATCH_HOSTED_E2E_STRICT: '1' | ||
| COREPACK_ENABLE_DOWNLOAD_PROMPT: '0' | ||
| run: | | ||
| set -u | ||
| # The public proxy intermittently returns 503 "Service temporarily | ||
| # over capacity" — that is the documented reason the older live-API | ||
| # suites were pulled from the PR matrix (see the `e2e` job). Retry the | ||
| # whole suite a couple of times before calling it a real failure, so a | ||
| # transient 503 does not block merges through a required check. | ||
| for attempt in 1 2 3; do | ||
| echo "::group::hosted-e2e attempt $attempt" | ||
| cargo test -p socket-patch-cli --test e2e_hosted_production -- \ | ||
| --ignored --nocapture --test-threads=4 | ||
| status=$? | ||
| echo "::endgroup::" | ||
| if [ "$status" -eq 0 ]; then | ||
| exit 0 | ||
| fi | ||
| echo "::warning title=hosted-e2e attempt $attempt failed::retrying" | ||
| sleep $((attempt * 20)) | ||
| done | ||
| echo "::error title=hosted-e2e::suite failed on all 3 attempts" | ||
| exit 1 | ||
|
mikolalysenko marked this conversation as resolved.
|
||
| if-no-files-found: warn | ||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.