test(e2e): add chainsaw-native release upgrade testing lane - #3276

Dev-cluster validation of the seed/verify canary commands surfaced two real
bugs, now fixed and re-verified (postgres=3, mariadb=3 rows):

- postgres: the seed writes SQL to psql over a heredoc on stdin, but
  `kubectl exec` without `-i` does not forward stdin, so CREATE/INSERT silently
  never ran. Add `-i`.
- mariadb: the nano server container ships the `mariadb` client (MariaDB 11
  dropped `mysql`) and does not bind 127.0.0.1 — connecting there fails and its
  retry OOM-kills the container. Use the `mariadb` client against the
  mariadb-test-primary Service (writable primary) from mariadb-test-0.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
Signed-off-by: Myasnikov Daniil <[email protected]>

Pre-commit's zizmor audit flags any `>> $GITHUB_ENV` write inside a composite
action as a cross-boundary env write (github-env, "may allow code execution") —
even the identical line passes in a workflow file. Move the SANDBOX_NAME write
back into each job as a one-liner (the pre-composite, zizmor-clean form) and
have the e2e-prepare composite only copy the workspace + run prepare-env.
GITHUB_JOB keeps the e2e and upgrade-e2e sandboxes distinct.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
Signed-off-by: Myasnikov Daniil <[email protected]>

…dfs baseline

The etcd v1alpha2 adoption snapshot cannot succeed in an e2e sandbox: it needs a
trusted-cert (ACME) external S3 endpoint, which a sandbox (example.org, no
DNS/ACME, self-signed in-cluster cert) cannot provide — matching the dev10
findings. Set migrations.etcdAdoptSkipBackup=true in the platform Package so
migration 50 runs the real adoption without the infeasible snapshot. Keep
seaweedfs on the root tenant (the historical default install-cozystack.bats also
uses) for a realistic baseline, but drop the fragile cozy-backups-creds wait —
the adoption no longer depends on the seaweedfs S3/bucket/creds chain.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
Signed-off-by: Myasnikov Daniil <[email protected]>

The composites extracted here were written against the pre-78fc3f917 e2e
job, which copied the checkout to /tmp/$SANDBOX_NAME, ran every step from
there and removed it on teardown. main has since dropped that workspace:
e2e is a single job on an ephemeral runner, so there is no cross-job
rendezvous and the checkout is never reused.

Resolving the rebase in favour of the composites would have reintroduced
it, including the unguarded `rm -rf /tmp/$SANDBOX_NAME` that commit
deliberately deleted rather than guarded — when an upstream step fails the
unconditional `Set sandbox ID` is skipped, SANDBOX_NAME stays empty, and
the `if: always()` removal expands to `rm -rf /tmp/`.

Drop the Prepare/Remove workspace steps and the `cd /tmp/$SANDBOX_NAME`
prefixes from e2e-prepare, e2e-collect and e2e-teardown, and point the
collect and chainsaw-report upload paths at _out/. Same treatment for the
upgrade-e2e job's own two `cd` prefixes.

Signed-off-by: Myasnikov Daniil <[email protected]>

main gained a `labeled` pull_request trigger whose plan/resolve guards
discard every label event except `full-e2e`. This lane's documented opt-in
("any PR a maintainer opts in with the upgrade-e2e label") therefore did
nothing after the rebase: adding the label fired a `labeled` event, `plan`
skipped, `finalize` skipped with it, and upgrade-e2e never satisfied its
needs. Before the rebase there was no `labeled` trigger at all, so the label
was only ever read on push — the gap appeared with the new base, not here.

Admit `upgrade-e2e` alongside `full-e2e` in both allowlists so labelling an
open PR starts a run, and use !cancelled() rather than always() in the job
gate, matching the sibling `e2e` job: a cancelled run must not go on to
occupy a 32-vCPU runner for up to 180 minutes.

Signed-off-by: Myasnikov Daniil <[email protected]>

The previous commit broadened the plan and resolve_assets label gates to admit
upgrade-e2e, but hack/promote-gate-contract.bats pins those two expressions as
exact grep -cF literals. Both counts fell to zero, bats-unit-tests failed, and
because "Unit & controller tests" gates finalize, the required E2E Tests check
and this PR's own upgrade lane were both skipped — the commit meant to make the
label start a run stopped every run instead.

I broke this by running the unit suite before writing that commit rather than
after; pre-commit does not cover bats-unit-tests, so it went unnoticed.

Update both assertions to the broadened expressions, and add a case pinning the
label end to end: admitted by the `labeled` allow-list in both plan and
resolve_assets, honoured by the job's own condition, and named "Upgrade E2E
Test" so it cannot collide with the required "E2E Tests" context and stop being
advisory. Note the label is double-quoted inside the JSON allow-list — matching
the single-quoted form counts zero and asserts nothing.

Verified non-vacuous: narrowing the plan gate back to the full-e2e-only literal
turns exactly one test red.

Also correct the MariaDB seed header, which still prescribed the 127.0.0.1
connection that commit f598c25 replaced — the same file documents that path as
OOM-killing the nano server container.

Signed-off-by: Myasnikov Daniil <[email protected]>

`cut -f2` prints the whole line when that line carries no delimiter, so a
single-component target like "v1" came back with min=1 and was resolved as if
it were v1.1 — walking down to the v1.0 line and printing a real-looking
baseline with exit 0. The lane would then install that baseline and report on
an upgrade path nobody asked for, which is worse than failing.

Pass -s so the delimiter-less line is suppressed, min comes back empty, and the
existing parse guard rejects it. Two-component targets are unaffected: the
guard rejects a missing minor, not a missing patch, and v1.6 still resolves.

In practice the lane only ever passes "" or a target derived from a
release-X.Y.Z branch, so this was latent rather than live.

Signed-off-by: Myasnikov Daniil <[email protected]>

The case I just added was vacuous. _fixture ships no v1.0.x or v1.1.x tag, so
the buggy reading of "v1" as v1.1 walked down to an empty v1.0 line and exited
non-zero for the wrong reason — the assertion held whether or not the defect
was present, which the mutation run showed: reverting -s left all eleven green.

Give the case its own tag set containing v1.0.9, so the buggy path has a real
baseline to return with exit 0 and the case can only pass because the target is
refused. Reverting -s now fails this case and only this case.

Signed-off-by: Myasnikov Daniil <[email protected]>

The upgrade-e2e label opt-in was restored in "make the upgrade-e2e label
opt-in actually start a run" by naming the label in `plan`'s and
`resolve_assets`' allowlists. Both gates still exist and that commit
still applies, but main has since grown three more places that read the
same label, and the opt-in is not live until all of them agree.

The concurrency key routes a `labeled` event into a separate `-label`
group, documented as the exact complement of `plan`'s guard so that only
a run publishing nothing is moved aside. Once `plan` admits upgrade-e2e
the complement is no longer exact: the run publishes, but sat in its own
group, which is two live publishers of the `E2E Tests` context on one
head SHA — the case the comment there warns produces a later green that
erases a real failure.

`e2e-report` carries the same `labeled` guard for the opposite reason,
to stay in lockstep with `plan`. The hazard it documents is running
while `plan` skipped; the hazard here is the reverse. `plan` overwrites
`E2E Tests` with `pending` at the start of every run it executes and
only `e2e-report` concludes it, so admitting the label to one and not
the other opens a pending required status on an upgrade-e2e label and
never closes it, leaving the PR unmergeable until someone pushes again.

`verify-release-candidate` is the third, and the one with teeth.
`e2e-report` needs it and, on a PR carrying the `release` label, posts
`E2E Tests` = failure when its result is anything but success. So an
upgrade-e2e label added to a promote PR would start `plan` and
`e2e-report`, skip the candidate check between them, and stamp a red
over an already-green required status — with no `unlabeled` trigger to
undo it. Admitting the label there keeps the three in step.

The lane stays advisory. `e2e-report` does not gain upgrade-e2e in
`needs`, so the lane's result never moves the verdict, and on an
upgrade-e2e-labelled run the sibling `e2e` job still supplies the real
one.

Assisted-By: Claude <[email protected]>
Signed-off-by: Myasnikov Daniil <[email protected]>

hack/bats-no-exit-trap.bats, added to main after this branch opened, is
red on this file: eleven tests install `trap 'rm -rf "$tmp"' EXIT` and
the file declares no debt, so `bats-unit-tests` fails, which skips
`finalize` and with it both the required `E2E Tests` check and this PR's
own lane.

Convert rather than declare the debt. A handler installed inside an
@test replaces the one the bats binary keeps for its own bookkeeping,
and a test that then fails prints no TAP line at all — not `not ok`,
nothing — so the suite reads green while a case is failing. Removing the
scratch directory as the last statement of the body is the documented
alternative: both runners set -e, so on failure the removal is
unreachable and the directory survives for inspection, which is what a
failed test wants anyway.

The file header now says so, next to the cleanup it explains, so the
next person adding a test here copies the right shape.

Verified: hack/bats-no-exit-trap.bats green, and all eleven cases in
this suite still pass.

Assisted-By: Claude <[email protected]>
Signed-off-by: Myasnikov Daniil <[email protected]>

Review found the case named "line match is dot-anchored (1.5 does not
match v155.x)" does not test that: removing the `sed 's/\./\\./g'`
escaping from latest_on_line leaves all ten cases green.

Checked it, and the reason is narrower than the report. Over the tags
stable_desc emits — `^v[0-9]+\.[0-9]+\.[0-9]+$` and nothing else — the
escaping, the trailing `$` and the `^` are each unfalsifiable: such a tag
carries exactly one `v` and it is at the front, so no fixture can be
built that the escaped and unescaped patterns disagree about. v155.0.0
in particular matches neither reading, so the old fixture pinned nothing
at all.

One piece of the anchoring a fixture CAN reach is the leading `^v` pair.
v11.5.0 ends in "1.5.0" and `sort -rV` puts it above v1.5.3, so a
pattern that has lost that prefix resolves the wrong line entirely.
Swap the fixture to it, rename the case to the property it now pins, and
record in both files what is and is not falsifiable here, so the next
reader does not have to redo the experiment.

Verified non-vacuous: dropping `^v` from latest_on_line's pattern turns
exactly this case red; the other ten stay green.

Assisted-By: Claude <[email protected]>
Signed-off-by: Myasnikov Daniil <[email protected]>

`comm -13 "$dir/unbound-pv.txt" ... || true` turns a missing baseline
file into an empty diff, and an empty diff reads as "nothing newly
broken". A verify phase run without its seed phase therefore reported
the platform healthy having compared nothing at all — the most expensive
kind of green. The sibling suites already guard theirs: verify/vm and
verify/redis fail fast with `[ -f "$before" ] || { ...; exit 1; }`.

Assert both baselines exist before either diff, and drop the `|| true`
that was swallowing the missing-file error along with everything else.

The two `kubectl get` calls had the same shape of hole one layer down: a
pipeline takes the exit status of its LAST command, so `kubectl get pv |
awk ... > file` left an empty list and a passing diff whenever the
kubectl call failed. Write the listing to a file first, where `set -e`
sees the failure.

Assisted-By: Claude <[email protected]>
Signed-off-by: Myasnikov Daniil <[email protected]>

…e image

`curl -sSL ... -o _out/assets/nocloud-amd64.raw.xz` exits 0 on an HTTP
error and writes the response body into the file, so an expired token or
a 404 lands as a "disk image" and surfaces minutes later as a
decompression error that names nothing useful.

Add --fail. The behaviour is unchanged from the code this composite was
extracted from, but it now serves both the e2e and the upgrade lane, and
the release PR path — the one that downloads a draft-release asset with a
token that can expire — is the path the upgrade lane exists for.

Assisted-By: Claude <[email protected]>
Signed-off-by: Myasnikov Daniil <[email protected]>

Both Chainsaw phases take their report name from .chainsaw.yaml, so both
wrote chainsaw-report.xml into the same directory and the workflow always
uploaded it as `upgrade-chainsaw-report`. On the passing path that is the
verify report, which is what the step comment assumed. On the failing
path it is whatever ran last — a lane that dies during seed publishes the
SEED report under the verify name, and the case where someone is actually
reading the file is exactly the failing one.

Rename per phase in the make targets, on the way out, and collect both
names. A phase that never ran leaves no file, which is the answer the
artifact should give. `rm -f` before each run so a report left by an
earlier phase cannot be renamed as this one's.

Assisted-By: Claude <[email protected]>
Signed-off-by: Myasnikov Daniil <[email protected]>

The headroom check derives its job list by matching workflow jobs on
the prepare-env / prepare-cluster make targets, and its coverage
cross-check demands that every workflow file naming packages/core/testing
yield at least one such job. Moving the sandbox bring-up into
.github/actions/e2e-prepare left the target inside the action and only
runs-on in the workflow, so the sweep matched neither the e2e nor the
upgrade-e2e job and the cross-check turned red — exactly the case the
file's header named as its floor and declined to cover.

Both legs now follow a local `uses: ./…` into the action file it names,
and into a local action that action uses in turn, so a job is matched
wherever the make target actually sits and a workflow is read together
with the actions it names. A `uses:` naming a reusable workflow is
deliberately not followed: .github/workflows is already swept file by
file, and the calling job carries no runs-on for this check to size.

The sweep now emits JOB / REF / ERROR records so both legs read one
derivation instead of two that can disagree. A local reference the
strict path pattern cannot read, one that resolves to no readable
action.yaml, and an action that reaches itself are all refusals rather
than silent drops — a marker this cannot open is a lane it cannot see.

Signed-off-by: Myasnikov Daniil <[email protected]>

…s them

The lane's doc listed three of the four shared composite actions, omitting
the one that fetches the Talos disk and the PR patch.

While correcting it, say what the extraction costs elsewhere: the sandbox
bring-up marker that hack/sandbox-runner-headroom.bats matches jobs on now
sits inside an action rather than in the workflow, which is why that guard
follows a local `uses:`. Someone moving another step into a composite
should know the guard has a stake in it.

Assisted-By: Claude <[email protected]>
Signed-off-by: Myasnikov Daniil <[email protected]>