[Backport release-1.6] test(e2e): run the merge-gating lanes on Talos containers - #4437

The srv1-srv3 nodes of both lanes that gate a merge -- the same-repo one
in pull-requests.yaml and the fork one in e2e-fork.yaml -- run as Talos
containers instead of QEMU guests, so a tenant worker sits at L2 rather
than L3. Measured in CI on commits where both substrates ran the same
tree: whole-job wall clock down 24-49 minutes, essentially all of it in
Chainsaw, while the two tenant-Kubernetes suites go from 1143-2040s to
466-809s and stop depending on the runner's vgif flag.

The node-join soft-red gate goes with the substrate, which is what it
was for. It existed because nested virtualisation on the shared runners
degraded far enough that a worker registering in minutes elsewhere could
miss any deadline this test could afford; #3513 established that the
discriminator was the host's kvm_amd vgif flag, and removing a nesting
level removes what it acted on. So hack/e2e-node-join-soft-red.sh, the
SOFT-RED-node-join.txt marker and the soft_red job outputs are gone from
all four workflows, and a missed deadline is an ordinary red again. The
::warning annotation survives, because the reason a run went red should
be legible without opening the log, and so does the 124-versus-everything
-else test, because "the workers were slow" and "the wait never ran"
send a reader to different places.

Done by swapping the substrate inside the existing e2e jobs rather than
by promoting a separate container job. Those jobs carry more than a
substrate -- the GitHub App token, the report-overrun guard, the images
list, the SSH breakpoint -- and promoting a job that never had them
would have dropped four features without saying so.

Three properties of container mode drive the design, and each fails
quietly rather than loudly. machine.kernel.modules is a silent no-op --
kernel_module_spec.go returns early on ModeContainer with no error and
no event -- so the host loads openvswitch and zfs before the nodes start
and the workflow step asserts both rather than trusting them; a missing
module surfaces an hour later looking like a CNI or a storage
regression. Docker gives a privileged container its own tmpfs /dev
seeded once at creation, so /dev/kvm works but ZFS zvols created later
never appear and linstor-csi fails ControllerPublishVolume; the compose
file binds the host devtmpfs, which is also why talosctl cluster create
docker can never carry LINSTOR. And there is no ARP VIP, so
192.168.123.11 stands in for the QEMU lane's .10.

The QEMU machinery stays in the tree and keeps running: nightly.yaml and
e2e-tag.yaml still call `make prepare-env`, so DRBD, the replicated
StorageClass, its Immediate binding mode, the tenant StorageClass
propagation that only applies to remotely-accessible classes, and the
Cozystack Talos node image with its extensions all keep nightly and
release-candidate coverage. None of them is exercised per pull request
any more. That is the trade.

Nothing on the PR path builds or downloads a nocloud disk now, so
build-talos drops the talos-nocloud step and the talos-image artifact.
build-talos itself stays: finalize consumes its digest fragment.

One property this does not fix, recorded beside the runner class because
it will be read off a red run eventually: a container node reports the
HOST's CPU and memory as capacity, and hack/e2e-container-up.sh counters
that with kubelet systemReserved. That corrects the scheduler and not
what a pod reads from /proc, so a workload sizing itself from
/proc/cpuinfo or /proc/meminfo is configured for the whole runner.

Also carried here because the same files carry them: the ghcr.io
pull-through mirror removal (see #4007 -- whichever lands first makes the
other a no-op), the authoritative HelmRelease readiness gate, the
backup-credential preflight, the prepull guard against an image-less
container fragment, and folding the two standalone OIDC suites into the
latest-version tenant cluster.

Adapted for release-1.6: this branch has no e2e-fork.yaml, no soft-red
gate, no ghcr.io mirror and no backup e2e, so those hunks are dropped
together with the tests for helpers that exist only on main; e2e-tag.yaml
stays on QEMU. The platform Package generator carries the API endpoint
and nothing else, because the linstor drbd.enabled and kubevirt-cdi
importerResources overrides need chart values this branch does not ship;
the container lane gets e2e-side substitutes for them in later commits.
run-kubernetes.sh takes the storage-class helper, the local-lane pool
baseline, the scoped drain and the OIDC fold, while cleanup keeps its
best-effort `|| true`. The OIDC suite references in select-install.sh and
the chainsaw README go with the suites here rather than in a later commit.

(cherry picked from commit 186547d)
Signed-off-by: Myasnikov Daniil <[email protected]>
Assisted-by: LLM

Two ways the gate did not do what its arguments promise.

The list read carried no bound. kubectl defaults to no request timeout,
and the read sits above the deadline test at the bottom of the loop, so
against a wedged apiserver it blocks there and timeout_seconds stops
being a ceiling: what arrives is the job timeout as a kill, instead of
the not_ready dump this function exists to print. Bounded with kubectl's
own --request-timeout rather than a `timeout` wrapper, because the unit
suite substitutes a kubectl shell function and `timeout` execs a binary.
The final diagnostic read is bounded the same way.

`now` was sampled before the read, so a read costing more than the poll
interval was priced at zero against the deadline and bought the loop
another whole iteration past it. Sampled after, on both branches.

And the count >= minimum_count half of the gate was pinned by nothing:
every fixture returned 11 or 12 items against a minimum of 11, so the
condition could be deleted outright and all five tests stayed green.
That threshold is the direct replacement for the `wc -l` existence
backstop this branch removes from hack/e2e-install-cozystack.bats -- the
only thing separating "everything is Ready" from "almost nothing got
created", since three HelmReleases that exist are all Ready. Both new
cases were checked by reverting the code they cover.

Signed-off-by: Myasnikov Daniil <[email protected]>
Assisted-By: LLM
(cherry picked from commit ec7f172)
Signed-off-by: Myasnikov Daniil <[email protected]>

…ortable

The readiness gate resets its stability window when a list read fails,
and nothing held that: the existing "transient list error" case fails on
the first read, when there is no window to invalidate, so removing the
reset left the suite green. The new case breaks a read mid-window, where
a gate that kept counting through the gap would accept a release set it
never observed stable across it.

The parallel fan-out contract asked make for --output-sync, which arrived
in GNU Make 4.0 while macOS still ships 3.81 -- so on a contributor's
machine the suite died on an unknown option instead of reporting the
contract. Guarded by version, with the reason in place. CI runs 4.x, so
the contract is still enforced where it gates a merge.

Adapted for release-1.6: the --output-sync guard is dropped together
with hack/unit-test-parallelism.bats, which this branch does not have
because the unit-test fan-out it covers is not backported.

Assisted-By: LLM
(cherry picked from commit 8456fac)
Signed-off-by: Myasnikov Daniil <[email protected]>

.chainsaw.yaml states the rule and says why: an op that expires before
its inner bounds do is a SIGKILL on the process group, which takes out
the diagnostics the failure branch exists to print, while an inner
timeout leaves a partial capture and lets the remaining steps run.

refresh-and-assert-full-health, added on this branch, breaks it. The
poll deadline is 270s, the last in-loop read can start just under it and
run 25s, and the failure branch then reads again for another 25s: 320s
of inner bounds under a 5m op. Raised to 6m.

verify-processes-and-security-context breaks it harder and did so
before this branch: 240s for the process counts, then 120s for
connectionString and 120s for generations.reconciled, sequentially, is
480s under the same 5m. Fixed here rather than filed, because it is the
same defect one step away, found by applying the rule to the neighbour.
Raised to 9m.

The kubernetes-latest ceiling gets the measurement instead of a change.
The OIDC lifecycle folded onto that cluster adds up to 1m + 600s + 1m of
passing-path waits that the 67m op does not price, and raising the op is
not available: a guard holds COZY_OP_CEILING equal to both suites'
timeout, and two suites at 79m plus their teardowns exceed the job's
215-minute cap. On the green run of this branch the whole Test took
569.72s against the 4020s op, with kubernetes-previous at 470.19s. The
ceilings only bind once something is already hung, which is what the
existing residual note (#3666) says; now it says it with a number.

Adapted for release-1.6: the kubernetes-latest op here is 40m and no
guard ties it to another figure, so its note states what it does not
price instead of citing the COZY_OP_CEILING guard, the 67m ceiling and
the main-branch measurement, none of which exist on this branch.

Assisted-By: LLM
(cherry picked from commit b0e0f38)
Signed-off-by: Myasnikov Daniil <[email protected]>

pull-requests.yaml said the QEMU substrate "is not wired into any
workflow now" and that prepare-cluster and hack/e2e-prepare-cluster.bats
stay in the tree for a nightly lane that will carry the coverage. Both
halves are wrong in the direction that costs the most: nightly.yaml and
e2e-tag.yaml already run `make prepare-env`, which calls prepare-cluster,
so anyone trusting that comment deletes live code and takes nightly and
every release-candidate run with it. docs/agents/e2e-testing.md states it
correctly, so the tree disagreed with itself.

hack/e2e-platform-packages.sh said the lane replaces "only LINSTOR with
an otherwise-identical Package". It replaces two, and the kubevirt-cdi
one is not identical: it raises the importer memory limit from CDI's
600M to 4Gi and the request from 60M to 256Mi. The consequence belongs
next to the override -- the merge-gating lane does not exercise the
shipped CDI default, and whether a tenant worker disk imports at 600M is
answered by the QEMU lanes, which take no override.

Also drops the dead half of `resolve_assets`. It resolved
nocloud-amd64.raw.xz and exported disk_id; the container lane boots no
disk and nothing in this workflow reads that output any more, so all the
lookup could still do was fail a release-labelled PR over an asset it
does not use. The job stays -- e2e's release arm gates on its result --
and now proves what is still worth proving, that the draft release
exists. e2e-tag.yaml keeps resolving the asset, because it consumes it.

Adapted for release-1.6: nightly.yaml never runs from this branch, so
the corrected comment names e2e-tag.yaml alone as the lane that still
boots QEMU. The platform-packages comment is dropped because the
generator here replaces no Package, and resolve_assets keeps this
branch's own trigger condition.

Assisted-By: LLM
(cherry picked from commit ad00f35)
Signed-off-by: Myasnikov Daniil <[email protected]>

The teardown ends both merge-gating lanes under `if: always()`, and this
branch made it `exit "$cleanup_rc"`. So a `docker compose down` hiccup, a
busy zpool or a failed rmdir after a fully passing suite posts the
required "E2E Tests" status as a failure, and nothing downstream can tell
that apart from a test that actually broke. The runner is discarded when
the job ends, so there is no next run for the strictness to protect; a
reused host is a developer machine, and an annotation serves it. Loud,
not fatal.

Two comments described mechanisms that are gone. The breakpoint step
still explained reading a soft-red output that this branch removed, so a
maintainer would go looking for it; the node-join deadline now fails its
suite like any other, which is what makes `failure()` sufficient. The
build-talos header still described building the nocloud disk and e2e
needing it, when the job builds neither and the container lane downloads
no disk -- the profile is compiled by nightly and by the tag build's
`make assets` instead, and what a PR still compiles is installer.yaml,
which shares every input with it bar the platform and output format.

And the platform-package override now states its consequence for both
packages it replaces, not only for CDI: the merge-gating lane exercises
neither the shipped CDI default nor `drbd.enabled: true`.

Adapted for release-1.6: the gate here is the "E2E Tests" job itself
rather than a commit status, so the teardown comment says so; the
breakpoint step never had the soft-red condition on this branch and is
left alone; the build-talos header names only the tag build, since
nightly.yaml does not run from this branch; and the e2e-fork.yaml and
platform-packages hunks have nothing to apply to.

Assisted-By: LLM
(cherry picked from commit 62f2234)
Signed-off-by: Myasnikov Daniil <[email protected]>

SANDBOX_NAME is per-checkout and these are not, which reads like an
oversight until you notice that hack/e2e-compose.yaml's container names
and the data-srvN zpools are global too. Two checkouts on one host
therefore share nodes and pools, and the second one's teardown destroys
the first one's storage.

Recorded rather than fixed. CI cannot reach it -- the lane already asks
for 24 vCPU and 72 GiB against a 32 vCPU / 128 GiB runner, so a second
one does not fit beside it -- and isolating it properly means renaming
the pools, which the LINSTOR storage-pool registration reads.

Signed-off-by: Myasnikov Daniil <[email protected]>
Assisted-By: LLM
(cherry picked from commit 00c6eb1)
Signed-off-by: Myasnikov Daniil <[email protected]>

The three checks that System-mode artifacts are gone after the mode
switch were `if kubectl get ... >/dev/null 2>&1`, which reads every
non-zero exit as "the object is gone". An RBAC denial, a timeout or a
missing CRD produces the same exit, and stderr went to /dev/null, so the
suite reported a clean System teardown for never having observed one.

--ignore-not-found is what separates absent from unaskable: absent is
exit 0 with empty output, everything else stays non-zero. Preferred over
matching NotFound in the message, which depends on wording and, under a
runner with tracing on, captures the trace instead of the error.

Each probe is pinned individually rather than as a group: the new case
breaks one query per iteration while the other two answer absent, so
reverting any single probe to the bare form reddens the suite. Verified
by doing exactly that three times. A test that broke all three at once
would have passed on any one surviving probe, which is how the first
attempt at this test fooled me.

Signed-off-by: Myasnikov Daniil <[email protected]>
Assisted-By: LLM
(cherry picked from commit b30c7ef)
Signed-off-by: Myasnikov Daniil <[email protected]>

The linstor chart adds a drbd-logger sidecar to every satellite, and on
the container lane's shared kernel there is no DRBD for it to tail, so
it exits at once. The satellite pod is then never Ready, piraeus never
registers the node, LINSTOR reports no nodes and no PVC binds. main
drops the sidecar through a chart value; the chart on this branch has
none, and the lane must not change what the chart ships.

So post-install prep applies a LinstorSatelliteConfiguration of its own
on the container lane, ahead of the linstor waits, with the same
DaemonSet patch main renders: delete drbd-logger from the generated
satellite DaemonSet. The two DRBD-only initContainers are already gone
through the chart's Talos satellite configuration. Once all three nodes
are Online, prep reads the live DaemonSets back and fails if a
satellite still carries the sidecar or none was found at all, so a
configuration that silently stopped applying cannot pass for one that
worked. The QEMU lane applies nothing and keeps the chart as shipped.

Signed-off-by: Myasnikov Daniil <[email protected]>
Assisted-by: LLM

On the container lane every volume is on `local`, which is node-pinned
and WaitForFirstConsumer. Without --strict-topology external-provisioner
passes every topology segment as requisite and only prefers the node the
scheduler picked, so linstor-csi may provision elsewhere when that node
is full. A CDI importer mounts two such volumes and the scratch one is
created only after the pod is scheduled, so it can land on another node
and leave the importer unschedulable forever with nothing erroring.
main turns the flag on in the linstor chart; the chart on this branch
cannot, and the lane must not change what it ships.

So the last install test, on the container lane only, adds the same
csi-provisioner override to the live LinstorCluster, waits for
piraeus-operator to carry it into linstor-csi-controller and for that
rollout to finish, and asserts the flag on the Deployment. `args` has no
patchMergeKey, so the list is restated in full as piraeus-operator
v2.10.2 renders it. It runs last because the earlier tests that change
platform values upgrade linstor, and an upgrade re-renders the
LinstorCluster without the override. For the same reason the tenant
Kubernetes suites and both vminstance Tests re-check the flag before
they import, so an override lost to a later upgrade fails by name rather
than as an import stuck at ImportScheduled.

Signed-off-by: Myasnikov Daniil <[email protected]>
Assisted-by: LLM

CDI's own 600M worker memory limit was seen to OOM the decompress and
convert of a tenant worker disk near 100% on the container lane, after
which CDI retries from scratch and the DataVolume cycles forever rather
than failing. The same import completes at 600M on QEMU nodes, so what
needs the headroom is the substrate. main passes the raise through a
kubevirt-cdi chart value that this branch does not have.

The last install test now merges the same figures into the CDI CR on
the container lane: 4Gi and 256Mi for memory, CPU left at CDI's 750m and
100m because the failure was memory. It has to be the CR, not CDIConfig,
because the operator reconciles CDIConfig.spec from the CR and reverts a
direct edit, and the test waits for CDIConfig's published default to
read 4Gi, since that is what CDI hands its worker pods. The QEMU lane
keeps CDI's defaults.

Signed-off-by: Myasnikov Daniil <[email protected]>
Assisted-by: LLM

The vminstance suite waits for each VMDisk's DataVolume to be Ready
before anything consumes it. On `replicated` that works because the
class binds Immediate; on the container lane's `local` class, which is
WaitForFirstConsumer, CDI starts no importer for a disk nothing has
consumed yet, so the DataVolume sits in PendingPopulation and the suite
times out with no error anywhere. main makes the vm-disk chart request
immediate binding for every source; the chart on this branch requests it
only for uploads, and the lane must not change what it ships.

So on the container lane, right after each VMDisk is applied, the suite
waits up to 180s for CDI to create the DataVolume and its claim and
annotates both with cdi.kubevirt.io/storage.bind.immediate.requested,
the same request the chart already writes for uploads. Both, because
the DataVolume decides what phase CDI reports and the claim is what the
populator reads to start the import without a consumer. The QEMU lane
annotates nothing.

Signed-off-by: Myasnikov Daniil <[email protected]>
Assisted-by: LLM

The container lane moves out of "In-flight direction (not yet the merged
standard)" and into the body, because it is now what both merge-gating
lanes run. The node-join section is rewritten the other way round: it
described a deadline that failed the suite without blocking the lane,
and there is no longer any such carve-out to describe.

What the section gains is the part that was missing while the lane was a
spike -- what moves off the per-PR path with the substrate, and where it
still gets covered. DRBD and the replicated StorageClass were already
named; Immediate binding, tenant StorageClass propagation and the
Cozystack Talos node image were not, and all three now say nightly and
release-candidate rather than reading as lost. The host CPU and memory
visibility gap is stated too, since it is what a reader meets first on a
red run and it looks like a product bug.

Adapted for release-1.6: this branch never had the node-join soft-red
carve-out, so there is no section to rewrite for it, and nightly.yaml
does not run from here, so the QEMU coverage the section points to is
e2e-tag.yaml alone. The section also describes the e2e-side settings
that stand in for chart values this branch does not have -- the
no-drbd satellite configuration, CSI strict topology, the CDI worker
memory ceiling and immediate binding for VM disks -- and it lands after
them for that reason.

(cherry picked from commit 12157a2)
Signed-off-by: Myasnikov Daniil <[email protected]>
Assisted-by: LLM