test(e2e): stop blocking merges on the tenant node-join deadline - #3932

## What this PR does

The tenant node-join wait in the two kubernetes e2e suites has stopped
discriminating. Nested virtualisation on the shared runners degrades far
enough that a worker which registers in minutes elsewhere can miss any
deadline this test can afford, so a red on that one wait says as much
about the machine the run landed on as about the product. What it
produces in practice is a re-run, and a signal that teaches people to
re-run is a signal that gets the meaningful reds re-run too.

The suite still fails on it. That is the part worth reading twice,
because it is what separates this from hiding a flake: the script exits
non-zero, Chainsaw records a failure, and its whole catch runs
(previous-instance container logs, the host snapshot, the data-plane
capture, the event dump), every one of which is keyed on the test
failing. The JUnit report says a test failed. What changes is one layer
up. The wait writes a `SOFT-RED-node-join.txt` marker beside its
diagnostics, and a new lane-side gate, `hack/e2e-node-join-soft-red.sh`,
reads that marker to decide whether the job blocks. Softening inside the
suite would have bought a green check by throwing all of that evidence
away.

The gate opens on one shape only, every failed test in the run carrying
that marker, and blocks on everything else, including every way it can
fail to answer the question: a report it cannot read, a failed test the
report does not name, a failure with no marker, a run whose report
records no failure at all. A marker is written only when `timeout`
reports 124 on that one wait, so an assertion after the join, or the
wait failing to run, stays an ordinary red. What the marker attributes
is the test rather than the individual failure: Chainsaw writes one
testcase per Test and combines that Test's step errors into its single
failure element, so a second step error rides soft alongside the
deadline, while a failed `catch`, `finally` or step-level `cleanup`
rides soft for a different reason worth keeping apart, since at the
pinned version those run with no step report and never reach the failure
element at all. The convention doc states both rather than working
around the granularity, since the suite stops at the wait and nothing it
exists to prove runs after the marker.

Only the lanes that block a merge ask the gate. `pull-requests` and
`e2e-fork` do; `nightly` and `e2e-tag` run the same suites and stay
hard. `e2e-tag` is the release candidate's own e2e check, where a soft
pass would ship a release whose tenant clusters brought up no worker
nodes. `nightly` is the only place an e2e failure reaches anyone who did
not open a pull request, since nothing in this tree sends a notification
anywhere, so the red job is the notification. A unit guard holds that
split in both directions and derives the lane set from the workflows
that run the suite, so a new lane has to be placed on one side or the
other before it passes.

One consequence of a softened red is worth naming rather than
discovering later: the job finishes green, so its result stops answering
whether the suite passed, and it has two kinds of consumer that each
break their own way. A step gated on the job failing silently stops
running. The SSH breakpoint in `pull-requests` is that case, and it is
the only interactive way into a sandbox whose node-join never finished,
so the E2E step records what the gate decided and the breakpoint's
condition reads it back; the step is label-gated, so the session opens
for a PR someone deliberately marked for debugging rather than on every
softened run. A job that reports the result silently starts reporting a
pass. That is the required `E2E Tests` commit status in both
pull-request lanes, and on a softened run it now says the node-join
deadline was missed, that the suite failed, and that the merge is not
blocked on it, instead of saying the suite passed. The handover is four
links long, and a guard reads each of them inside the block it has to
live in rather than anywhere in the lane: the step carrying the gate's
id writes the decision to the output sink, the job that runs the suite
publishes it naming that step, the job that reports the result reads it
off that job, and it branches on the value ahead of the branch that
would otherwise call the run a pass. A link matched file-wide passes
while it sits on a neighbour, and the value is then empty for good with
nothing to say so. The first link is read tighter still, and in both
directions: the write has to be an unconditional statement in the same
branch as the exit that makes the job green, that branch has to fall
through to a written non-zero exit, no other green exit may leave the
step, the step may not be marked unable to fail, and the write may not
appear anywhere else. A write under a condition of its own records the
decision on some runs and not others; an exit that escapes the branch
turns a red the gate REFUSED to soften into a green job with nothing
recorded; and a write on the passing path would report a suite that
passed as one that missed the deadline. What the guard cannot do is
stated with it: it cannot tell one reporter from two, it reads the
branch as text rather than evaluating it, and every block it reads is
delimited by indentation.

What is given up is real. A regression whose only symptom is that
workers never reach Ready no longer blocks a merge, and these two suites
are the only ones in the tree that boot tenant worker nodes. The
mechanism sees a timer, not a cause, so a deterministic node-join
regression rides soft as readily as a slow runner does, and the failed
job stops being the thing that opens a debugging path on its own. That
is a debt rather than a design, and the way out is written down beside
the convention it bends: calibrate the runner canary against real lane
data, then move the tolerance under the canary's alert so the lane
softens a run measured as degraded and blocks one measured as healthy.
Until that calibration exists the canary cannot carry the decision,
which is why it does not carry it today.

The deadline goes from 18m to 29m, and the reason is bounded on purpose.
29m is this test's own budget and nothing else: its clock starts where
the wait starts, after the machinedeployment wait, the LoadBalancer
assignment and the healthz probe, each with a ceiling of its own, while
the chart's timers have been running since the Machine was created. The
autoscaler's `maxNodeProvisionTime` measures from that creation and the
MachineHealthCheck's `nodeStartupTimeout` from the later of creation and
`InfrastructureReady`, restarting at that transition, so the offset is
not fixed and no ordering against either timer is claimed. What the
chart's 20m and 30m give is a scale to pick against: a budget over the
first leaves room for a remediation cycle to fit inside the window,
which is a possibility rather than a guarantee, since a run can spend
the whole window on one worker that never comes back.

The operation ceiling both suites give the script rises with the
deadline (50m to 67m), and the diagnostics phase budget rises inside it
(7m to 8m), because the collectors gated ahead of the serial console
cost more between them than the old budget covered. Both budget guards
read their terms from the sources that set them rather than restating
them. The e2e job cap rises from 180 to 215 minutes in all four lanes,
and its guard is honest about what it can hold: it derives the pair's
operation and catch ceilings from source and requires one figure across
every lane that runs the suite, and it records that the suites' full
summed ceilings (226 minutes counting the `finally` legs and the
storageclass-fallback Test) have never fit any cap in that file,
including at 180. The margin above the derivable bound is empirical
instead: three nightlies that failed both bringup Tests ran their
full-suite e2e job in 149, 158 and 160 minutes under the previous
ceilings, a run of that shape gains at most 12 minutes per suite here,
and 215 clears the resulting projection by about 30 minutes where 180
cleared its measurement by 20. The prose that restates the cap outside
the workflows moves with it, in the Chainsaw config and in the two
report collectors that derive their own headroom from it.

Test coverage: a new unit suite for the gate pins the direction it fails
in (an unreadable report, a failed testcase the report leaves unnamed or
names with the empty string or with blanks only, a failure without a
marker, and a report recording no failure all block), the lane split in
both directions, the four-link handover of the softening decision, and
the job cap. The fixtures put the unnameable case ahead of a named suite
on purpose, because an awk record runs to the next testcase and a case
placed last cannot exercise the leak the tally exists to catch. The
existing node-join suite gains guards that the marker is written only
under the status meaning the deadline expired, that it is written ahead
of the diagnostics phase rather than behind it, and that the deadline
quoted in prose across the tree is the one the wait gives, in words as
well as in minutes. Every one of those guards was checked by mutating
the thing it protects and confirming it, and only it, goes red.

### Screenshots

No UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:

### Release note

```release-note
test(e2e): the tenant node-join deadline still fails its suite and still collects its diagnostics, but no longer blocks a merge in the pull-request lanes; nightly and the release-tag lane stay hard
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

- **New Features**
- E2E checks can now report a successful, non-blocking status when the
only failure is the recognized tenant node-join deadline.
- Failure diagnostics and suite results remain available for these
tolerated failures.

- **Bug Fixes**
- Improved detection and attribution of node-join deadline failures,
avoiding false-positive soft passes.

- **Documentation**
- Updated E2E timing, diagnostic budgets, and failure-handling guidance
to reflect the longer test windows.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

… containers (#4437)

Backport of #4020 to `release-1.6`, without its product changes.

On this branch kubernetes-previous and kubernetes-latest failed on
tenant node-join in 8 of the last 11 E2E runs: tenant workers sit one
virtualization level too deep on the shared runners and get their CSR
signed after the window closes. #4020 fixed that on main by running the
merge-gating lane on Talos containers. This brings the same lane here,
but 1.6 is a patch line and nothing it ships should change, so the four
product changes #4020 carried are replaced by e2e-only steps. Each one
is gated on the container lane (`COZY_LINSTOR_DRBD_ENABLED=false`,
`COZY_E2E_STORAGE_CLASS=local`), QEMU path stays as it was.

- linstor `drbd.enabled`: post-install prep applies its own
`LinstorSatelliteConfiguration` `e2e-no-drbd` that deletes the
drbd-logger sidecar, then reads the live DaemonSets back. The name has
to sort after the chart's `cozystack-plunger` because piraeus merges
configurations by name.
- linstor `--strict-topology`: the last install test patches
csi-provisioner args onto the live `LinstorCluster` and waits for the
rollout, tenant suites and vminstance re-check the flag before they
import.
- kubevirt-cdi importer resources: same test patches
`spec.config.podResourceRequirements` on the `cdi` CR and waits for
CDIConfig to report 4Gi.
- vm-disk immediate binding: vminstance suite annotates the DataVolume
and PVC with `cdi.kubevirt.io/storage.bind.immediate.requested` on
`local`.

Only file outside `hack/` and workflows is
`packages/core/testing/Makefile`, the e2e sandbox driver, same as on
main. OIDC suites are folded into kubernetes-latest like on main.
`e2e-tag.yaml` (rc validation) stays on QEMU, also like main. Backup
preflight, unit-test parallelism and the node-join soft-red from #3932
are not included.

`run-kubernetes.sh` here is 1163 lines against 5782 at the base of
#4020, so the lane parts were ported by hand instead of backporting ~20
diagnostics PRs first. Adapted commits say what was dropped.

### Testing

- `make unit-tests` green, POSIX sh sweep clean.
- Real check is E2E on this PR: install log must show the `e2e-no-drbd`
apply and the strict-topology and CDI patches, and kubernetes-latest,
kubernetes-previous and vminstance must be green.

One assumption: nothing upgrades linstor or kubevirt-cdi after the last
install test, otherwise the live patches get re-rendered away. If that
happens the tenant suites fail on the strict-topology check by name.

```release-note
NONE
```