test(e2e): warm the ghcr.io mirror before the tenant suites need it - #3768

Diagnostics, the injection guard and the best-effort path all hold up. I traced all seven kubectl failure modes by execution and every one returns 0 under set -eu, and the Job still exits non-zero on missed tags, so both halves of that claim are true at once. Tag selection matches what the suites actually do (sort_by(.) | .[-1] / .[-2]) against the real versions.yaml. 35 mutations against ghcr-mirror.sh, 25 caught, so the tests are real.

Requesting changes for the numbers and one test gap. Most are inline, two could not be anchored.

Budget worst case is 175s, not 150s. Reads go 5 to 7 and each one is timeout -k 5 20 (run-kubernetes.sh:1055-1056), so with the grace the ceiling is 125 to 175, not 100 to 150. Also cozy_diag_phase_has_time is a start gate, so the extra ~50s does not come from nowhere, it comes out of talos_image_cache_diagnose being declined more often. Worth saying that in the body instead of "stays inside the phase budget gate", which is true of the check but not of the phase.

ghcr_mirror_diagnose declares in its own header that it always returns 0, and nothing tests it. The line is outside the diff so I cannot put this inline, it is the final return 0 of that function. Changing it to return "$rc" survives all 63 tests, and a non-zero there aborts the { ... } >&2 group under the caller's set -e, dropping exactly the Job status and access log sections this PR adds.

One caveat missing from the body. The warm-up is not free on a run where the egress allow fails. It fires at install time gated only on the Deployment existing, but whether workers use the mirror is decided later in resolve_ghcr_mirror_endpoint, which returns empty when _apply_ghcr_mirror_egress_policy fails. On that run the mirror serves, the Job warms both tags (~244 MiB with the double read you document), workers fall back to public ghcr.io anyway, and the whole fetch is pure addition to the same throttled leg, concurrent with Pre-pull platform images. You apply this reasoning to arm64 but not to this case.

Separately, this file aborts mid-suite under cozytest when /bin/sh is bash, 44 of 63 tests never run. It reproduces on base so it is not yours, I put the reproduction on #3453.

All five inline points are closed and I confirmed each by mutation. What is still open is two sentences of the description, and one of my two unanchored points was wrong when I made it.

The wall-clock gap is closed, ghcr-mirror_test.bats:1313-1326 times the script and asserts elapsed against the budget before the count. Replacing the wall-clock loop with a hardcoded while [ $n -lt 3 ] reddens test 57, and the same mutant survived the whole suite before. The clock assertion cannot flake in the false-negative direction either, since the loop exits only once elapsed reaches the budget and start is taken after t0.

The stale 634 MiB is gone from all three consumers, grep -n 634 now returns only the provenance note. Three fifths agrees in both places. The not deployed early return is pinned by a valid image plus an apply capture asserted empty, and deleting the return 0 fails test 63. -T 600 is read out of the emitted script and required to be at least 600, and dropping it to 5 fails test 58.

My point 7 was wrong and it is my error, so it is worth stating plainly. I claimed return "$rc" in place of the final return 0 in ghcr_mirror_diagnose survives all 63 tests. It kills 11, because each of those does out=$(ghcr_mirror_diagnose 2>&1) and bats' set -e aborts the test on a non-zero assignment. A hard return 1 kills the same 11. The second half of my claim was wrong too, the final return 0 sits after the { ... } >&2 group so changing it cannot abort the group, and the caller is ghcr_mirror_diagnose || true, which absorbs a non-zero return and disables errexit inside the function anyway. Nothing to do here.

Two things still open, both in the description rather than the code.

Budget worst case is 175s, not 150s. Five bounded reads become seven, and with the 20s timeout and 5s grace the ceiling is 7 by 25. Even ignoring the kill grace the pair is 100 to 140, so roughly 100s to roughly 150s is neither figure. And cozy_diag_phase_has_time checks the deadline only on entry, so a section admitted with a second left runs its full 175s and the cost lands on the next gate. The tree is clean here, the phase inventory claims no per-section worst case at all, so this is a description fix.

The egress-allow caveat is still not written down, and it is no longer hypothetical. _apply_ghcr_mirror_egress_policy failing leaves the endpoint empty, so ghcr_registry_mirrors_block emits nothing and the tenant is never pointed at the mirror, while warm_ghcr_mirror fires gated only on the Deployment existing. I should have credited hack/e2e-install-cozystack.bats:106-112, which already documents the consequence for the reachability variant. What is missing is the policy-failure trigger.

Which brings me to the thing I would most want said. Both suites at this head warmed the cache cleanly and no worker ever reached the mirror. The run prints ghcr-mirror ready and egress allow applied, the worker-egress policy True, the Job Complete with warmed: v1.35.6 v1.34.9 and fetched 8, failed 0, and then no siderolabs/kubelet request appears in the mirror's access log, twice, once per suite, each followed by the node-join failure. None of that is caused by this PR and the diagnostics you added are what made it legible. But it does mean the 2x upstream read was paid by the Job and by nobody else on these runs, so the 2x is paid once either way is false on exactly them, and the feature merges with no run in which a worker benefited. Say that rather than leaving the reader of a future green suite to infer it.

Smaller notes. activeDeadlineSeconds: 3600 bounds node time across all four attempts, not completion, so a deadline kill SIGTERMs mid-loop and the three summary lines never print, leaving the diagnostic's --tail=5 showing fetched lines and no verdict on precisely the slow run where which-tag-is-warm is the question. Test 61 pins only that the field is numeric. The ~61 MiB and ~300 MiB figures are single-read arithmetic in a file that documents a 2x read four lines later, so the next person raising the tag count budgets half. And kubectl logs job/... reads one Pod of up to four, so the section can print a failed attempt's tail under a get job line reporting Complete.

The cozytest blind spot is unchanged and widened, 46 of 65 tests never execute on a host where /bin/sh is bash, up from 50 of 63. Cause is unchanged and pre-existing, the trace folds into $out. CI stays green because its /bin/sh is not bash. Stating it since this PR touches both the file and its library: it neither causes nor fixes that, and the 40 tests it adds are inert on such a host.