chore(release): promote v1.6.4-rc.1 -> v1.6.4 (#4533)

Promotes `v1.6.4-rc.1` to stable `v1.6.4` without rebuilding containers.
Their digests remain bit-for-bit what passed rc e2e; the tag-rewritten
packages tree is re-serialized as a temporary OCI candidate and pinned
here by digest. On merge, pull-requests-release.yaml verifies that
candidate against this tree before creating the write-once `v1.6.4` tag,
retagging the candidate and container digests to `v1.6.4` (+`:latest`
when this is the newest stable), publishing the stable cozy-installer
chart, and publishing the release. Do NOT squash-merge (decision B): the
stable tag must attach to a real merge commit.

⚠️ **RC e2e gate bypassed** — this promotion did not verify green full
e2e evidence for `v1.6.4-rc.1`.

ℹ️ E2E already ran against the immutable rc, so this promote PR does not
run E2E by default. A maintainer may add the `full-e2e` label to run the
full suite again.

✅ Includes `docs/changelogs/v1.6.4.md`, which finalize uses verbatim as
the published release body.

✅ Website docs PR opened/refreshed on `cozystack/website` (branch
`update-docs-v1.6.4`), generated from the `release-1.6.4` staging
branch. **Do NOT merge that PR until `v1.6.4` is published** — merging
it early flips the site's latest-version pointer to an unpublished
version and 404s its docs/API links (see the PR body).

chore(release): promote v1.6.4-rc.1 -> v1.6.4 (#4533)

Promotes `v1.6.4-rc.1` to stable `v1.6.4` without rebuilding containers.
Their digests remain bit-for-bit what passed rc e2e; the tag-rewritten
packages tree is re-serialized as a temporary OCI candidate and pinned
here by digest. On merge, pull-requests-release.yaml verifies that
candidate against this tree before creating the write-once `v1.6.4` tag,
retagging the candidate and container digests to `v1.6.4` (+`:latest`
when this is the newest stable), publishing the stable cozy-installer
chart, and publishing the release. Do NOT squash-merge (decision B): the
stable tag must attach to a real merge commit.

⚠️ **RC e2e gate bypassed** — this promotion did not verify green full
e2e evidence for `v1.6.4-rc.1`.

ℹ️ E2E already ran against the immutable rc, so this promote PR does not
run E2E by default. A maintainer may add the `full-e2e` label to run the
full suite again.

✅ Includes `docs/changelogs/v1.6.4.md`, which finalize uses verbatim as
the published release body.

✅ Website docs PR opened/refreshed on `cozystack/website` (branch
`update-docs-v1.6.4`), generated from the `release-1.6.4` staging
branch. **Do NOT merge that PR until `v1.6.4` is published** — merging
it early flips the site's latest-version pointer to an unpublished
version and 404s its docs/API links (see the PR body).

[Backport release-1.6] feat(kubevirt): expose migration configuration…

… through platform values (#4402)

This reopens a fork backport from a branch in this repository so its CI
can run. The commit is the one from #4339 by @yankawai, unchanged. On
release-1.6 the pull-request CI pushes the images it builds, and a run
from a fork has no registry credentials, so #4339 stopped at the build
jobs and never reached e2e. The description below is the author's.

## What this PR does

Manual backport of #4254 to `release-1.6`.

The automated backport, #4322, stopped with conflict markers in four
files as its only commit, so it is red on DCO and unmergeable as it
stands.

Every conflict has the same cause: the `kubevirt.disabledFeatureGates`
work landed on main after 1.6 was cut, and the cherry-pick carried it
into hunks this line does not have. Four differences follow:

- platform values gains a `kubevirt` block holding `migrations` alone,
and the iaas bundle gains only the migrations hop;
- the wiring test takes the four migrations cases and the case for a
null `kubevirt` block, without the assertions that belong to
disabledFeatureGates cases this line does not have;
- `update_idempotency_test.sh` gains `run_update_logged`, a four-line
helper that arrived on main with that work and the new cases need;
- the kubevirt Makefile header counts six sed patches rather than five.

Verified on this branch: 9 kubevirt chart tests, 31 platform wiring
tests (102 across the platform suite), and `update_idempotency_test`
PASS under GNU make and sed. Removing the new awk guard from the
Makefile turns that test red naming the migrations block, so the ported
guard is doing work. On this line `(.Values.kubevirt).migrations` is the
only reader of `.Values.kubevirt`; rewriting it as
`.Values.kubevirt.migrations` turns the null-block case red with a nil
pointer.

Feature summary, unchanged from #4254: cluster-wide migration settings
could only be applied by patching the KubeVirt CR, because the kubevirt
chart did not render `migrations` and the platform bundle did not
forward it. `kubevirt.migrations` now passes through the generated
Package to `spec.configuration.migrations`. A hand patch of that Package
is undone on the next platform render, so this hop is the only supported
setter.

```yaml
kubevirt:
  migrations:
    bandwidthPerMigration: 625M
    parallelMigrationsPerCluster: 2
    parallelOutboundMigrationsPerNode: 1
```

### Screenshots

Not a UI change.

### Downstream repositories

Walked the trigger map against the diff: platform values, the iaas
bundle and the kubevirt chart. The v1.6 reference page of the platform
values had no `kubevirt` section, so the row goes there as a follow-up
that should land with this backport.

- [ ] No downstream repository is affected by this change
- [x] [cozystack/website](https://github.com/cozystack/website) -
follow-up: cozystack/website#705

### Release note

```release-note
feat(kubevirt): expose `kubevirt.migrations` in the platform values and forward it to `spec.configuration.migrations` on the KubeVirt CR, so live-migration bandwidth and parallelism can be set without patching a Package that the platform re-renders.
```

smee nil-safe global patch, rebased onto main afc003c; candidate for …

…a standalone PR

chore(release): promote v1.6.3-rc.2 -> v1.6.3 (#4075)

Promotes `v1.6.3-rc.2` to stable `v1.6.3` without rebuilding containers.
Their digests remain bit-for-bit what passed rc e2e; the tag-rewritten
packages tree is re-serialized as a temporary OCI candidate and pinned
here by digest. On merge, pull-requests-release.yaml verifies that
candidate against this tree before creating the write-once `v1.6.3` tag,
retagging the candidate and container digests to `v1.6.3` (+`:latest`
when this is the newest stable), publishing the stable cozy-installer
chart, and publishing the release. Do NOT squash-merge (decision B): the
stable tag must attach to a real merge commit.

⚠️ **RC e2e gate bypassed** — this promotion did not verify green full
e2e evidence for `v1.6.3-rc.2`.

ℹ️ E2E already ran against the immutable rc, so this promote PR does not
run E2E by default. A maintainer may add the `full-e2e` label to run the
full suite again.

✅ Includes `docs/changelogs/v1.6.3.md`, which finalize uses verbatim as
the published release body.

✅ Website docs PR opened/refreshed on `cozystack/website` (branch
`update-docs-v1.6.3`), generated from the `release-1.6.3` staging
branch. **Do NOT merge that PR until `v1.6.3` is published** — merging
it early flips the site's latest-version pointer to an unpublished
version and 404s its docs/API links (see the PR body).

chore(release): promote v1.6.3-rc.2 -> v1.6.3 (#4075)

Promotes `v1.6.3-rc.2` to stable `v1.6.3` without rebuilding containers.
Their digests remain bit-for-bit what passed rc e2e; the tag-rewritten
packages tree is re-serialized as a temporary OCI candidate and pinned
here by digest. On merge, pull-requests-release.yaml verifies that
candidate against this tree before creating the write-once `v1.6.3` tag,
retagging the candidate and container digests to `v1.6.3` (+`:latest`
when this is the newest stable), publishing the stable cozy-installer
chart, and publishing the release. Do NOT squash-merge (decision B): the
stable tag must attach to a real merge commit.

⚠️ **RC e2e gate bypassed** — this promotion did not verify green full
e2e evidence for `v1.6.3-rc.2`.

ℹ️ E2E already ran against the immutable rc, so this promote PR does not
run E2E by default. A maintainer may add the `full-e2e` label to run the
full suite again.

✅ Includes `docs/changelogs/v1.6.3.md`, which finalize uses verbatim as
the published release body.

✅ Website docs PR opened/refreshed on `cozystack/website` (branch
`update-docs-v1.6.3`), generated from the `release-1.6.3` staging
branch. **Do NOT merge that PR until `v1.6.3` is published** — merging
it early flips the site's latest-version pointer to an unpublished
version and 404s its docs/API links (see the PR body).

[Backport release-1.6] feat(clickhouse): add configurable version par…

…ameter (#4054)

# Description
Backport of #3476 to `release-1.6`.

[Backport release-1.6] fix(kubernetes): retry the tenant CNI install …

…instead of uninstalling it (#4033)

Backport of #3552 to `release-1.6`.

Tenant cilium HelmRelease on this branch sets
`install.remediation.retries: -1` with no `strategy:`, so
helm-controller uses its default install remediation, which is
uninstall. Cilium is the tenant cluster's only CNI, so a tenant whose
install misses its budget gets the CNI torn down and reinstalled once
per cycle, and `-1` lets that repeat without end. `RetryOnFailure` keeps
applied manifests in place and retries the failed install as an upgrade
instead. Strategy is set on both actions because the retry of a failed
install runs as an upgrade.

Shipped broken in v1.6.0, v1.6.1 and v1.6.2. `RetryOnFailure` appears in
none of the 19 helmrelease templates on this branch, and the Flux CRD
release-1.6 already ships accepts it - enum at
`spec.install.strategy.name` and `spec.upgrade.strategy.name`, with the
`retryInterval` CEL guard.

Cost is carried over from the original PR: a genuine upgrade failure now
retries on failed manifests instead of rolling back. Controller can't
tell a retried install from a real upgrade, and what it replaces was an
unbounded rollback-and-retry flap.

Auto backport did not apply here, and it was not a conflict. Bot picked
four commits from the PR branch, three of them empty `ci: re-run`
commits, and `git cherry-pick` exits non-zero on an empty pick with no
unmerged paths, which the action reads as a conflict. Picking
`80a209044` alone is the whole backport and applies with a line offset
only. Resulting tree matches the bot's four-SHA-with-three-skips recipe
byte for byte.

### Testing

- `make test` in `packages/apps/kubernetes` - 18 suites, 186 tests
green.
- New suite is mutation-proven rather than revert-proven: setting
`install.strategy.name` to `Rollback` reds exactly the install test,
removing the `upgrade.strategy` block reds exactly the upgrade test,
other 185 unaffected both times.
- `make unit-tests` and `make test-controllers` green, `make generate`
leaves no drift.
- No migration touched, `targetVersion` still 54.

```release-note
fix(kubernetes): retry the tenant CNI install instead of uninstalling it
```

test(tests): cut the noise in unit and e2e job logs (#3973)

## What this PR does

Unit test job log is 238,794 lines and 31MB, and 233,444 of those are
`set -x` trace from `hack/cozytest.sh`. The one failing assertion sits
at line 238,573. Nothing in that stream is needed there, `run_one`
already tees raw output to `$log` and the fail handler dumps all of it
for whichever test fails, so dropping the live stream loses output of
passing tests only.

`COZYTEST_TRACE` defaults to 1, so e2e call sites in
`packages/core/testing/Makefile` keep the live stream. For a long e2e
suite that stream is the only progress signal, and the only record at
all when a step timeout or a dead runner kills the job before the fail
handler runs. `bats-unit-tests` sets 0. Running the runner by hand stays
verbose, that is the case where a human is actually watching.

Second thing, previous-instance log capture now orders by restart
recency. Container cap (`COZY_PREVLOG_MAX=12`) is spent in namespace
order and "unrelated" meant only "in another namespace", time was never
a key. So a container that restarted once during bring-up competed on
equal footing with one that crash-looped during the test, and won
whenever the pod list reached it first. Two measured e2e jobs spent 1034
and 1545 lines on `virt-api` and `cdi-apiserver`, both restarted once at
install time by `Unable to create certwatcher`, hours before any app
suite ran. On a live stand 178 containers carry a restart and their
timestamps span 24 days, so the cap binds hard enough for order to
decide everything.

This orders, it does not filter. Install-time restart is the explanation
when install is what broke, and the capture has no reference point for
"when the test began" that would make a cutoff honest. Namespace stays
primary key, because a stale restart in the namespace under test beats a
fresh one three namespaces away. So it fixes cap displacement, not raw
volume of a single run.

`lastState.terminated.finishedAt` joins the row. The guard on it is
three deep and the innermost one is load bearing, which I got wrong
first time round and review caught: `finishedAt` is a metav1.Time behind
`json:",omitempty"`, omitempty cannot omit a struct, and the zero value
marshals to `null`. The kubelet does synthesize a terminated state with
no time, ContainerStatusUnknown on a pod deleted mid-run, which also
bumps restartCount and so clears the restart filter. kubectl renders
that leaf as the literal `<no value>`, and `<` sorts above every digit,
so under the reverse sort that row landed FIRST and displaced the most
recent real restart. Exactly the inversion the ordering exists to
prevent, plus a header reading `previous instance ended <no value>`.
Fixed at both layers because they fail independently: the template
guards the leaf so the string cannot be produced, and the sort
normalises any field that is not shaped like a timestamp to empty so the
ordering holds even if some other producer emits one.

Worth naming how that slipped through. I probed the template against a
live cluster, saw no `<no value>`, and read it as the guard being
correct. It only meant no container on that cluster was in that state.

Reading the field also fixes a latent bug it would otherwise have
caused: `read -r ... restarts` folds a trailing field into the last
variable, so the dump header would print
`restarts=1|2026-08-22T09:06:00Z`. Now it names the time, which is the
first thing telling a reader whether the dump predates the test.

### What quiet mode keeps and what it drops

Two things are dropped, and the second one came out of review rather
than out of me.

First, xtrace and stdout of tests that pass. You cannot diff a passing
test's trace against the failing one inside the same job log, and a
warning printed by a test that passed is gone.

Second, a test that is KILLED rather than failed loses its trace as
well. The fail handler runs only after the pipeline returns, so a hang
reaped by the job timeout never dumps anything, and quiet mode has by
then dropped the live stream. Verbose mode never noticed, because the
trace was already printed. Measured: verbose leaves the line the test
blocked on, quiet leaves the `╭` and nothing else. So this one is a real
regression on the unit lane, not a pre-existing gap.

An INT/TERM trap is the obvious repair and it does not work. POSIX sh
defers a trap until the current foreground command returns, so
installing one stops the runner dying on the signal at all: a suite hung
on `sleep 30` took 30s to exit with the trap against 2s under the
default disposition. GitHub escalates INT, then TERM, then KILL, so the
deferred handler gets SIGKILLed before it ever runs, and the grace
period goes on waiting for the hang. Built it, measured it, reverted it,
and the measurement is in the comment now so the next reader does not
repeat it. Making it work needs the test pipeline moved into the
background with `wait`, which is the interruptible one, and that is
surgery on the path every suite in this repo takes for a partial trace
in a rare case.

What is left for that case is the escape hatch below, which for this
lane costs 48s and no cluster. The same argument is why e2e keeps the
stream on: re-running there is an hour and a live cluster, so the record
has to be written as it goes.

Everything else stays. The `╭`/`╰` pair per test with elapsed time, so
which tests ran, in what order and how long each took is all still
there. Full trace, stdout and stderr of the failing test, same bytes as
before, because `run_one` tees to `$log` and the fail handler dumps it,
and that path is untouched. Anything printed outside `run_one` as well,
file level code, `.bats` parse errors, and the `--- running <file> ---`
markers. A hang still shows up as `╭` with no matching `╰`. And `make
unit-tests` stops at the first failing test anyway, so there was never
more than one dump to read.

Measured on a suite that is actually red (`hack/ghcr-mirror_test.bats`
fails on my machine because it reads a live cluster): verbose 1472
lines, quiet 63 lines, and the post-failure dump is 23 lines and
byte-identical in both modes.

Getting the trace back, both paths checked:

```
COZYTEST_TRACE=1 make unit-tests     # env prefix, 29200 trace lines
make unit-tests COZYTEST_TRACE=1     # make cmdline, same
hack/cozytest.sh hack/foo.bats       # by hand, verbose by default
```

This lane needs no cluster and runs locally in 48s, so anyone who wants
the trace just reruns it. That is exactly why the same argument does not
extend to e2e, and why the default stays 1 there.

Nothing else consumed that stream. Unit lane stdout goes only to the job
log, `cozyreport.sh` folds in crust-gather snapshots and the trap writes
those to disk independently, and the unit lane produces none of them.

### Verification

Measured on this PR's own run: unit job went from 238,794 lines and 31MB
with 233,444 trace lines, to 5,720 lines and 580KB with 0 trace lines
and 1548 tests OK. Those two totals are not strictly like for like, the
old run was red and this one is green, so the honest comparator is
233,444 trace lines down to 0.

Quiet output is byte-for-byte the non-trace subset of verbose, and the
fail handler dump is unaffected (checked with a deliberately failing
suite). Verbose output unchanged against main, byte-identical sorted on
`hack/bats-no-exit-trap.bats`. Other suites are not self-stable run to
run (temp names, epochs, stdout/stderr interleaving) so that is the
ceiling on that check, not a finding.

Eight tests on the capture and seven on the switch. Every fix is
mutation proven: reverting the template leaf guard, removing the sort
key normalisation, dropping the read variable, dropping `-s`, silencing
an e2e recipe, removing the fail handler dump, flipping the default, and
making the switch a no-op are each caught. Three of those guards did not
bite when first written, which review also found: both stability
fixtures were pre-sorted ascending, which is what sort's last-resort
whole-line compare reproduces with no `-s` at all, and no test fed a
six-field row through the whole script, so deleting the read variable
left the suite green at 39/39. Whole unit lane, 83 files, 1503 passed
and 1 failed. The failure is `hack/ghcr-mirror_test.bats`, it reads a
live cluster and fails identically under main's own runner.

### Screenshots

Not a UI change.

### Downstream repositories

Walked the trigger map file by file against the diff. Nothing under
`hack/` is moved or renamed, no make target changes what it does
(`bats-unit-tests` runs the same files to the same verdict, only
quieter), and `hack/package.mk`, `hack/common-envs.mk`,
`hack/update-crd.sh` and `hack/e2e-prepare-cluster.bats` are untouched,
which is what ccp, external-apps-example, talm and ansible-cozystack
couple to. No package `values.yaml`, `values.schema.json`, CRD,
namespace, label, annotation or metric name is involved.

- [x] No downstream repository is affected by this change

### Release note

```release-note
test(tests): bats unit test trace is now opt-in via COZYTEST_TRACE and quiet by default in `make unit-tests`, and the e2e previous-instance log capture now picks the most recently restarted containers first
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

- **Improvements**
- Test runs can suppress live trace output for passing tests while
preserving full diagnostics for failures.
  - Trace visibility can be customized through the test configuration.
- Previous-instance logs are prioritized by the most recent container
termination within the preferred namespace.
  - Captured log headers now show when the previous instance ended.
  - Undated logs are retained and placed after dated entries.

- **Tests**
- Added coverage for trace controls, failure handling, and recent-log
ordering.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

chore(release): promote v1.6.2-rc.1 -> v1.6.2 (#3904)

Promotes `v1.6.2-rc.1` to stable `v1.6.2` without rebuilding containers.
Their digests remain bit-for-bit what passed rc e2e; the tag-rewritten
packages tree is re-serialized as a temporary OCI candidate and pinned
here by digest. On merge, pull-requests-release.yaml verifies that
candidate against this tree before creating the write-once `v1.6.2` tag,
retagging the candidate and container digests to `v1.6.2` (+`:latest`
when this is the newest stable), publishing the stable cozy-installer
chart, and publishing the release. Do NOT squash-merge (decision B): the
stable tag must attach to a real merge commit.

⚠️ **RC e2e gate bypassed** — this promotion did not verify green full
e2e evidence for `v1.6.2-rc.1`.

ℹ️ E2E already ran against the immutable rc, so this promote PR does not
run E2E by default. A maintainer may add the `full-e2e` label to run the
full suite again.

✅ Includes `docs/changelogs/v1.6.2.md`, which finalize uses verbatim as
the published release body.

✅ Website docs PR opened/refreshed on `cozystack/website` (branch
`update-docs-v1.6.2`), generated from the `release-1.6.2` staging
branch. **Do NOT merge that PR until `v1.6.2` is published** — merging
it early flips the site's latest-version pointer to an unpublished
version and 404s its docs/API links (see the PR body).