test(e2e): drop the ghcr.io pull-through mirror - #4007

…image (CDI clone)

Rebase of #3294 onto main after the Phase 2 KubernetesNodes split (#3315).
The worker disk source and the in-guest installer pin move from
packages/apps/kubernetes to packages/apps/kubernetes-nodes, where the pool
objects now live; the per-node-group `nodeGroups.<name>.image` union becomes
the pool-level `image` value of the KubernetesNodes chart.

Adds packages/system/kubernetes-worker-image, an opt-in catalog that imports a
golden Talos worker image into cozy-public once per (schematicID, version). A
pool that sets image.builtin CDI-clones that golden instead of streaming the
raw image over HTTP per worker, which is the per-worker Image Factory
dependency behind the kubernetes-* node-join flake (#3231). image.factory
keeps the HTTP path and lets a pool point at its own mirror/schematic/version.
With `image` omitted the render is byte-identical to before -- the stored
render snapshot is unchanged, so existing workers do not roll on upgrade.

The e2e suites switch to the clone path: the per-worker talos-image-cache
mirror and its manifest, helper and tests are removed, the catalog is enabled
in the sandbox, and the run waits for the golden to import before creating the
tenant. The ghcr.io kubelet-image mirror is untouched and still spliced into
both tenant CRs.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <[email protected]>

…source

Five findings from the #3294 review rounds, each with a guard that fails
without the fix.

[MAJOR] The Talos<->Kubernetes support matrix keyed only on talos.version, so
a pool could keep a supported talos.version and still boot a Talos release
outside the kubelet's support window through image.builtin.version or
image.factory.version, with no render failure. The matrix moves into a named
template in _helpers.tpl and both paths consult it -- the pool default at the
top of nodegroup.yaml, the resolved version where the image is resolved. A
Talos minor the table does not list still passes, unchanged, because failing
closed on a hand-maintained table would block an operator moving to a newer
Talos before the table is updated.

[MINOR] schematicID, version and imageFactoryURL reach an unquoted YAML scalar
in the KubevirtMachineTemplate, and a value carrying a newline or a YAML
metacharacter injects keys into the pool's own template. The schema cannot
narrow them -- the pinned cozyvalues-gen derives `pattern` from the value type
and has no annotation for a custom one -- so they are pattern-checked at
render, which is where this chart already validates the kubelet reservation
fields against the same hazard. The clone-path DataVolume name is quoted too;
the HTTP url deliberately is not, because quoting it changes the KMT content
hash and rolls every live worker.

[MINOR] A golden was referenced only from inside a VM dataVolumeTemplates
source, so nothing told an operator which pools still pinned it -- and a pool
on `builtin: {}` does not name it in its own CR either. Every cloned worker
disk now annotates its source golden, and the catalog documents the one read
that answers "is this golden still in use".

[MINOR] An images[] entry is effectively immutable once its golden exists: the
DataVolume is named from (schematicID, version) while storage, storageClass
and the URL are rendered into a spec that cannot be patched in place. The
catalog said the opposite, inviting a `storage` bump. It now says to add a new
entry instead.

[MAJOR, residual] Nothing tied the parent kubernetes chart's talos defaults to
kubernetes-nodes'. Both declare "keep in sync", the pool chart's `make update`
copies neither, and the catalog drift guard reads only one of the two files. A
one-sided bump now trips three checks instead of none.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <[email protected]>

Self-review of the two commits above, three blockers.

The golden-deletion procedure was wrong for the case it was written for. It
told the operator to query live worker DataVolumes and read an empty result as
permission to delete. A pool sits at zero workers whenever the autoscaler has
nothing to place -- minReplicas: 0 is the default -- and it still pins its
golden, so that query answers "empty" for exactly the pool whose next scale-up
breaks. The pool declarations are the authority now, with the live disks as the
second read; both empty is the go-ahead.

Nothing asserted that the e2e suites take the clone path. Dropping the image
block from the KubernetesNodes heredoc, or landing it at the wrong indent, puts
the pool back on the HTTP import while every suite still passes -- the catalog
imports a golden nobody clones. That is the same silent-fallback shape this
branch already had to fix once, so it gets a guard, mutation-proven by deleting
the block and watching it fail.

The catalog suite covered only accepted input. Both `images[]` fields are
`required`, and what that refusal buys is the golden's name: an entry missing
either renders `talos-worker--<version>` or `talos-worker-<id>-`, and two such
entries collide on one DataVolume, so the second replaces the first and every
pool pinning it clones the wrong OS. Two cases now assert the refusal.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <[email protected]>

Four blockers, plus the immutability finding carried over from the second
round and the minors alongside it.

The optional-package helper emitted a Package carrying a variant and
nothing else, and package_reconciler takes a component's HelmRelease
values from the Package alone, so the worker image catalog documented
four operator knobs that no supported path could set. That left the two
scenarios the feature exists for -- air-gapped and non-replicated
storage -- with a render guard naming a fix the operator could not
apply. The helper now takes the same components argument its
non-optional twin already had, iaas.yaml forwards a new platform-level
kubernetesWorkerImage block, and the nine other optional packages get
the same path.

The transient headroom a clone needs was written down only in the CI
harness. It is a property of the feature, so it belongs in diskSize and
in the catalog's storage description, with the arithmetic spelled out.

The catalog told operators to set nodeGroups.<name> on a Kubernetes CR,
inert since the Phase 2 split, one paragraph above the query that gets
it right.

talos.version promised the matrix covers every pairing; the helper's own
comment says the opposite three paragraphs above the code, and
cozyvalues-gen copies the optimistic half into the schema, the README
and the dashboard. Both now say what the guard does.

An entry whose golden exists is now enforced immutable rather than
merely documented so, because the documented recovery -- delete the
golden -- runs into the deletion blast radius the same file warns about.
Beside it the catalog refuses a duplicate (schematicID, version) and
pattern-checks both fields, so it can no longer create a golden whose
name a pool cannot reconstruct.

The absence and Failed golden guards now fire only for a pool still
waiting for its first disk. A pool that cloned months ago has no stake
in the source, and aborting its whole HelmRelease on a periodic
reconcile was the blast radius the mid-import case already refuses.

An empty storageClass on either side now resolves against the cluster
default before the cross-class comparison, which is the one corner that
cannot be a clone and was passing unchecked.

The image resolution is a named template both nodegroup.yaml and
talos-reconcile-job.yaml call, carrying the format checks with it, so
the boot disk and the in-guest installer cannot drift apart -- the
argument this PR already made for the support matrix.

The pool field is renamed image -> osImage. It landed one character from
the existing images (chart-internal helper overrides) with an unrelated
meaning and the same air-gapped motivation in its description. This is
new CR surface that has not shipped, so the rename is free now and a
migration later.

Also: the schema enum drift guard iterated over a command substitution
whose failure looked like an empty result, so a renamed property
disarmed it; the e2e golden wait reported nothing when the outer timeout
fired before the inner one; and ComputePlane's pool-shape allowlist now
records the exclusion of osImage rather than leaving it merely absent.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <[email protected]>

…sserted

Ran every new conditional through deletion -- force the condition false,
and for an and-chain drop each conjunct -- then ran the suite against
each mutant. A mutant that leaves the suite green is a term nothing
asserts, so it can be removed in good faith by the next person to touch
the file.

Three survived, all in code this branch added:

The presence check on each drift comparison in the catalog. Without it a
golden whose spec carries no http source, no storage request or no class
compares an empty string against the rendered value, reports drift that
did not happen, and blocks every later upgrade of the catalog.

The (not $poolCloned) conjunct on the Failed golden guard. The suite had
a pool whose sibling had cloned and a pool whose own clone was
unfinished, but not the case the gate exists for: a golden that went
Failed after this pool was already whole.

The $workerSC presence term. Nothing covered an empty pool storageClass
with no default StorageClass to resolve it against, which is the arm
where the comparison is skipped rather than guessed.

One mutant still survives in this region, on the gate in front of the
second support-matrix call. It is unreachable from any schema-valid
render -- the matrix ships one row whose Kubernetes list is a superset
of the version enum -- and it is covered structurally instead, by the
greps in hack/check-worker-image-catalog-defaults.bats that assert both
call sites exist and that the second is fed the resolved version. That
is stated in the bats file rather than left to be rediscovered.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <[email protected]>

The second model read the same diff and reached five defects the
single-model pass did not, all in code this branch added.

A sibling pool answered for a pool that had never cloned. The
already-cloned gate matched worker disks by the name prefix
disk-system-<cluster>-<pool>-, and a pool named md0-gpu produces disks
carrying md0's prefix. With the golden absent or Failed, md0's guards
were suppressed by md0-gpu's disk and md0's new workers pointed at a
source that was gone -- silently, which is the outcome the guards exist
to prevent. Each cloned disk now names its own pool in an annotation and
the match is exact. This only touches the clone path, so no existing
worker's template changes.

A golden that declares no StorageClass sits on whatever default was in
force when CDI bound its PVC, not on today's. Substituting the current
default for it could wave through a cross-class copy or refuse a pool
that does match. The class now comes from the PVC, which is where the
concrete value lives.

More than one StorageClass may carry the default annotation at once --
the normal state midway through swapping a cluster's default -- and
Kubernetes resolves it by creation time. The helper emitted every match
concatenated, producing a class name that does not exist and refusing a
pool whose class matched the default actually in force. It now picks the
most recently created, as Kubernetes does.

A Talos version was accepted without its patch segment. Both regexes
took vMAJOR.MINOR, which names an artifact the Image Factory does not
publish and an installer tag that does not exist, so the failure landed
in the guest instead of at render.

And the remedy printed by the cross-class guard could not be followed.
The previous commit pointed it at the catalog entry's storageClass; the
immutability guard added by this same branch rejects that edit once the
golden exists, which is the only state in which this branch runs. The
message now names the two moves that work: a new (schematicID, version)
entry on the right class with the pool repointed at it, or deleting the
golden once nothing clones it.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <[email protected]>

The three format guards run for every pool, including one with osImage
omitted, so they apply to values a running cluster may already hold. The
schema does not stop any of them: values.schema.json and the cozyrds
openAPISchema type talos.version, talos.schematicID and
talos.imageFactoryURL as bare strings with no pattern, so the CR accepts
a malformed one and the pool's HelmRelease is what fails.

Reproduced on a live stand rather than argued from the template. A
KubernetesNodes carrying talos.version "v1.13" was admitted with the
value intact and its HelmRelease went InstallFailed naming the format.
Editing an unrelated field on that pool afterwards -- maxReplicas 1 to 3
-- was accepted by the CR and reached .spec.values of the HelmRelease,
which stayed InstallFailed: the edit is taken everywhere and applied
nowhere. That is the cost being documented, and it is why the shapes are
named in prose where an operator reads them.

Pattern is not expressible here. cozyvalues-gen writes types.go,
values.schema.json, the README table and the cozyrd from values.yaml,
and its annotation set has no pattern; a pattern added to
values.schema.json by hand is erased by the next make generate, which is
how this was checked. Teaching the generator that annotation is the
follow-up that would let the rejection land on the CR instead.

schematicID already documented its shape. version and imageFactoryURL
now do too, the latter noting that a builtin pool builds no source URL
and so is not held to it.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <[email protected]>

… image (CDI clone) (#3294)

<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

Tenant worker boots from talos OS disk image that CDI streams over HTTP
from image factory, and it does that once per worker. Same bytes N
times, and N chances for a fetch to fail. In closed or rate-limited
environment that is wrong shape.

This PR adds opt-in catalog and per-pool selector that reads from it.

`packages/system/kubernetes-worker-image` imports one golden DataVolume
into `cozy-public` per `(schematicID, version)`. Disabled by default,
enable by adding `cozystack.kubernetes-worker-image` to
`bundles.enabledPackages`. Its knobs (`imageFactoryURL`, `storageClass`,
`storage`, `images`) are set through
`spec.components.platform.values.kubernetesWorkerImage` on
cozystack-platform Package. That needed optional package helper to learn
components argument, which it did not have, so every optional package
registered through it could name variant and nothing else. Nine other
packages get same path for free.

`KubernetesNodes` gets `osImage` field with two mutually exclusive
forms:

- `osImage.builtin` boots pool workers by CDI-cloning golden from
catalog. Clone is storage layer CSI copy, so OS image never crosses pod
network and factory answers once for whole fleet instead of once per
worker.
- `osImage.factory` keeps HTTP import and lets pool pin own
`imageFactoryURL`, `schematicID` and `version`. This is bring-your-own
path for operator running own factory or caching mirror, and it is what
pool gets when `osImage` is omitted.

In-guest talos installer written by reconcile job follows same
resolution, so pool that pins release boots it and upgrades into it
instead of being flipped back to cluster default halfway.

This is pool half of what #3950 asks for. That issue wants platform
level defaults for two channels, worker OS image and in-guest registry
mirrors. Here image channel gets per-pool source and shared cache,
platform default plumbing is left to the issue.

### Removed e2e workaround

Suites used to deploy `talos-image-cache` Deployment and point tenant
CRs at it, on theory that public factory stalling mid-transfer is what
makes workers miss node-join deadline. Work on #3513 settled it
differently, what separates run that joins from run that stalls is host
`kvm_amd` `vgif` flag, and stall reproduces on runs where cache was up
and serving. So cache, its manifest, helper and test suite are gone and
suites clone golden instead. That also puts clone path under e2e, unit
tests alone never reached it.

Stacked on #4007 which removes other half of same band-aid on same
evidence. Base of this PR points at that branch, so diff here is only
this feature. Merge #4007 first.

### Existing workers do not roll

With `osImage` omitted rendered `KubevirtMachineTemplate` is byte
identical to what chart produced before. KMT is named by hash of its own
content, so any drift would re-image every worker VM in fleet on next
upgrade. Stored render snapshot is untouched by this PR, that is the
evidence and not the claim.

### What fails at render

Clone that would silently degrade into something slower or wrong is
refused instead of rendered. Golden absent, or its import in terminal
`Failed` phase. Pool `storageClass` different from golden one, because
CDI cant CSI-clone across storage classes and falls back to host
assisted copy over pod network, which is exactly the transfer
`osImage.builtin` removes. Empty class on either side resolves against
cluster default class before comparing, so pool at `storageClass: ""` is
checked too. `diskSize` below golden storage, because clone target
cannot be smaller than source. Talos release the matrix lists as
unsupported, whether pool reached it through `talos.version` or through
`osImage.*` override. Matrix passes Talos minor it does not list, same
as before, so what it promises is that known-bad pairing is refused, not
that every pairing is known.

Golden that is only mid-import is not an error. CDI holds worker disk
until source populates, and failing there would abort whole pool
HelmRelease for length of a download. Same reasoning now covers pool
that already cloned: absent or `Failed` golden fails render only for
pool still waiting for its first disk, otherwise reconcile of healthy
pool would abort over source it no longer reads.

Catalog side refuses duplicate `(schematicID, version)` entries, entries
whose schematic or version cannot produce name pool can reconstruct, and
edit to `storage`, `storageClass` or source URL of entry whose golden
already exists. Last one is not patchable in place, so without guard it
lands as rejected patch on next upgrade.

Golden that declares no StorageClass is read from its PVC, not from
whatever default is in force today, because those differ once cluster
default moves. Cluster with two default classes resolves same way
Kubernetes does, newest wins. Pool that already cloned is identified by
annotation naming exactly that pool, not by disk name prefix, because
pool `md0-gpu` produces disks carrying `md0` prefix.

Three drift guards live in
`hack/check-worker-image-catalog-defaults.bats`, because values that
have to agree sit in files nobody edits together. One-sided talos bump
trips all three.

### Testing

- `helm unittest`: kubernetes-nodes 117, kubernetes 145,
kubernetes-worker-image 14, core/platform 143, render snapshot unchanged
- full `make unit-tests`: 82 charts, 1742 helm-unittest cases, 1485 bats
cases, exit 0
- mutation pass over every guard term this diff adds: each term deleted
in turn, suite required to notice. Four terms are not caught by `helm
unittest`, and the count an earlier revision of this description carried
was wrong. One is the second support-matrix call, pinned instead by the
greps in `hack/check-worker-image-catalog-defaults.bats:96-110`. The
other three were reported by @IvanHunters and re-confirmed here by
re-running his mutations: dropping the written
`kubernetes-worker-image.cozystack.io/pool` annotation leaves the suite
green (only the reader is asserted), replacing the legacy
`storageclass.beta.kubernetes.io/is-default-class` disjunct with `false`
leaves it green, and forcing the `if $goldenStorage` presence term open
leaves it green while forcing it shut reddens two cases. All three are
test-coverage gaps rather than live defects, and they are deliberately
left open: this round closes only the two findings that change behaviour
- `go build` and `go vet` on root and `api/apps/v1alpha1`
- root `make generate` and `pre-commit` clean

### Verified on a live cluster

The clone path was exercised end to end on a throwaway three-node Talos
stand (OCI, Talos v1.13.6, LINSTOR/ZFS, `replicated` StorageClass)
installed from this branch's `packages/` tree. This covers what neither
`helm template` nor the unit suites can answer.

- The catalog wires up from `bundles.enabledPackages` alone:
`PackageSource cozystack.kubernetes-worker-image` reconciled, its
HelmRelease installed, and the golden `talos-worker-<schematic>-v1.13.6`
imported to `Succeeded` in 108s (6Gi, RWX, `replicated`).
- A pool on `osImage.builtin` clones it on the storage layer. Both
worker DataVolumes went `CSICloneInProgress` then `Succeeded` in 43s
with `Running=False reason=Completed message="Clone Complete"`, carrying
`source.pvc` and no `source.http`, and no importer or upload Pod
appeared in the tenant namespace.
- The clone yields a bootable Talos root disk. Both worker VMs reached
`Running`/`Ready`, joined the tenant cluster and went `Ready` about 66s
after creation, and the tenant's cilium, coredns, csi, ingress-nginx and
metrics-server addons all installed on them.
- Access modes match across the hop: golden `ReadWriteMany` 6Gi to
worker `ReadWriteMany` 20Gi on the same `replicated` class, grown by
CDI.
- The written annotations are correct in practice: each worker disk
carries `kubernetes-worker-image.cozystack.io/golden` naming the golden
and `.../pool` naming `kubernetes-wimg-md0`.
- helm-controller converges on a builtin pool: forcing a reconcile left
the content-hashed `KubevirtMachineTemplate` name unchanged and the VMI
and DataVolume UIDs identical, so a reconcile does not roll the pool.
- Cross-namespace clone RBAC is in place as shipped: `create datavolumes
--subresource=source -n cozy-public` resolves to `yes` for a tenant
ServiceAccount through the `cdi-clone-dv` RoleBinding, and CDI runs with
`cloneStrategyOverride: csi-clone`.

Both behavioural findings were reproduced on the same stand. A
`KubernetesNodes` carrying `talos.version: v1.13` was admitted by the
API with the value intact and its HelmRelease went `InstallFailed`
naming the format; a subsequent unrelated edit (`maxReplicas` 1 to 3)
was accepted by the CR and reached `.spec.values` of the HelmRelease
while it stayed `InstallFailed`, so the edit is taken everywhere and
applied nowhere. And a pool on `osImage.builtin` with
`talos.imageFactoryURL: ""` -- a value the API preserves rather than
defaulting away -- renders and clones normally with the guard scoped,
which is the fix in this round.

### Screenshots

Not applicable, no UI change.

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [x]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
cozystack/terraform-provider-cozystack#39
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
feat(kubernetes): worker pool can boot from shared golden talos image instead of importing OS image over HTTP per worker. Enable optional `cozystack.kubernetes-worker-image` package to seed one golden DataVolume per (schematicID, version) in `cozy-public`, configure it through `kubernetesWorkerImage` in cozystack-platform values, then set `osImage.builtin` on `KubernetesNodes` pool to clone it. `osImage.factory` keeps HTTP path and adds per-pool overrides for `imageFactoryURL`, `schematicID` and `version`. Pools that leave `osImage` unset render as before and their workers do not roll.

The render also refuses a `talos.schematicID`, `talos.version` or `imageFactoryURL` -- and their `osImage.*` overrides -- whose shape cannot produce a working image reference: a plain lowercase alphanumeric schematic, a full `vMAJOR.MINOR.PATCH` release, and a plain `http(s)` URL. Every refused shape already produced an unbootable worker before this change, but it rendered, so a pool holding one moves from Ready-with-no-workers to a failed HelmRelease. The values schema types all three as bare strings, so the rejection lands on the HelmRelease rather than on the CR: the CR is admitted with the bad value, and later edits to unrelated fields on that pool are accepted and reach the HelmRelease values without being applied. `imageFactoryURL` is checked only on the paths that build a source URL from it, so a pool on `osImage.builtin` may leave it blank.
```