fix(kubernetes): keep tenant addon CRDs when a release is uninstalled - #3586

Actionable comments posted: 2

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@packages/system/vertical-pod-autoscaler-crds/Makefile`:
- Around line 12-19: Update the awk transformation in the Makefile’s update
target to guarantee that each CustomResourceDefinition receives the
helm.sh/resource-policy annotation even when metadata.annotations is absent.
Create the annotations block before adding the policy, or fail before replacing
the generated file if it cannot be inserted; do not allow update to finish with
an unannotated CRD.
- Line 16: Update the curl invocation in the Makefile download target to include
the --fail option, ensuring HTTP 4xx and 5xx responses stop the command before
the generated template is modified by the subsequent awk step.

Verified the mechanics end to end and they hold. Rendering all four charts, every
CRD carries helm.sh/resource-policy: keep (12 / 2 / 10 / 3) and nothing else
does — the Gateway API ValidatingAdmissionPolicy pair is correctly left out. I
re-ran the stamping awk over the five committed files and it is a no-op, and I ran
the real update: fetch for gateway-api-crds (kubectl kustomize ... ?ref=v1.5.1)
and vsnap-crd: both come back byte-identical to what is committed here.

The tests are not self-asserting. Dropping the annotation from one vsnap CRD turns
two cases red; appending an unannotated CRD to the Gateway API file turns the
wildcard case and the document-count pin red. hack/helm-unit-tests.sh is green
across the whole repo.

On the trade-off: reinstall after a keep-uninstall is safe. Helm excludes kept
resources from the delete set without touching their metadata, so
meta.helm.sh/release-name and -namespace survive; release name and
storageNamespace are fixed in these HelmReleases, so the retry install passes the
ownership check and adopts the CRDs rather than failing on "exists and cannot be
imported". Existing clusters need no migration either — the platform upgrade runs
helm upgrade on these releases and patches the annotation onto the CRDs already in
place. Coverage looks complete for the stated scope: every tenant addon that renders
CRDs from templates/ is now annotated, and the ones that ship CRDs under crds/
(velero, gpu-operator) were never at risk since Helm does not delete those.

Two things worth resolving, both about consistency inside this block of PRs rather
than correctness here.

The stamping step is copy-pasted three times, and #3595 is hardening the fourth
copy in the opposite direction. The same awk one-liner appears verbatim in
gateway-api-crds/Makefile:22, vertical-pod-autoscaler-crds/Makefile:18 and
vsnap-crd/Makefile:23, alongside a fourth, differently-shaped and non-idempotent
variant already in etcd-operator-crds/Makefile:20. #3595 gives that fourth one a
count guard — one annotation per CRD or the update fails — and defers writing
templates/ until after the check. The three added here have neither. A CRD that
arrives without a metadata.annotations block, or with annotations: {} in flow
style, is skipped silently, the step exits 0, and since rm -rf templates has
already run there is nothing to fall back to. The suite in this PR does catch it in
CI, which the description says, but #3595 has decided the guard belongs at the point
of failure. A shared hack/stamp-crd-keep.sh — the kind-aware idempotent matcher
from this PR plus the count guard from #3595 — would leave one implementation
instead of four.

The suites are much heavier than the equivalent in #3590. 387 of the 456 added
lines are tests, and most of that is repetition:
gateway-api-crds/tests/resource_policy_test.yaml:35-141 is twelve near-identical
cases that each re-check what the matchMany case at :18 already covers, and
prometheus-operator-crds/tests/resource_policy_test.yaml:24-142 is ten more. #3590
gets the same guarantee for cert-manager-crds in a single wildcard case. Worth
converging on one shape across the block.

Smaller points:

• The description says a CRD from a later upstream bump is "covered without editing
  the test", but hasDocuments: count: 14 at :32 (and count: 2 in the VPA suite)
  is exactly what will need editing on such a bump. Both properties are worth having;
  the sentence just reads as if the pin were not there.
• Three of the four suites scope to no template and assert
  kind == CustomResourceDefinition on every document. That is stricter than
  intended: a non-CRD resource appearing upstream would fail them for the wrong
  reason. The Gateway API suite gets this right with documentSelector + matchMany.
  Conversely it is the only one with a suite-level templates: (:15), so a second
  file under templates/ would go untested — moot today, since update: writes
  exactly one.
• renders twelve CRDs and the admission policy pair (:30) bakes the count into the
  case name as well as the assertion; the name will go stale first.
• vsnap-crd/Makefile:22 creates tmp outside the loop, so a mid-loop awk failure
  leaves it behind.

Out of scope, but the same class exists on the management cluster: kamaji-crds
(tenantcontrolplanes), piraeus-operator-crds, kubevirt-operator,
kubevirt-cdi-operator, metallb, objectstorage-controller,
mariadb-operator-crds, kubeovn and external-secrets all render CRDs from
templates/ without the annotation. Not urgent — those HelmReleases set no
install.remediation, and the default there is to perform no action — but the same
protection would cost the same.

CI is red on "Unit & controller tests" for an unrelated reason: the EXIT-trap freeze
guard in hack/cozyreport.bats:2553 does not list
hack/run-kubernetes-talos-diagnostics_test.bats, which landed on main with #3548.
This PR touches no .bats file.

Approving — none of the above blocks the fix.