feat(cluster-api): add the Proxmox infrastructure provider and in-cluster IPAM as optional packages - #4087

…ntegration (#4439)

## What this PR does

Nominates @themoriarti (Marian Koreniuk) as a **Reviewer** for the
Proxmox integration, following the process in
[CONTRIBUTOR_LADDER.md](https://github.com/cozystack/cozystack/blob/main/CONTRIBUTOR_LADDER.md#reviewer):

- `.github/CODEOWNERS`: adds @themoriarti as a code owner of
`packages/system/capi-providers-infraprovider-proxmox/` and
`packages/system/proxmox-*/` (Proxmox CCM and CSI). Each rule keeps the
full `/packages/system/` owner set, so no existing owner loses coverage,
and the catch-all invariant (`hack/codeowners-invariant.bats`) holds.
- `MAINTAINERS.md`: adds him to the Reviewers table.

The scope is deliberately narrow: shared code such as
`packages/apps/kubernetes*`, `api/apps` and the generic in-cluster IPAM
provider stays with the existing owners.

### Why

Marian has maintained the Proxmox integration since the first Proxmox
bundle and is now landing the Cluster API infrastructure provider, CCM
and CSI (#4087, #4088, #4089). Against the Reviewer requirements:

- well over 10 accepted PRs, contributing since 2024;
- has reviewed more than 20 PRs;
- in-depth knowledge of, and commitment to, the Proxmox area.

**Sponsors:** @kvaps (Ænix) and @kingdonb (Urmanac), which satisfies the
"at least one sponsor from a different employer" requirement.

### Notes for merging

- The Proxmox package directories arrive with #4087 and #4089. Merging
this PR after them is cleanest, though a rule for a path that does not
exist yet is harmless.
- GitHub silently ignores CODEOWNERS handles without write access. After
merge, an org admin needs to add @themoriarti as a repository
collaborator with the **Write** role (currently read).

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
docs(governance): add @themoriarti as Reviewer for the Proxmox integration.
```

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Chores**
* Expanded code ownership and review coverage for Proxmox integrations,
including infrastructure provider, cloud controller manager, and storage
components. No end-user functionality changed.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

…rker pools (#4088)

## What this PR does

**Series behind #3481** — three separable pull requests:

1. #4087 — the Cluster API providers (capmox + in-cluster IPAM)
2. **#4088 — the `substrate` switch** in `kubernetes` and
`kubernetes-nodes`
3. #4089 — the Proxmox CSI driver and cloud-controller-manager

#4087 and #4088 share no files and can be reviewed in parallel. #4089
builds on this one.

---

#4087 adds the Cluster API providers; this PR teaches the two
managed-cluster
charts to point at them.

### The shape

A single value, `substrate: kubevirt | proxmox`, on both charts — not a
parallel
set of charts. The control plane is Kamaji either way; what changes is
the
infrastructure half:

| | `kubevirt` (default) | `proxmox` |
|---|---|---|
| `Cluster.infrastructureRef` | `KubevirtCluster` | `ProxmoxCluster` |
| worker template | `KubevirtMachineTemplate` | `ProxmoxMachineTemplate`
|
| apiserver Service | `ClusterIP` | `LoadBalancer` |
| tenant CCM/CSI | in-cluster (`kccm`, kubevirt-csi) | rendered off; the
Proxmox pair is #4089 |

The KubeVirt path renders byte-for-byte what it rendered before. Checked
by
rendering `kubernetes-nodes` with default values at `main` and at this
head and
diffing the output, not only by the snapshots — the snapshots cover
`nodegroup.yaml` alone, and an earlier revision of this branch changed
the
reconcile Job while they stayed green. The parent chart's default render
differs
only in the random fields of `talos-secrets`, which differ between two
renders of
`main` as well.

`osImage` (#4171) is refused on the proxmox substrate. Only half of it
could be
honoured there — `factory.version` reaches the machineconfig's installer
image
while the worker disk keeps coming from the Proxmox template — and a
field that
half-works is worse than one that says no. `talos.version` and
`talos.schematicID` still set the installer target.

### Three things that only show up once workers are off-cluster

**The apiserver endpoint.** A KubeVirt worker runs inside this cluster
and
reaches the tenant apiserver by ClusterIP. A Proxmox worker does not: it
lives on
the hypervisor's L2 and cannot route to `10.96.0.0/16`. The machine
provisions
fine, `VMProvisioned=True`, `providerID` set — and then never joins,
with no CSR
to show for it. On the proxmox substrate the Kamaji Service is therefore
a
`LoadBalancer`, and the reconcile Job reads
`.status.loadBalancer.ingress[0].ip` instead of `.spec.clusterIP`.

**And DNS, for the same reason.** The worker machineconfig carried
`nameservers: ${COREDNS_IP}` — the management cluster's CoreDNS
ClusterIP,
equally unreachable. The proxmox path takes its own resolvers
(`proxmox.dnsServers`) and the chart refuses to render without them,
rather than
writing an address that cannot answer.

**`externalManagedControlPlane` and the managed-by annotation.**
capmox's
webhook nil-derefs on a `ProxmoxCluster` whose control-plane endpoint is
empty
unless `spec.externalManagedControlPlane` is set, which is correct here
— Kamaji
owns the endpoint. Separately, the `cluster.x-k8s.io/managed-by: kamaji`
annotation that the KubeVirt path carries must **not** be set on the
Proxmox
object: with it, CAPI never sets an ownerRef and capmox stops at
`Waiting for Cluster Controller to set OwnerRef`, leaving the address
pool
uncreated and every worker waiting on infrastructure.

### Linked clones by default

`proxmox.full: false` — worker disks are linked clones of the template
rather
than full copies. Measured on a ZFS-backed store: **8 KiB per worker
instead of
10.2 GiB**. Operators who want independent disks set `full: true`; the
value is
plumbed straight through to `ProxmoxMachineTemplate`.

`proxmox.storage` is a full-clone parameter and is refused with a linked
clone:
a linked clone always lives on the template's storage, and both Proxmox
(`parameter 'storage' not allowed for linked clones`) and capmox's own
CRD
(`Must set full=true when specifying storage`) reject the pair. The
chart says
so at render time, naming both values, instead of failing at apply time
with an
error about a `ProxmoxMachineTemplate` nobody wrote.

### Sizing

`ProxmoxMachineTemplate` takes `numCores` and `memoryMiB`, not
Kubernetes
quantities, so the pool's `resources.cpu`/`memory` are converted
(millicores →
cores, bytes → MiB, both rounded up). The rounding is pinned in
`tests/substrate_branches_test.yaml` with `1500m` and `8G`, where
rounding up and
down disagree; `tests/substrate_proxmox_sizing_test.yaml` covers the
refusal to
size a pool that gives neither.

### Known upstream behaviour this surfaces

`TalosConfigTemplate.spec` is immutable by schema, so the reconcile Job
cannot
rewrite an endpoint it has already published — `kubectl apply` fails
with
`TalosConfigTemplate.Spec is immutable`, and the template has to be
deleted for
the Job to recreate it. A fresh install never meets it; it appears when
the
apiserver address changes on an existing cluster. Handling it changes
the
KubeVirt path too, so it is not in this PR: it is a separate change on
`proxmox/tct-replace`, with its own upgrade note.

### Evidence

```
$ KUBECONFIG=<tenant> kubectl get nodes -o wide
kubernetes-px-md0-jmzds-ndnl6  Ready  <none>  75s  v1.35.6  10.0.0.160  Talos (v1.13.6)

$ KUBECONFIG=<tenant> kubectl get pods -A
cozy-cilium  cilium-…            Running
kube-system  konnectivity-agent  Running
kube-system  coredns-…           Running
```

No management-cluster address remains in the rendered
`TalosConfigTemplate`.

### Testing

`make unit-tests` — exit 0. `apps/kubernetes` 26 suites / 314 tests;
`apps/kubernetes-nodes` 29 suites / 160 tests / 12 snapshots, all
unchanged on
the default path. Each of the ten commits passes both suites on its own.

Every substrate branch is asserted on both sides, and every assert goes
red on
the mutation it exists for: inverting the `SVC_IP` condition, inverting
the
`nameservers` condition, `ceil`→`floor` in the sizing, removing the
proxmox
gate from each of the kccm and csi templates (the two RBAC bindings
included),
pinning the retention lookup back to `KubevirtMachineTemplate`, widening
the
IPv6 class of the `dnsServers` validation, dropping the parent chart's
`dnsServers` validation, the storage-with-linked-clone refusal, and the
`osImage` refusal. Negated `containsDocument` asserts carry `any: true`.

`hack/talos-reconcile-heredoc_test.bats` additionally runs the rendered
heredoc
through a real shell: a `dnsServers` entry with a substitution is
refused and
named, an IPv6-shaped one with a substitution is refused too, and a real
address survives the shell unchanged.

### Screenshots

Not applicable — no UI change.

### Downstream repositories

I walked the trigger map in `docs/agents/contributing.md` against this
diff, file by file. What it matches is below. **No box is ticked**,
because every follow-up it points at depends on a decision that is not
mine to make: #3481 asks whether Proxmox becomes a supported platform at
all, and writing its documentation or its Terraform schema before that
is answered would be work built on an assumption. I have not filed
speculative PRs or issues in those repositories for the same reason. If
the direction is accepted, I open them; a human should decide which.

**cozystack/terraform-provider-cozystack — matches, no follow-up yet.**

- `packages/apps/kubernetes/values.schema.json` and
`packages/apps/kubernetes-nodes/values.schema.json` both gain fields:
`substrate`, and a `proxmox` object (`allowedNodes`, `dnsServers`,
`ipv4Config`, and on the pool side `templateTags`, `network`, `storage`,
`full`). The provider hand-maintains a schema, a model and an
expand/flatten pair per Kind, with no codegen from these files.
- `values.yaml` gains defaults for those fields (`substrate: kubevirt`,
`proxmox.full: false`, `prefix: 24`). The provider restates defaults it
models and sends its own, so a stale one wins silently rather than
showing as a diff.

**Everything else: no match.** No app added, renamed or removed (website
app lists); no version enum, `kind`, `plural`, `release.prefix` or
output Secret/Service name changed; `packages/core/platform/values.yaml`
untouched; the `*-rd` files are regenerated, not renamed (ccp); no
`hack/` change; no node prerequisites (talm, ansible); no telemetry
metric; no commit-convention change (community).

- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
feat(kubernetes): add a `substrate` value to the `kubernetes` and `kubernetes-nodes` applications, selecting whether worker VMs run on KubeVirt in this cluster (`kubevirt`, the default and unchanged) or on an external Proxmox VE cluster through capmox (`proxmox`). On the Proxmox substrate the tenant apiserver is published through a LoadBalancer and worker disks default to linked clones.
```

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added support for choosing KubeVirt or Proxmox for Kubernetes clusters
and node pools.
* Added Proxmox configuration for nodes, networking, storage, DNS,
templates, cloning, and IPv4 allocation.
* Proxmox deployments now use the appropriate cluster, machine,
networking, and control-plane resources.

* **Validation**
* Added checks for supported substrates, required Proxmox settings,
sizing, and incompatible options.

* **Documentation & Tests**
* Documented the new settings and expanded coverage for rendering,
validation, cloning, sizing, and template replacement.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

…olds (#4483)

## What this PR does

`capi-providers-core` ships the Cluster API core manifests twice: as
`files/core-components.yaml`, which a reviewer reads, and as
`files/components.gz`, which the chart mounts into the cluster. #2946
added a `startupProbe` to `capi-controller-manager` and edited only the
YAML; the blob was never regenerated, so the probe never reached a
cluster. This regenerates the blob from the YAML (`gzip -9 -n -c
files/core-components.yaml > files/components.gz`), so the decompressed
blob matches the YAML byte for byte, and the only change it carries is
the six lines of #2946.

### What this reaches, and what it does not

New installs only. capi-operator re-applies a provider when the hash of
its spec and its config Secret changes (`applied-spec-hash`); the
ConfigMap holding the components is in neither, so a cluster that
already runs `capi-providers-core` keeps its manager without the probe
until the provider's version moves. Whether existing clusters should get
the probe, and how, is left open on purpose: moving `spec.version` and
the ConfigMap name together would force the re-apply, but that is a
version decision for the package, not a side effect of regenerating a
blob.

### Also in this diff

- `.helmignore`: `files/.*-components.yaml` is a glob, so it matched
only names beginning with a dot, and `helm package` shipped the readable
YAML alongside the blob that duplicates it. Now
`files/*-components.yaml`.
- `hack/capi-core-components_test.bats` holds the two files to the same
bytes from now on; it goes red on a YAML edited without regenerating the
blob.

Found by the drift check written for the Proxmox provider packages in
#4087, pointed at the package that was already here; split out because
it changes what every new install runs and deserves its own release
note.

### Testing

`bats hack/capi-core-components_test.bats`: 1/1; red with a line
appended to the YAML, green again after `gzip -9 -n -c`. `helm package`
of the chart now contains `files/components.gz` and
`files/metadata.yaml` only.

### Screenshots

Not applicable, no UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change

### Release note

```release-note
fix(cluster-api): ship the `startupProbe` from #2946 in the packaged core provider components. New installs get it; a cluster that already runs the core provider keeps its manager without the probe until the provider's version moves, because capi-operator re-applies components only when the provider spec or its config Secret changes.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Tests**
* Added a check that the compressed core components match the YAML
source, with a diff provided when they differ.
* **Chores**
* Updated chart packaging exclusions to cover component YAML files
regardless of whether their names begin with a dot.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

…x-backed tenants (#4089)

## What this PR does

**Series behind #3481** — three separable pull requests:

1. #4087 — the Cluster API providers (capmox + in-cluster IPAM)
2. #4088 — the `substrate` switch in `kubernetes` and `kubernetes-nodes`
3. #4089 — the Proxmox CSI driver and cloud-controller-manager

#4088 has merged, so this diff is the eight commits on top of it:
`proxmox.pool`,
`csi-proxmox`, `ccm-proxmox`, `delete.yaml`, the tests, the e2e suite,
one README
sentence on token privileges, and the reconcile Job's
replace-on-immutable step
for existing clusters. The `substrate` value this gates on is on `main`.

### One chain, not two components

Storage and the cloud-controller-manager are one chain, not two
components:

```
kubelet --cloud-provider=external
   → taint node.cloudprovider.kubernetes.io/uninitialized
      → CCM applies topology.kubernetes.io/region|zone + providerID
         → CSI node plugin starts
            → PVC binds
```

The CSI node plugin reads region and zone off its own Node and exits
fatally
without them:

```
F main.go:126] Failed to get region or zone for node: …, region: , zone: ,
   see documentation about topology labels
```

Those labels come from the cloud-controller-manager, and the CCM ignores
a Node
until the kubelet's `--cloud-provider=external` has tainted it. Shipping
the CSI
driver alone gives a tenant a StorageClass, a healthy controller, and a
node
plugin in CrashLoopBackOff.

The CSI driver has no equivalent knob for this. Its `features.provider`
accepts
`default` and `capmox` and validates neither, so an unrecognised value
falls
through to the default path — the key is therefore not set at all. The
cloud-controller-manager is the opposite case: `capmox` is a real mode
there and
is set, because the default one writes a Node providerID of
`proxmox://<node>/<vmid>` while capmox has already written
`proxmox://<SMBIOS UUID>` on the Machine.

So the CCM has no `enabled` switch and its credentials reference is
required:
on this substrate, a cluster without a CCM is a cluster where every
worker keeps
the `uninitialized` taint and nothing schedules. A chart that cannot
name the
Secret should stop rather than render that. The CSI release depends on
cilium
only; the CCM is a Deployment of this chart, not a HelmRelease, so
nothing in
the tenant can depend on it — the node plugin waits on the labels the
CCM
applies, which is the ordering that matters.

### Packaging

| Package | Chart | App |
|---|---|---|
| `packages/system/proxmox-csi` | proxmox-csi-plugin 0.5.12 | v0.20.0 |

The vendored chart is the tenant-side half only. It **drops the upstream
`values.talos.yaml` preset**, which pins the workloads onto
`node-role.kubernetes.io/control-plane` and
`node.cloudprovider.kubernetes.io/platform: nocloud`: a Kamaji tenant
has no
control-plane nodes at all — its control plane is a Deployment in the
management
cluster — so that preset leaves the components unschedulable. It is
published as
a component of the existing `kubevirt` variant of
`kubernetes-application`
rather than in a new variant: a second variant would rename the
artifacts of
all eighteen tenant addons, and exactly one component differs between
the
substrates.

The cloud-controller-manager is not a vendored chart. It is a Deployment
the
`kubernetes` chart renders itself
(`templates/ccm-proxmox/controller.yaml`),
because it runs here, in the management cluster, and there is nothing of
it to
install in the tenant.

Credentials never reach the tenant. Each controller names a Secret in
the
tenant's namespace of this cluster (`proxmox.csi.credentialsSecretName`,
`proxmox.ccm.credentialsSecretName`), an init container composes the
config
file from it into a memory-backed emptyDir at start-up, and the
`kubernetes`
release stores only the Secret's name. They are separate knobs from the
capmox
credentials, and from each other, so an operator can scope them:
attaching
disks is a different blast radius from creating VMs, and the CCM only
reads
inventory. `proxmox.insecure` defaults to `false`; a stand whose Proxmox
serves
a self-signed certificate sets it to `true`. `proxmox.csi.enabled:
false`
switches off the controller as well as the tenant release, and needs no
CSI
token.

### Where the hypervisor token lives, and why it decides the shape

Both controllers run in the tenant's namespace of **this** cluster and
drive the
tenant over its admin kubeconfig. Only the CSI node plugin goes into the
tenant,
and it mounts no credentials.

Installing them into the tenant would hand the token to the tenant. The
resource
definition lists the admin kubeconfig in `secrets.include`, so a tenant
is
cluster-admin of its own cluster, and the vendored chart writes the
driver's
`config.yaml` — token included — wherever it is installed. The
privileges the
driver documents are granted on `/`, so such a token reaches every VM
and every
volume on the Proxmox cluster. The KubeVirt path already has the shape
that
works, in `templates/csi/deploy.yaml`, and the upstream components
support it:
the CSI controller's `--kubeconfig` is documented for running out of
cluster.

Three consequences of the split are handled rather than left to be
discovered:
the vendored chart has no switch for its controller, so the tenant
release sets
`replicaCount: 0`; its Secret template is gated on a non-empty
`config.clusters`,
so an empty list keeps the credential out; and `CSIStorageCapacity` is
published
by the provisioner sidecar, which now owns those objects through a Pod
that does
not exist in the tenant, so `CSIDriver.storageCapacity` is patched to
false by a
kustomize postRenderer — the chart hardcodes it, and left true the
scheduler
refuses every Pod with "node(s) did not have enough free storage".

Volumes are owned per cluster. The driver names them
`vm-<controllerVmID>-pvc-…`
and owns every volume as VMID 9999 unless `controllerVmID` is set, which
leaves
different clusters' volumes indistinguishable on shared storage. Each
cluster
sets its own, derived from its tenant namespace and release name, so two
tenants' clusters with the same name still get different owners.

**This does not make one tenant unable to reach another's volumes.** A
tenant
can still create a static PV naming a foreign `volumeHandle`, and the
controller
will attach it, detach it or delete it: the driver checks neither the
VMID a
node ID names nor the volume a handle names against the cluster it
serves.
Closing that needs a per-tenant Proxmox scope, and the charts can now
produce
one: `proxmox.pool` on the `kubernetes-nodes` chart puts every worker VM
into a
Proxmox resource pool as capmox clones it, replacements included, so one
ACL on
`/pool/<name>` follows every VM without being granted again. The README
states
the scope where an operator configuring the chart reads: a storage of
the
tenant's own (`Datastore.Allocate` on a storage reaches every volume on
it),
`VM.Audit`/`VM.Config.Disk` on that pool for the CSI controller,
`VM.Audit` and
`VM.GuestAgent.Audit` on it for the CCM — a NotReady Node whose VM the
token
cannot see is reported as gone and deleted — and what that scope still
leaves
open: the tenant's own disks, which a cluster-admin of the tenant could
reach
anyway.

### Deleting a cluster frees its disks

The pre-delete hook deletes the CSI driver in Step 2, and a volume is
freed only
by the driver answering DeleteVolume, so a Step 1b runs first: it
deletes the
tenant's claims (`--wait=false`; a claim in use carries
`kubernetes.io/pvc-protection`, so it clears once nothing mounts it),
then the Pods holding each claim, then waits for the PVs rather than the
PVCs.
The Pods are found by listing every Pod with the claims it mounts and
matching
in the shell: a JSONPath filter on the claim name fails for a whole
namespace as
soon as one Pod there mounts two claims, and would find no Pod at all.
The claims go first because a claim being deleted is one the scheduler
places
no new Pod on, so a Pod that a StatefulSet brings back cannot mount it
again.
Without the step the disks stayed on the hypervisor with the tenant's
data in
them and no PV object left to name their owner. A tenant apiserver that
does not
answer is told apart from a tenant with no claims: the first goes
through the
hook's backstop and shows in its summary, the second is the clean case.
The
probe is bounded with `timeout`, not `--request-timeout`, which the
repo's hook
check refuses because a non-zero value diverts an in-cluster kubectl
call to
localhost:8080.

On the KubeVirt path two things change. The pre-delete hook's Step 5
checks
that the DataVolume CRD exists before deleting DataVolumes, so a hook
that runs
where CDI is not installed no longer fails after every earlier step has
run; the
hook runs only at deletion. The worker pool's talos-reconcile Job
changes at
upgrade: its Role's TalosConfigTemplate rights are narrowed to the
pool's own
template and gain a named `delete`, and its script gains the replace
step, which
moves the Job's content-hash name. Every existing worker pool therefore
gets a
new reconcile Job when it upgrades. Where a pool's live
TalosConfigTemplate still
matches the render, that Job changes nothing; where it has drifted, the
Job now
deletes and recreates it instead of failing. Machines already running
keep the
bootstrap data they joined with; Machines created afterwards boot from
the
current template.

### Existing clusters: the TalosConfigTemplate CABPT will not let us
patch

CABPT's validating webhook rejects any change to
`TalosConfigTemplate.spec`
(`TalosConfigTemplate.Spec is immutable`). The kubelet flag this PR adds
changes that spec, so on a cluster created before it the reconcile Job's
`kubectl
apply` fails with `is immutable` on every run, and a pool whose
`ProxmoxMachineTemplate` also changed (say, `proxmox.pool` set) rolls
its workers
off the old template: they join without `--cloud-provider=external`, the
CCM
leaves them unmanaged, and the CSI node plugin exits on the missing zone
label.
The last commit makes the Job replace the object when apply reports the
spec
immutable: delete, apply again, log both. Existing Machines keep their
bootstrap
data; new ones are built from the replaced template. Pinned by
`talos_tct_immutable_test.yaml`, and the RBAC the Job runs with covers
exactly
that delete (`resourceNames` of its own template).

### Evidence

```
$ KUBECONFIG=<tenant> kubectl get node -o custom-columns=…
kubernetes-px3-md0-rp4s5-rgr48  prox-home  prox-home  proxmox://b903f905-7b4f-4048-b1b6-3e8f58530d8f

$ kubectl -n tenant-proxmox3 get deploy
kubernetes-px3-pcsi-controller  1/1     kubernetes-px3-pccm  1/1

$ KUBECONFIG=<tenant> kubectl get pods -n csi-proxmox
csi-plugin-node  3/3  Running        (the controller Deployment has 0 replicas)

$ KUBECONFIG=<tenant> kubectl get secret -A | grep -c token_secret
0

$ KUBECONFIG=<tenant> kubectl get pvc csi-probe
csi-probe  Bound  pvc-…  1Gi  RWO  proxmox
```

That the volume is real, not merely `Bound` in the API, and that it is
this
tenant's:

```
$ qm config <vmid>
scsi1: main-pool:vm-798497-pvc-…,size=1G
```

Deleting the PVC frees the disk from the storage; the e2e probe checks
that too.

### Known limitation

The CCM logs every five minutes on PVE 9:

```
error get ha-groups 500 cannot index groups: ha groups have been migrated to rules
```

Proxmox 9 changed the HA-groups API. Node initialization is unaffected —
labels
and providerID are applied — but the CCM image v0.15.0's HA-groups
lookup is
not compatible with PVE 9.

### Testing

`make unit-tests` — exit 0. `apps/kubernetes` 28 suites / 338 tests;
`apps/kubernetes-nodes` 33 suites / 182 tests / 12 snapshots. Each of
the eight
commits passes both suites on its own.

What the new asserts pin, each shown to go red on the mutation it exists
for:
the tenant release carries no credentials and no `valuesFrom`; the
controllers
run here with the Secret mounted and the tenant as their target;
`csi.enabled:
false` renders no CSI controller and needs no CSI token; the kubelet
runs with
`cloud-provider: external` on proxmox and without it on kubevirt; Step
1b and
the cleanup Role's admin-kubeconfig entry render on proxmox and not on
kubevirt,
and Step 1b deletes the claims before the Pods; `proxmox.pool` reaches
the
`ProxmoxMachineTemplate` and an empty one renders no `pool`; each tenant
gets
its own volume owner id; the CCM runs in capmox mode; the CCM tolerates
exactly
two taints. `proxmox.ccm`, `proxmox.csi` and `proxmox.insecure` are
optional
in the schema — each has a default, and the render enforces what the
substrate
needs — so a `Kubernetes` object written against `main`'s `proxmox`
block stays
valid; the API review gate reports no sizeable change at any of the
commits.

The e2e suite ships `.disabled` and no CI lane runs it, so the substrate
has no
automated coverage until a runner with the `proxmox` label exists. What
is in
the tree is runnable: the fixture is rendered by `run-proxmox.sh` with
explicit
defaults (GNU `envsubst` does not expand `${VAR:-default}`, so the
defaults live
in the script and the render refuses a fixture with a variable left in
it —
pinned by `hack/proxmox-e2e-fixture_test.bats`), the runner reads the
`.conf`
kubeconfig a runner outside the management cluster can reach, the
StorageClass
and the PVC probe read the same storage variable, the probe checks the
disk is
gone after the claim is deleted, `keep_tenant` is honoured by the
suite's own
teardown, the workflow fails on an unset stand variable instead of
falling back
to a hardcoded address, and its schedule runs only where
`vars.PROXMOX_NIGHTLY`
is set, since the job needs a self-hosted runner with the `proxmox`
label. The
fixture takes the stand's resource pool through `COZY_PVE_POOL`.

### Screenshots

Not applicable — no UI change.

### Downstream repositories

I walked the trigger map in `docs/agents/contributing.md` against this
diff, file by file. What it matches is below. **No box is ticked**,
because every follow-up it points at depends on a decision that is not
mine to make: #3481 asks whether Proxmox becomes a supported platform at
all, and writing its documentation or its Terraform schema before that
is answered would be work built on an assumption. I have not filed
speculative PRs or issues in those repositories for the same reason. If
the direction is accepted, I open them; a human should decide which.

**cozystack/terraform-provider-cozystack — matches, no follow-up yet.**

- `packages/apps/kubernetes/values.schema.json` gains
`proxmox.ccm.credentialsSecretName`, `proxmox.csi` (`enabled`,
`credentialsSecretName`, `storageClasses[]`) and `proxmox.insecure`,
with defaults in `values.yaml`. Same coupling as the substrate PR:
hand-written schema, model and expand/flatten, and defaults the provider
restates and sends itself.

**cozystack/website — unclear, deliberately left unticked.** This PR
adds one package under `packages/system/`, not under `packages/apps/` or
`packages/extra/`, so the app-list trigger does not fire. Whether
`proxmox-csi` counts as a "platform component" for
`guides/platform-stack/_index.md` is a judgement about that page's
scope: it is not part of the platform stack — it is an addon installed
*into* a tenant cluster, like the KubeVirt CSI driver, which has no
entry there either. I read that as no, but I am not confident enough to
decide it for you, so the box stays empty. The README section on token
privileges and per-tenant scope is the kind of text `operations/` pages
carry; that follows the decision #3481 asks for.

**Everything else: no match.** No app package added or renamed; no
version enum, `kind`, `plural` or `release.prefix` change;
`packages/core/platform/values.yaml` untouched;
`packages/core/platform/sources/kubernetes-application.yaml` gains one
component but is not a renamed `*-rd` file (ccp); `hack/` gains
`e2e-chainsaw/_lib/run-proxmox.sh`, a suite directory shipped
`.disabled`, and `proxmox-e2e-fixture_test.bats`, which runs with the
other bats files; no `ApplicationDefinition` semantics or
`release.chartRef.kind` enum change (external-apps-example); no node
prerequisites; no telemetry metric; no commit-convention change.

- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
feat(kubernetes): add the Proxmox CSI driver and cloud-controller-manager for tenant clusters on the `proxmox` substrate, giving them PersistentVolumes backed by Proxmox storage. The cloud-controller-manager is not optional there: worker kubelets run with `--cloud-provider=external`, so without it every node keeps the `uninitialized` taint and the CSI node plugin cannot start.
```

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

## New Features
- Added Proxmox VE as an infrastructure option alongside KubeVirt for
Kubernetes clusters and worker pools.
- Added configuration for Proxmox networking, DNS, VM templates,
storage, node placement, credentials, IPv4 allocation, and storage
classes.
- Added Proxmox cloud-controller-manager and CSI integrations.
- Added validation for required Proxmox settings and unsupported
combinations.

## Bug Fixes
- Improved recovery when Talos configuration templates cannot be updated
because they are immutable.

## Documentation
- Documented Proxmox configuration, requirements, and supported
parameters.

## Tests
- Added coverage for Proxmox rendering, sizing, integrations,
validation, and KubeVirt compatibility.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->


### Verified on a live stand

Run against three throwaway tenants on Proxmox VE 9.2, each built and
destroyed:

- every Secret in the tenant cluster scanned for the token — none found,
while
  the PVC round-trip works;
- a PVC bound, mounted, written and read back, its disk visible on the
  hypervisor as `vm-798497-pvc-…` rather than `vm-9999-…`;
- `helm uninstall` with the volume still mounted by a running Pod, after
which
  the disk is gone from the storage with nothing done by hand.

What that run pinned: the CCM's capmox mode, capacity-aware scheduling
off
with `CSIDriver.storageCapacity` patched, and the teardown Role's
admin-kubeconfig entry.

A second run, on the head of this PR, used a pool per tenant and two
privilege-separated tokens holding exactly the README's rows, on a
throwaway
tenant and then on the stand's long-lived cluster upgraded in place:

- the worker VM lands in the pool as capmox clones it, and so does its
replacement after the Machine is deleted — the CCM initialises the new
node and
  the CSI attaches a disk to it with no ACL touched;
- no `401` or `403` from the hypervisor in the CSI controller or the CCM
over
the whole run, once the roles were granted to the user as well as the
token;
- the PVC round-trip on both clusters, the disk gone from the storage
after the
claim; `helm uninstall` with a claim held by a running Pod deletes the
claim
  before the Pod and waits for the volume to be released;
- the in-place upgrade replaced the immutable template on the first
apply
conflict and the re-created worker came up with
`--cloud-provider=external`.