Releases: cozystack/cozystack
Release list
v1.6.4-rc.1
[Backport release-1.6] feat(kubevirt): expose migration configuration…
v1.6.3
v1.6.3 (2026-09-03)
A patch release with a security fix restricting access to the kube-ovn admission webhook, four bug fixes covering managed Kubernetes, ClickHouse, PostgreSQL and telemetry, a reworked Marketplace/Admin dashboard, configurable ClickHouse versions, a WorkloadMonitor bucket-metrics endpoint override, a Keycloak KMS proxy update and Vault CA support, plus ansible-cozystack dependency bumps and a batch of new documentation.
Security
- fix(kubeovn-webhook): restrict the webhook port to the API server: The
kube-ovn-webhook/mutate-podshandler returned namespace-derived annotations to any caller that could reach its serving port, and nothing previously stopped an arbitrary in-cluster pod from being that caller. ACiliumNetworkPolicynow denies pod and world traffic to the webhook's port while allowing only the API server (which runs on the host network), closing the exposure tracked as GHSA-g883-q79m-8225 without needing the client-certificate path, which the platform's current admission configuration cannot use (@kvaps in #3955, backport #3975).
Features and Improvements
-
feat(dashboard): rework the Marketplace entry point and the Admin portal: The Console is now the dashboard's entry point instead of the Marketplace, so opening the dashboard shows what a tenant already runs rather than a "choose a product" landing page; the Marketplace opens the grouped application list directly, one click away. Administration (Tenants, Modules, External IPs, per-tenant Info) moves from Console into an always-visible Admin tab, while the two cluster-wide operator areas (Capacity, Backup Classes) keep their own gates. Tenants and Modules are now rendered as a tenant hierarchy tree instead of a flat table, so sub-tenants and inherited modules are visible instead of missing or erroring, and External IPs lists LoadBalancer services across every accessible tenant in one table. Secret values can be copied without being revealed on screen first (@kvaps in #3928, backport #3952).
-
feat(clickhouse): add configurable version parameter: The ClickHouse server and Keeper images were hardcoded to
24.9.2.42, so — unlike postgres, mariadb, mongodb, opensearch, rabbitmq and redis — ClickHouse instances carried no engineversionin the Cozystack API. Aversionvalues key (v25.8,v25.3,v24.9, defaultv24.9for byte-identical existing renders) now selects the server and Keeper image tags, withhack/update-versions.shrefreshing the supported-version map from Docker Hub (@scooby87 in #3476, backport #4054). -
feat(workloadmonitor): bucket metrics endpoint override:
cozystack-controllermeters S3 bucket sizes for billing by discovering a VictoriaMetrics endpoint through a namespace label, which assumed the monitoring stack scraping SeaweedFS always runs in the same cluster. A newcozystackController.seaweedfsMetricsEndpointchart value lets a deployment point bucket-size queries at a remote Prometheus-compatible endpoint instead, with no behavior change when unset. This also fixes a latent billing bug where any metrics-query failure silently zeroed out recorded bucket sizes; the controller now retains the last known values and surfaces the failure via aBucketMetricsUnavailableevent and error metrics instead of wiping billing data (@lllamnyp in #3443, backport #3558). -
feat(keycloak): let the KMS proxy trust a private Vault CA: The KMS-encrypting DB proxy could previously only reach a Vault whose certificate chains to a publicly trusted root, so a Vault behind an internal load balancer with a self-signed certificate was reachable only over plain HTTP.
encryption.kms.vaultnow accepts a CA either inline ascaBundleor via an existingcaSecretName/caSecretKey, letting the proxy verify a private Vault over HTTPS; existing installations are unaffected when neither key is set (@sircthulhu in #3874, backport #3981).
Fixes
-
fix(postgres): let in-place restore re-bootstrap instead of wedging: An in-place PostgreSQL
RestoreJob(restoring an app back into itself) never came back up — the cluster stayed inSetting up primarywith crash-looping recovery pods, or the HelmRelease upgrade failed outright. Two independent bugs in the restore patch caused this: the restored cluster's WAL-archiveserverNamecollided with the recovery source's, anduseSystemBucketstayed enabled alongside explicit restore S3 coordinates, tripping the chart's own compatibility guard. Both are fixed, so in-place restores now re-bootstrap to a healthy cluster instead of wedging (@androndo in #3959, backport #3988). -
fix(clickhouse): scheme the backup S3_ENDPOINT on the system-bucket flow: On the default
useSystemBucketbackup flow, the ClickHouse chart passed a scheme-less host straight into theclickhouse-backupsidecar'sS3_ENDPOINT, whichaltinity/clickhouse-backup's AWS SDK endpoint resolver rejects outright — so every ClickHouse backup and restore on the default flow failed withwas not a valid URI. The chart now prependshttps://on that path, matching how the platform already schemes the equivalent endpoint for other backup strategies, so default-flow ClickHouse backups and restores work again (@androndo in #3963, backport #3987). -
fix(kubernetes): retry the tenant CNI install instead of uninstalling it: The tenant Cilium HelmRelease had no install strategy, so Flux used its default
RemediateOnFailure, which uninstalls Cilium — the tenant cluster's only CNI — between retry attempts whenever the initial install ran long (for example while waiting for the first worker node to register), then reinstalls it on the next attempt, repeating without end. The strategy is nowRetryOnFailure, which keeps the applied manifests in place and retries a failed install as an upgrade, so the CNI is no longer torn down and the agent DaemonSet rolls out as soon as a worker registers (@lexfrei in #3552, backport #4033). -
fix(telemetry): make the documented opt-out cover both collectors: Telemetry is reported by two components —
cozystack-operatorandcozystack-controller— but only the operator had a reachable opt-out switch; the documentedcozystackOperator.disableTelemetrysetting could never reach the controller, which kept posting application counts every 15 minutes even after an operator followed the published opt-out procedure. The platform chart now threads atelemetry.disabledkey into the controller as well, so the documented opt-out actually disables both collectors (@kvaps in #3791, backport #3916).
Dependencies
-
chore(keycloak): bump KMS proxy to 0.2.2: Updates
keycloak-kms-proxyto v0.2.2 and pins the image to its published digest, fixing decryption for reads whose result set spans a join (@myasnikovdaniil in #3995, backport #3996). -
[ansible-cozystack] chore(deps): update k3s to v1.36.3+k3s1 and v1.36.4+k3s1: Renovate dependency bumps of the k3s Kubernetes distribution used by the management-cluster bootstrap role (@app/renovate in cozystack/ansible-cozystack#66, #69).
-
[ansible-cozystack] chore(deps): track cozystack installer v1.6.2: Renovate dependency bump of the
cozy-installerDocker image tag to follow the latest Cozystack patch release (@app/renovate in cozystack/ansible-cozystack#68).
Documentation
-
[website] fix(telemetry): document the opt-out for both collectors: Documents both opt-out steps now that the fix above threads
telemetry.disabledthrough the platform, replacing the previously incomplete single-step procedure (@kvaps in cozystack/website#651). -
[website] docs(home): refresh the screenshot gallery for the reworked console: Retakes the home-page screenshot gallery against the reworked console (Marketplace, Modules, Tenants, Admin), replacing slides that showed a header and sidebar layout that no longer exist (@kvaps in cozystack/website#662).
-
[website] docs(images): refresh the product screenshot and its dark variant: Retakes the standalone product screenshot referenced by the announcement post and outside links, which still showed a console two redesigns old (@kvaps in cozystack/website#663).
-
[website] feat(site): Live Demo — the real console, mocked, in the browser: Adds a
/demopage running the realcozystack-uiconsole with a mocked Kubernetes API, so a visitor can explore the actual marketplace and console without a cluster, signup, or install (@tym83 in cozystack/website#669). -
[website] feat(demo): self-refreshing console demo + Resources nav grouping: Adds the demo's source as an overlay with a weekly-refresh workflow so the mocked bundle tracks upstream
cozystack-uiautomatically, and groups secondary navigation links under a Resources menu (@tym83 in cozystack/website#670). -
[website] fix(demo): embed console under site nav, fix refresh + deep create: Fixes the live demo breaking on a hard refresh of any deep route by serving the SPA at a separate path and embedding it in an iframe under a real Hugo page, and keeps the site's own navigation visible above the demo (@tym83 in cozystack/website#676).
-
[website] fix(demo): create works for every catalog type: Fixes demo ...
v1.6.3-rc.2
[Backport release-1.6] feat(clickhouse): add configurable version par…
v1.6.3-rc.1
[Backport release-1.6] fix(kubernetes): retry the tenant CNI install …
v1.7.0-alpha.1
Cozystack v1.7.0-alpha.1
Preview build cut from main at 67b4e23. This is an alpha, not a release candidate: it is never promoted to a stable release and it carries no upgrade guarantee. 174 pull requests have landed on main since v1.6.0 — 29 features, 84 fixes, 4 of them breaking. The list below is a selection, not the full set.
Highlights
Worker pools become first-class objects
The kubernetes chart is now control-plane only. Worker pools moved out to their own KubernetesNodes resources, and migration 52 adopts existing pools on upgrade (#3315). Once a pool is its own object it can carry its own settings, and several arrived immediately: a node group can set the CPU request (#3862) and CPU limit (#3859) of its worker VMs, log the guest serial console (#3637), and route worker image pulls through a registry mirror (#3575).
ComputePlane — managed isolation for untrusted code
A new tenant module that provisions a single-tenant, Cozystack-managed cluster for catalog applications that execute user code (#3280), implementing the compute-plane design proposal. It is operator-owned: the Kamaji control plane, the KubeVirt-VM workers and the enablement addons are fixed by the module, so a tenant gets the isolated environment without owning the wiring. Turned on with the computeplane tenant toggle.
PostgreSQL point-in-time recovery
PITR is now a supported and tested capability of the backups.cozystack.io API rather than something an operator assembles by hand (#3383). A recovery target that lies past the end of the archive fails with a precise reason instead of a generic timeout, so a mistyped timestamp is diagnosable in seconds. MongoDB gains a native psmdb backup-strategy driver in the same API (#3562).
The dashboard opens on the Console
Visiting the dashboard now lands on the Console — what the tenant already runs — instead of the Marketplace landing page (#3928). The Marketplace becomes an ordinary destination one click away, and tenant-wide administration moves into the Admin portal. The Tenants page shows the tenant hierarchy, and External IPs are listed across every accessible tenant.
Tenant CA trust anchors across the catalog
The application CA is extracted into a key-free tenant Secret (#3407) and published as a ca.crt-only trust anchor by NATS (#3340) and Qdrant (#3341). MongoDB gets operator-managed TLS with external hostname SANs (#2692), the ClickHouse backup sidecar can trust a private S3 CA (#3385), and the Keycloak KMS proxy a private Vault CA (#3874).
Opt-in east-west encryption
networking.encryption.enabled turns on in-cluster transport encryption for the active datapath (#2984). It maps to KubeOVN's overlay IPsec, whose controller provisions the CA and per-node certificates through the Kubernetes CSR API — no cert-manager wiring, no per-application changes, pod-to-pod traffic encrypted whether or not the workload speaks TLS. Off by default.
No AGPL payload left in the tree
The FluxCD tenant addon (#3379) and the rebuilt Grafana image (#3378) were the last AGPL-3.0 content shipped by this Apache-2.0 repository, and both are gone. The addon is removed outright; Grafana now runs on the digest-pinned upstream image with signed catalog plugins installed at startup, which is possible because the VictoriaLogs datasource — the only reason for the rebuild — is now published in the official catalog.
kube-ovn builds in-tree
packages/system/kubeovn now pulls the upstream chart and builds its image in the monorepo, shaped like packages/system/cilium, retiring the cozystack/kubeovn-chart wrapper repository (#3460). One repository fewer, one release step fewer, and provenance a reader can follow without leaving the tree.
Breaking changes
spec.nodeGroupsis removed from theKubernetesCR (#3315). Worker pools, their health checks and the autoscaler provision-time budget now live on separateKubernetesNodesresources. Migration 52 adopts existing pools on upgrade.- The
addons.fluxcdoption of the Kubernetes app is removed (#3379). Tenant clusters that had it enabled keep their running Flux release; the companion migration carries the upgrade path. - Multus stages reference CNI plugins into the host
/opt/cni/bin(#3195), gated on the newnetworking.stageCniPluginsvalue and on for both bundles. Where staging is on the install is unconditional — a plugin is replaced on every pod recreation. - ApplicationDefinition release prefixes must be dot-free (#3411), constrained to
^[a-z0-9-]*$. A definition whose prefix carries a dot is rejected at admission and must be corrected before the CRD is applied.
Also in this build
- ClickHouse exposes a configurable version parameter (#3476).
- The Redis operator moves from the archived spotahome project to the freshworks-oss fork (#3406).
- Gateway API listeners advertise HTTP/2 via ALPN (#3398).
- ingress-nginx moves to controller v1.11.5 with the admission webhook restricted to the API server (#3798); the same restriction lands for the kube-ovn webhook (#3955).
- CloudNativePG is aligned to 1.30.0 (#3951), and an in-place PostgreSQL restore re-bootstraps instead of wedging (#3959).
- etcd members spread across nodes by default via soft pod anti-affinity (#3962).
- A failed tenant install — monitoring, gateway, etcd, seaweedfs, CNI, CSI, Harbor's nested system — is now retried in place instead of being uninstalled and recreated (#3630, #3633, #3621, #3552, #3581).
- On the release side, cutting a release candidate now freezes the line into
release-X.Y, so what a candidate validates is what ships (#3399).
Try it
helm upgrade --install cozystack \
oci://ghcr.io/cozystack/cozystack/cozy-installer --version 1.7.0-alpha.1 \
-n cozy-system --create-namespaceThe Talos node image for this build is attached to the release page as nocloud-amd64.raw.xz.
Full changelog: v1.6.0...v1.7.0-alpha.1
v1.6.2
v1.6.2 (2026-08-19)
A patch release with six fixes covering the backup-strategy controller, kube-ovn's webhook certificate, Velero CRD upgrades, CNPG barman-cloud backups, flux-shard-operator, and the published OpenAPI definitions, plus a release-pipeline reliability fix.
Fixes
-
fix(backupstrategy-controller): repair lookup-gated backup objects: The default backup
StrategyCRs and the VeleroBackupStorageLocationare gated on a Helmlookupperformed while the referenced object is still being created; when that lookup came back empty the objects were skipped permanently, since helm-controller does not re-render a release whose chart and values are unchanged. The gate now resolves the default bucket credentials Secret through the RESTMapper, bounds each check, and tolerates an absent Secret instead of looping, so the default backup objects are created reliably instead of silently vanishing for months (@mattia-eleuteri in #3524, backport #3731). -
fix(kube-ovn): reload kubeovn-webhook serving certificate on cert-manager renewal:
kube-ovn-webhookloaded its TLS serving certificate once at startup and never re-read it; once cert-manager renewed the backing Secret and the old certificate expired, the apiserver's calls to the webhook failed verification and, because theMutatingWebhookConfigurationusesfailurePolicy: Fail, every pod creation in tenant namespaces was rejected — includingvirt-launcherpods, blocking VMI startup. The webhook now serves its certificate through a reloading callback that re-reads the key pair when the mounted files change and widensrenewBeforeto 720h, so cert-manager renewals are honored without a pod restart (@IvanHunters in #3557, backport #3730). -
fix(velero): apply CRD updates on upgrade via CreateReplace: Velero's CRDs stayed frozen at whatever version was first installed, since Helm never touches a chart's
crds/directory on upgrade; when the Velero image moved to a version that added new backup phases, the apiserver rejected phase transitions against the stale CRDs and backups silently stopped while the HelmRelease stayed green. The Velero package now opts intoupgradeCRDs: CreateReplace, so CRDs are kept current on upgrade and backups keep working (@lexfrei in #3727, backport #3728). -
fix(backups): request S3 checksum only when required for barman-cloud (non-AWS S3 / Ceph RGW): CNPG's barman-cloud plugin sidecar defaulted to computing a flexible checksum on every upload, which several S3-compatible backends (Ceph RGW, some MinIO / Cloudflare R2 builds) reject outright, so every backup and WAL-archive upload to those backends failed and
ScheduledBackups never stored anything. Every barman-cloudObjectStoreCozystack creates — Keycloak's system DB, the postgres app's backup and recovery stores, and the platform-managed system-bucket store — now setsAWS_REQUEST_CHECKSUM_CALCULATION=when_required, a safe default accepted by both AWS S3 and the affected backends (@androndo in #3417, backport #3767). -
fix(flux-shard-operator): repair sharded helm-controller crashloop behind an HTTP proxy: The cloned
helm-controller-shard<i>Deployment inheritedHTTP_PROXY/HTTPS_PROXY/NO_PROXYfrom the flux-aio all-in-one wiring even though a standalone shard needs no external egress; behind an unreachable proxy the controller's blocking startup HTTPS call never completed, the manager never served/healthz, and every HelmRelease sharded to that controller was frozen. The sanitisation now also drops the inherited proxy env and adds astartupProbederived from the liveness handler, so sharded HelmReleases keep reconciling in proxied environments instead of crashlooping forever (@IvanHunters in #3546, backport #3818). -
fix(api): declare OpenAPIModelName for core and sdn types: The
coreandsdnAPI groups did not declareOpenAPIModelNamethe way theappsgroup already did, so their published OpenAPI definition names were Go import paths while every$refpointing at them escaped each slash — the two spellings never matched, the reference dangled, andkubectl apply --validatefailed on any resource against acozystack-apibuilt after the underlying Kubernetes 0.35 change. DeclaringOpenAPIModelNameforcoreandsdntoo makes every published definition name the dotted Kubernetes model name, so client-side validation against the published OpenAPI works again (@myasnikovdaniil in #3808, backport #3812).
Development, Testing, and CI/CD
- ci(release): complete the candidate-aware promotion pipeline on release-1.6:
release-1.6was missing the e2e and packages-verification jobs thatPromote RCrequires on its target base, sov1.6.1was promoted with the rc e2e gate bypassed and the next patch release could not even be dispatched. Adds therc-e2ejob, theverify-release-candidatechecks,hack/verify-promoted-packages.sh,hack/validate-changelog.shand regression tests pinning the pipeline's contract, so future patch releases offrelease-1.6run the same e2e and package-verification gates asmainbefore promoting, and the tag-time changelog is validated and ported from the tag rather than regenerated (@myasnikovdaniil in #3893).
Documentation
-
[website] docs: import the operator guides that lived in the cozystack repo: Moves the operator-facing guides that used to live in the
cozystackrepo over to the documentation site, so operators find them alongside the rest of the docs instead of scattered across two repositories (@myasnikovdaniil in cozystack/website#648). -
[website] docs(oidc): document private CA and staging trust: Documents how to configure tenant OIDC to trust a private certificate authority and staging certificates, closing a gap for operators running their own CA or testing with a staging issuer (@myasnikovdaniil in cozystack/website#650).
-
[website] feat(community): add a Community page and link it from the main menu: Adds a Community page linked from the site's main menu, giving visitors a single place to find how to get in touch with and contribute to the Cozystack community (@tym83 in cozystack/website#637).
-
[website] chore(telemetry): publish July 2026 and explain how the figures are derived: Publishes the July 2026 telemetry figures and documents how those figures are derived, giving the community visibility into adoption trends and how the numbers are calculated (@tym83 in cozystack/website#644).
-
[website] feat(blog): new Blockstor banner: Adds a new banner promoting Blockstor to the blog, improving the visibility of the storage control plane's announcement (@tym83 in cozystack/website#646).
Contributors
Thanks to everyone who contributed to this patch release:
Full Changelog: v1.6.1...v1.6.2
v1.5.4
v1.5.4 (2026-08-19)
v1.5.4 is the final release of the 1.5 line. It is a stability release: it backports fixes for a webhook certificate-renewal outage that could block all pod creation, a KubeVirt VMI validation failure, several release-blocking crashloops (velero, cert-manager, SeaweedFS, flux-shard-operator), a silent PostgreSQL restore data-integrity bug, and a raft of smaller reliability and CI fixes accumulated on the branch. It also closes the SeaweedFS 4.31 rename fallout on the 1.5.x line and pins the CAPI kubeadm bootstrap objects that a later upgrade to 1.6 would otherwise prune — both of which need an operator to act, and both of which are covered in the section below.
These notes are measured against v1.5.2, not v1.5.3. v1.5.3 was tagged but its GitHub Release was left a draft and never published, so no user ever received it and every operator upgrading arrives from v1.5.2. Comparing against v1.5.3 would silently drop four commits — two user-facing fixes — that nobody has seen in a release. The range is v1.5.2..v1.5.4, 82 commits across 27 pull requests.
⚠️ Breaking Changes and Required Actions
There are no breaking API or values changes in v1.5.4. There are two things that need an operator, and both of them can cost data or wedge an upgrade if they are skipped. Read this section in full before applying the v1.5.4 Platform Package.
Pre-upgrade checks
Run these against the management cluster before upgrading.
1. SeaweedFS 4.31 rename — classify every instance, and re-run the audit even if you have run it before.
Cozystack v1.5.0 bumped the vendored SeaweedFS chart from 4.0.405 to 4.31.0. Before 4.31 the chart named its workloads after the chart (seaweedfs-master, seaweedfs-filer, seaweedfs-volume), ignoring the Helm release name. 4.31 names them after the release, and the data-plane release is <name>-system, so every StatefulSet wanted to become seaweedfs-system-*. StatefulSet names are immutable, so Helm could not rename in place — it stood up a second, duplicate set beside the running one.
What that duplicate does depends on the cluster. With as many nodes as master replicas the new masters cannot schedule (hard pod anti-affinity against the old ones), so the duplicate sits Pending/CrashLoopBackOff and the original keeps serving. With more nodes than masters the new, empty set comes up — and because both sets carry identical pod labels, the seaweedfs-s3 Service load-balances across them while both filers write to the same seaweedfs-db Postgres metadata store pointing at different volume servers. That is a data-integrity incident, not a cosmetic duplicate: reads of existing objects through the new endpoint miss, new writes land on empty volumes, and two master sets hand out volume IDs from independent sequences into one shared metadata table.
Separately, and landing on the same upgrade, the v1.5.0 database split moved the CNPG Cluster/seaweedfs-db — the filer metadata store, i.e. the index for every object in the tenant's S3 — out of the <name>-system release into its own <name>-db release. Migration 43 shipped comparing the owning release against the literal string seaweedfs-system, so it only ever fired for an instance named seaweedfs; an instance named anything else was skipped and had its Cluster pruned as a removed resource, with CNPG taking the PVC along with it. That prune is not a one-shot: Helm computes deletions by diffing the last deployed revision against the new manifest, so a tenant whose <name>-system last succeeded on a pre-split revision recomputes the same deletion on every upgrade attempt, including attempts that fail for unrelated reasons.
v1.5.4 pins fullnameOverride: seaweedfs in system/seaweedfs, so workloads are named after the chart exactly as they were before 4.31 and upgrading adopts the running set and its volumes in place. Migration 43 is fixed to match the -system suffix, and migration 45 re-runs the hand-over for clusters that already ran the hardcoded version. Two states cannot be adopted that way, and the chart fails the render rather than guess — the enforcing guard is packages/system/seaweedfs/templates/naming-guard.yaml, with a sibling copy in extra/seaweedfs so the refusal is visible on the SeaweedFS application itself.
Step 0 — seaweedfs-db ownership (read-only, do this first). This one destroys data rather than duplicating it, so clear it before anything else.
kubectl get cluster.postgresql.cnpg.io -A \
-o custom-columns='NS:.metadata.namespace,NAME:.metadata.name,OWNER:.metadata.annotations.meta\.helm\.sh/release-name,KEEP:.metadata.annotations.helm\.sh/resource-policy'Read the rows where NAME is seaweedfs-db:
| OWNER | KEEP | Meaning |
|---|---|---|
<name>-db |
keep |
Handed over. Nothing to do. |
<name>-db |
(none) | Installed fresh on ≥ v1.5.0. Safe — <name>-system never rendered the Cluster, so it is not in that release's prune baseline. |
<name>-system |
(none) | At risk. Migration 45 hands it over on the next platform upgrade. Do not reconcile <name>-system before the migration runs. |
| (no row at all) | Already lost. The metadata index is gone: that tenant's S3 returns 500 and its objects are unreachable even though the volume PVCs still hold the bytes. No migration can rebuild it — restore the seaweedfs-db Postgres from a backup, or treat that tenant's object storage as lost. Note that <name>-db may still report Ready while this is true; trust the kubectl get cluster output, not the HelmRelease status. |
Step 1 — classify every SeaweedFS instance (read-only, mutates nothing).
hack/seaweedfs-naming-audit.sh # whole cluster
hack/seaweedfs-naming-audit.sh tenant-foo # or named namespaces| CLASS | State | Action |
|---|---|---|
L |
Only the chart-named generation is present. | None. The upgrade adopts it in place. |
S |
Only the release-named generation is present — the instance was installed fresh on 1.5.x, and its data lives on data1-seaweedfs-system-volume-* PVCs. |
Re-bind those volumes onto the chart-named PVC names before upgrading. Pinning the chart name without that renames the workloads away from the data, and Helm cannot move data between PVCs. Runbook Step 2. |
MIXED |
Both generations are present. | One is an empty duplicate and one holds the data, and nothing durable in the object graph says which — so the chart refuses. Classify the tenant and delete the empty generation so exactly one remains; the render then adopts the survivor with no further action. Runbook Step 1, then 2a or 3. |
Read the exit code, not just the table. The audit fails closed: a kubectl call that fails, or a Helm release payload it cannot decode, prints FATAL and exits non-zero, and the partial table must not be trusted. Only exit 0 means the table is the whole answer — and an empty table with exit 0 is a genuinely clean fleet.
If you have already run this audit, run it again on v1.5.4. The version of the script that shipped in v1.6.0 silenced every kubectl failure with 2>/dev/null, so a timeout or an RBAC denial produced an empty, "all clean" table byte-identical to an honestly clean fleet — a false clean, on the script whose output gates a runbook step that deletes PVCs. v1.5.4 is the first release on the 1.5 line to carry the audit at all, and it carries the fail-closed version (v1.6.1 and later carry it too). A clean result from a v1.6.0 checkout, or from main between 2026-07-20 and 2026-07-28, is not evidence of anything.
One unrelated filer change lands on the same upgrade and is worth knowing about while you are looking at this: the filer's postgres2 connection pool to that same seaweedfs-db metadata store was unconfigured, so every metadata lookup opened a fresh PostgreSQL connection and added seconds of latency to every S3 request. That is fixed here too (see the postgres2 connection pool entry below), and it needs no operator action — but if you have been treating slow S3 as a symptom of the rename, it may well have been this instead.
Recovery for S and MIXED is docs/operations/seaweedfs-431-rename-recovery.md. Do not guess which generation holds the data — the runbook exists because a duplicate that briefly served writes and later crashed is indistinguishable, on every durable signal, from one that never scheduled. A tenant that went 1.4.x straight to 1.6 never renamed and is unaffected by any of this; duplicates exist only on tenants that passed through 1.5.x.
2. The platform migration targetVersion moves from 45 to 46 — which changes what a later upgrade to 1.6 runs.
v1.5.4 is the first 1.5.x release stamped targetVersion: 46; v1.5.0 through v1.5.3 were all stamped 45. run-migrations.sh loops seq CURRENT (TARGET - 1), so a cluster that reaches 46 and later upgrades to v1.6 (targetVersion: 54) runs slots 46 through 53 — and never executes 1.6's own slot 45. This is a skip, not an ordering problem.
Slot 45 does not hold the same thing on both branches. On release-1.5 it is the SeaweedFS seaweedfs-db hand-over repair described above. On main and release-1.6 it is the pin that stamps helm.sh/resource-policy: keep onto the CAPI KubeadmConfigTemplate objects. 1.6 drops KubeadmConfigTemplate from the tenant kubernetes chart entirely — workers move to TalosConfigTemplate — so on that upgrade Helm sees the object in the previous release manifest, absent from the new one, and deletes it while the kubeadm-backed MachineSet is still mid-rollover with its bootstrap.configRef pointing at it. controller-manager then floods with reconcile errors, and where the Talos image fetch is slow or a MachineHealthCheck remediates, workers can han...
v1.6.2-rc.1
[Backport release-1.6] fix(backupstrategy-controller): repair lookup-…
v1.6.1
v1.6.1 (2026-08-05)
A patch release with seven fixes covering PostgreSQL, etcd, managed Kubernetes, Keycloak, cozystack-basics, and SeaweedFS, plus release-pipeline reliability fixes and a talm update adding declarative Talos preset knobs.
Fixes
-
fix(postgres-operator): align CNPG operator and CRDs to 1.28.2 for PVC resize-deadlock fix: A simultaneous
resources+sizechange on a single-instance PostgreSQL cluster could make the CloudNativePG operator delete the sole primary Pod, classify the PVC asresizing, and never recreate the Pod — wedging the cluster with zero instances and leaving the filesystem resize incomplete. Bumping the operator image and CRDs together to 1.28.2 (which carries upstream's fix, cloudnative-pg#9980 / cloudnative-pg#9981) resolves the deadlock (@scooby87 in #3510, backport #3542). -
chore(etcd-operator): bump etcd-operator to v0.5.4: Rolls up four upstream controller bug fixes: the operator no longer exempts the bootstrap seed from crash-loop self-heal, self-heal now also covers memory-backed etcd members,
--initial-cluster-stateis derived from cluster phase instead of the seed, and each EtcdCluster's PodDisruptionBudget switches frommaxUnavailabletominAvailable(existing clusters are reconciled onto the new field automatically on upgrade). Tenant etcd clusters recover more reliably from member crashes and are less likely to have their PDB block a node drain (@androndo in #3529, backport #3538). -
fix(kubernetes): render the talos-reconcile Job for the default md0 group: On a managed Kubernetes cluster left with the default (undeclared)
nodeGroups, the Job that creates each worker'sTalosConfigTemplateand patches the control-plane's cert SANs only iterated the user-supplied node-group map, so it silently skipped the implicitmd0group. Nothing failed at install time, sincemd0defaults to zero replicas, but the first scale-up (for example, autoscaler-driven growth after enabling ingress-nginx) left new Machines permanently blocked with no matchingTalosConfigTemplate. The Job now iterates the same helper that produces theMachineDeployment, somd0gets its reconcile Job like any explicitly declared group (@myasnikovdaniil in #3535, backport #3536). -
fix(keycloak-configure): patch HelmRelease in release namespace on teardown: The
keycloak-configurepre-delete Job cleared the FluxHelmReleasefinalizer in a hardcoded namespace that did not match where the release actually installs (cozy-keycloak), so itsServiceAccountwas forbidden to patch it, the Job retried forever, and theHelmReleasestuck inTerminating— blocking any uninstall or reinstall of Keycloak. The teardown Job now templates both the release name and namespace from the Helm release itself, so teardown completes correctly (@lexfrei in #3372, backport #3478). -
fix(cozystack-basics): gate the hostname VAP policies on the VAP API: The hostname
ValidatingAdmissionPolicytemplates rendered unconditionally, so a first install on a cluster where theValidatingAdmissionPolicyAPI is unavailable dropped the policies permanently — a later cluster upgrade that gains the API would not bring them back. The templates are now gated on.Capabilities.APIVersions.Has, so they render only where the API exists and are picked up automatically once it becomes available (@lexfrei in #3409, backport #3442). -
fix(seaweedfs): make naming audit fail closed on kubectl and payload errors:
hack/seaweedfs-naming-audit.sh, used by operators to classify SeaweedFS instances before the naming-migration cleanup, was fail-open — anykubectlfailure or unreadable Helm release payload produced an empty result table indistinguishable from a genuinely clean fleet. Since the runbook uses this script as the gate before deleting old PVCs, a transient API error could have green-lit destroying live data. Every query now fails loudly on error instead of silently reporting "nothing found," and incomplete evidence now falls back to a conservative "direction cannot be established" verdict rather than a wrong deletion candidate (@myasnikovdaniil in #3436, backport #3474). -
chore(release): don't activate
kubernetes-nodeson the release-1.6 line: Thekubernetes-nodesapp package was not ready to ship on the 1.6 line, so its single include is removed from theiaasplatform bundle — the platform stops activating the package on this line while its code, API types, andkubernetes-nodes-rdsystem package are all kept intact for when it is ready (@myasnikovdaniil in #3437).
Development, Testing, and CI/CD
-
ci(release): carry the finalize fixes onto the 1.6 line:
release-1.6was cut before three release-pipeline fixes landed onmain: droppingpersist-credentialson checkout (which had let a staleGITHUB_TOKENsilently win over the app token, so the stable tag push created no workflow run and the automated changelog/docs backstops never fired, as happened for v1.6.0), publishing the GitHub release with the merged changelog as its body instead of a placeholder, and dropping apaths-ignorefilter that could drop a changelog-only promotion PR. All three are backported so v1.6.1 releases correctly with real release notes (@myasnikovdaniil in #3530). -
fix(ci): overlay images from the PR base branch, and publish per-line artifacts: PR validation always overlaid unbuilt packages from
cozystack-packages:main, so arelease-1.6PR was tested againstmain's controller binaries against its own line's charts — which is exactly what made #3437 above fail install deterministically with a schema-validation error. Each maintainedrelease-X.Ybranch now builds and publishes its own packages artifact, and PR validation overlays from the artifact matching the PR's own base branch (@myasnikovdaniil in #3471, backport #3514). -
fix(release): make promote-retag digest verification media-type-agnostic: The v1.6.0 finalize run aborted partway through promoting rc images to stable tags because its post-copy digest check used
skopeo inspect --format '{{.Digest}}', which prints nothing for OCI artifacts likecozystack-packages, leaving most repositories without a stable tag and skipping the installer publish. The digest is now computed as the sha256 of the raw manifest, which works identically for container images and OCI artifacts, so promotion can no longer abort mid-way on this class of artifact (@myasnikovdaniil in #3435, backport #3473).
Other repositories
talm v0.34.0
-
[talm] feat(charts): add preset value knobs: Exposes
timeServers, control-plane componentextraArgs,registryMirrors, per-hostregistryTLS, multiple Layer2vips,network.preserveExisting, andnetwork.extraLinks(bonds, VLANs, extra addresses and routes) as values on the cozystack, generic, and talm presets, so a node's Talos machine config can be described declaratively instead of via a template fork. Every knob defaults empty and a stock render stays byte-identical; each input Talos would reject fails fast at render time with a hinted error (@lexfrei in cozystack/talm#232). -
[talm] chore(deps): migrate to Helm 4 and drop the cozystack/talos fork: Moves talm's vendored Helm template engine from v3 to v4 and drops the
cozystack/talosfork (carried solely for a--skip-verifyflag, now reimplemented locally), tracking stock upstream Talos v1.13.7. Golden render snapshots confirm the generated machine config is unchanged for users (@lexfrei in cozystack/talm#231).
Documentation
-
[website] feat(blog): add Cozystack 1.6 release and Blockstor announcement: Publishes the v1.6.0 release-announcement blog post — covering Talos Linux tenant workers, tenant-controlled OIDC, the
SecurityGroupAPI, hierarchical quotas, and in-place etcd-operator adoption — alongside a companion post announcing the open-sourcing of Blockstor, the LINSTOR-compatible storage control plane (@tym83 in cozystack/website#641). -
[website] chore(blog): repair front matter, links and bundle names: Follow-up cleanup on the two new blog posts, fixing front matter, internal links, and page bundle names (@tym83 in cozystack/website@4b5d1ad).
-
[website] chore: upgrade Hugo to 0.164.0 and convert HTML content to markdown: Upgrades the site generator to Hugo 0.164.0, requiring Node 22 under its new node permission model, and converts remaining raw-HTML content to markdown along the way, keeping the site buildable on current tooling (@tym83 in cozystack/website#636).
-
[website] chore(blog): make the two taxonomy axes disjoint: Cleans up the blog's tagging so its two taxonomy axes no longer overlap, making blog post categorization and filtering more consistent (@tym83 in cozystack/website#635).
-
[website] docs(talm): describe the preset value knobs for network and registries: Documents the new
talmpreset value knobs (cozystack/talm#232) for time servers, control-plane extra args, registry mirrors and TLS, VIPs, and network links, so operators can find the declarative equivalents to a template fork (@lexfrei in cozystack/website#633). -
[website] feat(hack): generate docs from a pre-tag ref and fail loudly on fetch errors: Hardens the docs-generation tooling to build from a ...
v1.6.1-rc.1
[Backport release-1.6] fix(postgres-operator): align CNPG operator an…