Skip to content

Releases: cozystack/cozystack

v1.6.4-rc.1

v1.6.4-rc.1 Pre-release
Pre-release

Choose a tag to compare

@github-actions github-actions released this 25 Sep 13:05
6f00d30
[Backport release-1.6] feat(kubevirt): expose migration configuration…

v1.6.3

Choose a tag to compare

@cozystack-ci cozystack-ci released this 04 Sep 07:57
c86d366

v1.6.3 (2026-09-03)

A patch release with a security fix restricting access to the kube-ovn admission webhook, four bug fixes covering managed Kubernetes, ClickHouse, PostgreSQL and telemetry, a reworked Marketplace/Admin dashboard, configurable ClickHouse versions, a WorkloadMonitor bucket-metrics endpoint override, a Keycloak KMS proxy update and Vault CA support, plus ansible-cozystack dependency bumps and a batch of new documentation.

Security

  • fix(kubeovn-webhook): restrict the webhook port to the API server: The kube-ovn-webhook /mutate-pods handler returned namespace-derived annotations to any caller that could reach its serving port, and nothing previously stopped an arbitrary in-cluster pod from being that caller. A CiliumNetworkPolicy now denies pod and world traffic to the webhook's port while allowing only the API server (which runs on the host network), closing the exposure tracked as GHSA-g883-q79m-8225 without needing the client-certificate path, which the platform's current admission configuration cannot use (@kvaps in #3955, backport #3975).

Features and Improvements

  • feat(dashboard): rework the Marketplace entry point and the Admin portal: The Console is now the dashboard's entry point instead of the Marketplace, so opening the dashboard shows what a tenant already runs rather than a "choose a product" landing page; the Marketplace opens the grouped application list directly, one click away. Administration (Tenants, Modules, External IPs, per-tenant Info) moves from Console into an always-visible Admin tab, while the two cluster-wide operator areas (Capacity, Backup Classes) keep their own gates. Tenants and Modules are now rendered as a tenant hierarchy tree instead of a flat table, so sub-tenants and inherited modules are visible instead of missing or erroring, and External IPs lists LoadBalancer services across every accessible tenant in one table. Secret values can be copied without being revealed on screen first (@kvaps in #3928, backport #3952).

  • feat(clickhouse): add configurable version parameter: The ClickHouse server and Keeper images were hardcoded to 24.9.2.42, so — unlike postgres, mariadb, mongodb, opensearch, rabbitmq and redis — ClickHouse instances carried no engine version in the Cozystack API. A version values key (v25.8, v25.3, v24.9, default v24.9 for byte-identical existing renders) now selects the server and Keeper image tags, with hack/update-versions.sh refreshing the supported-version map from Docker Hub (@scooby87 in #3476, backport #4054).

  • feat(workloadmonitor): bucket metrics endpoint override: cozystack-controller meters S3 bucket sizes for billing by discovering a VictoriaMetrics endpoint through a namespace label, which assumed the monitoring stack scraping SeaweedFS always runs in the same cluster. A new cozystackController.seaweedfsMetricsEndpoint chart value lets a deployment point bucket-size queries at a remote Prometheus-compatible endpoint instead, with no behavior change when unset. This also fixes a latent billing bug where any metrics-query failure silently zeroed out recorded bucket sizes; the controller now retains the last known values and surfaces the failure via a BucketMetricsUnavailable event and error metrics instead of wiping billing data (@lllamnyp in #3443, backport #3558).

  • feat(keycloak): let the KMS proxy trust a private Vault CA: The KMS-encrypting DB proxy could previously only reach a Vault whose certificate chains to a publicly trusted root, so a Vault behind an internal load balancer with a self-signed certificate was reachable only over plain HTTP. encryption.kms.vault now accepts a CA either inline as caBundle or via an existing caSecretName/caSecretKey, letting the proxy verify a private Vault over HTTPS; existing installations are unaffected when neither key is set (@sircthulhu in #3874, backport #3981).

Fixes

  • fix(postgres): let in-place restore re-bootstrap instead of wedging: An in-place PostgreSQL RestoreJob (restoring an app back into itself) never came back up — the cluster stayed in Setting up primary with crash-looping recovery pods, or the HelmRelease upgrade failed outright. Two independent bugs in the restore patch caused this: the restored cluster's WAL-archive serverName collided with the recovery source's, and useSystemBucket stayed enabled alongside explicit restore S3 coordinates, tripping the chart's own compatibility guard. Both are fixed, so in-place restores now re-bootstrap to a healthy cluster instead of wedging (@androndo in #3959, backport #3988).

  • fix(clickhouse): scheme the backup S3_ENDPOINT on the system-bucket flow: On the default useSystemBucket backup flow, the ClickHouse chart passed a scheme-less host straight into the clickhouse-backup sidecar's S3_ENDPOINT, which altinity/clickhouse-backup's AWS SDK endpoint resolver rejects outright — so every ClickHouse backup and restore on the default flow failed with was not a valid URI. The chart now prepends https:// on that path, matching how the platform already schemes the equivalent endpoint for other backup strategies, so default-flow ClickHouse backups and restores work again (@androndo in #3963, backport #3987).

  • fix(kubernetes): retry the tenant CNI install instead of uninstalling it: The tenant Cilium HelmRelease had no install strategy, so Flux used its default RemediateOnFailure, which uninstalls Cilium — the tenant cluster's only CNI — between retry attempts whenever the initial install ran long (for example while waiting for the first worker node to register), then reinstalls it on the next attempt, repeating without end. The strategy is now RetryOnFailure, which keeps the applied manifests in place and retries a failed install as an upgrade, so the CNI is no longer torn down and the agent DaemonSet rolls out as soon as a worker registers (@lexfrei in #3552, backport #4033).

  • fix(telemetry): make the documented opt-out cover both collectors: Telemetry is reported by two components — cozystack-operator and cozystack-controller — but only the operator had a reachable opt-out switch; the documented cozystackOperator.disableTelemetry setting could never reach the controller, which kept posting application counts every 15 minutes even after an operator followed the published opt-out procedure. The platform chart now threads a telemetry.disabled key into the controller as well, so the documented opt-out actually disables both collectors (@kvaps in #3791, backport #3916).

Dependencies

  • chore(keycloak): bump KMS proxy to 0.2.2: Updates keycloak-kms-proxy to v0.2.2 and pins the image to its published digest, fixing decryption for reads whose result set spans a join (@myasnikovdaniil in #3995, backport #3996).

  • [ansible-cozystack] chore(deps): update k3s to v1.36.3+k3s1 and v1.36.4+k3s1: Renovate dependency bumps of the k3s Kubernetes distribution used by the management-cluster bootstrap role (@app/renovate in cozystack/ansible-cozystack#66, #69).

  • [ansible-cozystack] chore(deps): track cozystack installer v1.6.2: Renovate dependency bump of the cozy-installer Docker image tag to follow the latest Cozystack patch release (@app/renovate in cozystack/ansible-cozystack#68).

Documentation

  • [website] fix(telemetry): document the opt-out for both collectors: Documents both opt-out steps now that the fix above threads telemetry.disabled through the platform, replacing the previously incomplete single-step procedure (@kvaps in cozystack/website#651).

  • [website] docs(home): refresh the screenshot gallery for the reworked console: Retakes the home-page screenshot gallery against the reworked console (Marketplace, Modules, Tenants, Admin), replacing slides that showed a header and sidebar layout that no longer exist (@kvaps in cozystack/website#662).

  • [website] docs(images): refresh the product screenshot and its dark variant: Retakes the standalone product screenshot referenced by the announcement post and outside links, which still showed a console two redesigns old (@kvaps in cozystack/website#663).

  • [website] feat(site): Live Demo — the real console, mocked, in the browser: Adds a /demo page running the real cozystack-ui console with a mocked Kubernetes API, so a visitor can explore the actual marketplace and console without a cluster, signup, or install (@tym83 in cozystack/website#669).

  • [website] feat(demo): self-refreshing console demo + Resources nav grouping: Adds the demo's source as an overlay with a weekly-refresh workflow so the mocked bundle tracks upstream cozystack-ui automatically, and groups secondary navigation links under a Resources menu (@tym83 in cozystack/website#670).

  • [website] fix(demo): embed console under site nav, fix refresh + deep create: Fixes the live demo breaking on a hard refresh of any deep route by serving the SPA at a separate path and embedding it in an iframe under a real Hugo page, and keeps the site's own navigation visible above the demo (@tym83 in cozystack/website#676).

  • [website] fix(demo): create works for every catalog type: Fixes demo ...

Read more

v1.6.3-rc.2

v1.6.3-rc.2 Pre-release
Pre-release

Choose a tag to compare

@github-actions github-actions released this 03 Sep 14:53
9c16916
[Backport release-1.6] feat(clickhouse): add configurable version par…

v1.6.3-rc.1

v1.6.3-rc.1 Pre-release
Pre-release

Choose a tag to compare

@github-actions github-actions released this 03 Sep 12:02
7c310e3
[Backport release-1.6] fix(kubernetes): retry the tenant CNI install …

v1.7.0-alpha.1

v1.7.0-alpha.1 Pre-release
Pre-release

Choose a tag to compare

@github-actions github-actions released this 02 Sep 14:31
67b4e23

Cozystack v1.7.0-alpha.1

Preview build cut from main at 67b4e23. This is an alpha, not a release candidate: it is never promoted to a stable release and it carries no upgrade guarantee. 174 pull requests have landed on main since v1.6.0 — 29 features, 84 fixes, 4 of them breaking. The list below is a selection, not the full set.

Highlights

Worker pools become first-class objects

The kubernetes chart is now control-plane only. Worker pools moved out to their own KubernetesNodes resources, and migration 52 adopts existing pools on upgrade (#3315). Once a pool is its own object it can carry its own settings, and several arrived immediately: a node group can set the CPU request (#3862) and CPU limit (#3859) of its worker VMs, log the guest serial console (#3637), and route worker image pulls through a registry mirror (#3575).

ComputePlane — managed isolation for untrusted code

A new tenant module that provisions a single-tenant, Cozystack-managed cluster for catalog applications that execute user code (#3280), implementing the compute-plane design proposal. It is operator-owned: the Kamaji control plane, the KubeVirt-VM workers and the enablement addons are fixed by the module, so a tenant gets the isolated environment without owning the wiring. Turned on with the computeplane tenant toggle.

PostgreSQL point-in-time recovery

PITR is now a supported and tested capability of the backups.cozystack.io API rather than something an operator assembles by hand (#3383). A recovery target that lies past the end of the archive fails with a precise reason instead of a generic timeout, so a mistyped timestamp is diagnosable in seconds. MongoDB gains a native psmdb backup-strategy driver in the same API (#3562).

The dashboard opens on the Console

Visiting the dashboard now lands on the Console — what the tenant already runs — instead of the Marketplace landing page (#3928). The Marketplace becomes an ordinary destination one click away, and tenant-wide administration moves into the Admin portal. The Tenants page shows the tenant hierarchy, and External IPs are listed across every accessible tenant.

Tenant CA trust anchors across the catalog

The application CA is extracted into a key-free tenant Secret (#3407) and published as a ca.crt-only trust anchor by NATS (#3340) and Qdrant (#3341). MongoDB gets operator-managed TLS with external hostname SANs (#2692), the ClickHouse backup sidecar can trust a private S3 CA (#3385), and the Keycloak KMS proxy a private Vault CA (#3874).

Opt-in east-west encryption

networking.encryption.enabled turns on in-cluster transport encryption for the active datapath (#2984). It maps to KubeOVN's overlay IPsec, whose controller provisions the CA and per-node certificates through the Kubernetes CSR API — no cert-manager wiring, no per-application changes, pod-to-pod traffic encrypted whether or not the workload speaks TLS. Off by default.

No AGPL payload left in the tree

The FluxCD tenant addon (#3379) and the rebuilt Grafana image (#3378) were the last AGPL-3.0 content shipped by this Apache-2.0 repository, and both are gone. The addon is removed outright; Grafana now runs on the digest-pinned upstream image with signed catalog plugins installed at startup, which is possible because the VictoriaLogs datasource — the only reason for the rebuild — is now published in the official catalog.

kube-ovn builds in-tree

packages/system/kubeovn now pulls the upstream chart and builds its image in the monorepo, shaped like packages/system/cilium, retiring the cozystack/kubeovn-chart wrapper repository (#3460). One repository fewer, one release step fewer, and provenance a reader can follow without leaving the tree.

Breaking changes

  • spec.nodeGroups is removed from the Kubernetes CR (#3315). Worker pools, their health checks and the autoscaler provision-time budget now live on separate KubernetesNodes resources. Migration 52 adopts existing pools on upgrade.
  • The addons.fluxcd option of the Kubernetes app is removed (#3379). Tenant clusters that had it enabled keep their running Flux release; the companion migration carries the upgrade path.
  • Multus stages reference CNI plugins into the host /opt/cni/bin (#3195), gated on the new networking.stageCniPlugins value and on for both bundles. Where staging is on the install is unconditional — a plugin is replaced on every pod recreation.
  • ApplicationDefinition release prefixes must be dot-free (#3411), constrained to ^[a-z0-9-]*$. A definition whose prefix carries a dot is rejected at admission and must be corrected before the CRD is applied.

Also in this build

  • ClickHouse exposes a configurable version parameter (#3476).
  • The Redis operator moves from the archived spotahome project to the freshworks-oss fork (#3406).
  • Gateway API listeners advertise HTTP/2 via ALPN (#3398).
  • ingress-nginx moves to controller v1.11.5 with the admission webhook restricted to the API server (#3798); the same restriction lands for the kube-ovn webhook (#3955).
  • CloudNativePG is aligned to 1.30.0 (#3951), and an in-place PostgreSQL restore re-bootstraps instead of wedging (#3959).
  • etcd members spread across nodes by default via soft pod anti-affinity (#3962).
  • A failed tenant install — monitoring, gateway, etcd, seaweedfs, CNI, CSI, Harbor's nested system — is now retried in place instead of being uninstalled and recreated (#3630, #3633, #3621, #3552, #3581).
  • On the release side, cutting a release candidate now freezes the line into release-X.Y, so what a candidate validates is what ships (#3399).

Try it

helm upgrade --install cozystack \
  oci://ghcr.io/cozystack/cozystack/cozy-installer --version 1.7.0-alpha.1 \
  -n cozy-system --create-namespace

The Talos node image for this build is attached to the release page as nocloud-amd64.raw.xz.

Full changelog: v1.6.0...v1.7.0-alpha.1

Download cozystack

v1.6.2

Choose a tag to compare

@cozystack-ci cozystack-ci released this 19 Aug 07:54
bfde94f

v1.6.2 (2026-08-19)

A patch release with six fixes covering the backup-strategy controller, kube-ovn's webhook certificate, Velero CRD upgrades, CNPG barman-cloud backups, flux-shard-operator, and the published OpenAPI definitions, plus a release-pipeline reliability fix.

Fixes

  • fix(backupstrategy-controller): repair lookup-gated backup objects: The default backup Strategy CRs and the Velero BackupStorageLocation are gated on a Helm lookup performed while the referenced object is still being created; when that lookup came back empty the objects were skipped permanently, since helm-controller does not re-render a release whose chart and values are unchanged. The gate now resolves the default bucket credentials Secret through the RESTMapper, bounds each check, and tolerates an absent Secret instead of looping, so the default backup objects are created reliably instead of silently vanishing for months (@mattia-eleuteri in #3524, backport #3731).

  • fix(kube-ovn): reload kubeovn-webhook serving certificate on cert-manager renewal: kube-ovn-webhook loaded its TLS serving certificate once at startup and never re-read it; once cert-manager renewed the backing Secret and the old certificate expired, the apiserver's calls to the webhook failed verification and, because the MutatingWebhookConfiguration uses failurePolicy: Fail, every pod creation in tenant namespaces was rejected — including virt-launcher pods, blocking VMI startup. The webhook now serves its certificate through a reloading callback that re-reads the key pair when the mounted files change and widens renewBefore to 720h, so cert-manager renewals are honored without a pod restart (@IvanHunters in #3557, backport #3730).

  • fix(velero): apply CRD updates on upgrade via CreateReplace: Velero's CRDs stayed frozen at whatever version was first installed, since Helm never touches a chart's crds/ directory on upgrade; when the Velero image moved to a version that added new backup phases, the apiserver rejected phase transitions against the stale CRDs and backups silently stopped while the HelmRelease stayed green. The Velero package now opts into upgradeCRDs: CreateReplace, so CRDs are kept current on upgrade and backups keep working (@lexfrei in #3727, backport #3728).

  • fix(backups): request S3 checksum only when required for barman-cloud (non-AWS S3 / Ceph RGW): CNPG's barman-cloud plugin sidecar defaulted to computing a flexible checksum on every upload, which several S3-compatible backends (Ceph RGW, some MinIO / Cloudflare R2 builds) reject outright, so every backup and WAL-archive upload to those backends failed and ScheduledBackups never stored anything. Every barman-cloud ObjectStore Cozystack creates — Keycloak's system DB, the postgres app's backup and recovery stores, and the platform-managed system-bucket store — now sets AWS_REQUEST_CHECKSUM_CALCULATION=when_required, a safe default accepted by both AWS S3 and the affected backends (@androndo in #3417, backport #3767).

  • fix(flux-shard-operator): repair sharded helm-controller crashloop behind an HTTP proxy: The cloned helm-controller-shard<i> Deployment inherited HTTP_PROXY/HTTPS_PROXY/NO_PROXY from the flux-aio all-in-one wiring even though a standalone shard needs no external egress; behind an unreachable proxy the controller's blocking startup HTTPS call never completed, the manager never served /healthz, and every HelmRelease sharded to that controller was frozen. The sanitisation now also drops the inherited proxy env and adds a startupProbe derived from the liveness handler, so sharded HelmReleases keep reconciling in proxied environments instead of crashlooping forever (@IvanHunters in #3546, backport #3818).

  • fix(api): declare OpenAPIModelName for core and sdn types: The core and sdn API groups did not declare OpenAPIModelName the way the apps group already did, so their published OpenAPI definition names were Go import paths while every $ref pointing at them escaped each slash — the two spellings never matched, the reference dangled, and kubectl apply --validate failed on any resource against a cozystack-api built after the underlying Kubernetes 0.35 change. Declaring OpenAPIModelName for core and sdn too makes every published definition name the dotted Kubernetes model name, so client-side validation against the published OpenAPI works again (@myasnikovdaniil in #3808, backport #3812).

Development, Testing, and CI/CD

  • ci(release): complete the candidate-aware promotion pipeline on release-1.6: release-1.6 was missing the e2e and packages-verification jobs that Promote RC requires on its target base, so v1.6.1 was promoted with the rc e2e gate bypassed and the next patch release could not even be dispatched. Adds the rc-e2e job, the verify-release-candidate checks, hack/verify-promoted-packages.sh, hack/validate-changelog.sh and regression tests pinning the pipeline's contract, so future patch releases off release-1.6 run the same e2e and package-verification gates as main before promoting, and the tag-time changelog is validated and ported from the tag rather than regenerated (@myasnikovdaniil in #3893).

Documentation

  • [website] docs: import the operator guides that lived in the cozystack repo: Moves the operator-facing guides that used to live in the cozystack repo over to the documentation site, so operators find them alongside the rest of the docs instead of scattered across two repositories (@myasnikovdaniil in cozystack/website#648).

  • [website] docs(oidc): document private CA and staging trust: Documents how to configure tenant OIDC to trust a private certificate authority and staging certificates, closing a gap for operators running their own CA or testing with a staging issuer (@myasnikovdaniil in cozystack/website#650).

  • [website] feat(community): add a Community page and link it from the main menu: Adds a Community page linked from the site's main menu, giving visitors a single place to find how to get in touch with and contribute to the Cozystack community (@tym83 in cozystack/website#637).

  • [website] chore(telemetry): publish July 2026 and explain how the figures are derived: Publishes the July 2026 telemetry figures and documents how those figures are derived, giving the community visibility into adoption trends and how the numbers are calculated (@tym83 in cozystack/website#644).

  • [website] feat(blog): new Blockstor banner: Adds a new banner promoting Blockstor to the blog, improving the visibility of the storage control plane's announcement (@tym83 in cozystack/website#646).

Contributors

Thanks to everyone who contributed to this patch release:

Full Changelog: v1.6.1...v1.6.2

Download cozystack

v1.5.4

Choose a tag to compare

@github-actions github-actions released this 19 Aug 09:29
fe2fbd0

v1.5.4 (2026-08-19)

v1.5.4 is the final release of the 1.5 line. It is a stability release: it backports fixes for a webhook certificate-renewal outage that could block all pod creation, a KubeVirt VMI validation failure, several release-blocking crashloops (velero, cert-manager, SeaweedFS, flux-shard-operator), a silent PostgreSQL restore data-integrity bug, and a raft of smaller reliability and CI fixes accumulated on the branch. It also closes the SeaweedFS 4.31 rename fallout on the 1.5.x line and pins the CAPI kubeadm bootstrap objects that a later upgrade to 1.6 would otherwise prune — both of which need an operator to act, and both of which are covered in the section below.

These notes are measured against v1.5.2, not v1.5.3. v1.5.3 was tagged but its GitHub Release was left a draft and never published, so no user ever received it and every operator upgrading arrives from v1.5.2. Comparing against v1.5.3 would silently drop four commits — two user-facing fixes — that nobody has seen in a release. The range is v1.5.2..v1.5.4, 82 commits across 27 pull requests.

⚠️ Breaking Changes and Required Actions

There are no breaking API or values changes in v1.5.4. There are two things that need an operator, and both of them can cost data or wedge an upgrade if they are skipped. Read this section in full before applying the v1.5.4 Platform Package.

Pre-upgrade checks

Run these against the management cluster before upgrading.

1. SeaweedFS 4.31 rename — classify every instance, and re-run the audit even if you have run it before.

Cozystack v1.5.0 bumped the vendored SeaweedFS chart from 4.0.405 to 4.31.0. Before 4.31 the chart named its workloads after the chart (seaweedfs-master, seaweedfs-filer, seaweedfs-volume), ignoring the Helm release name. 4.31 names them after the release, and the data-plane release is <name>-system, so every StatefulSet wanted to become seaweedfs-system-*. StatefulSet names are immutable, so Helm could not rename in place — it stood up a second, duplicate set beside the running one.

What that duplicate does depends on the cluster. With as many nodes as master replicas the new masters cannot schedule (hard pod anti-affinity against the old ones), so the duplicate sits Pending/CrashLoopBackOff and the original keeps serving. With more nodes than masters the new, empty set comes up — and because both sets carry identical pod labels, the seaweedfs-s3 Service load-balances across them while both filers write to the same seaweedfs-db Postgres metadata store pointing at different volume servers. That is a data-integrity incident, not a cosmetic duplicate: reads of existing objects through the new endpoint miss, new writes land on empty volumes, and two master sets hand out volume IDs from independent sequences into one shared metadata table.

Separately, and landing on the same upgrade, the v1.5.0 database split moved the CNPG Cluster/seaweedfs-db — the filer metadata store, i.e. the index for every object in the tenant's S3 — out of the <name>-system release into its own <name>-db release. Migration 43 shipped comparing the owning release against the literal string seaweedfs-system, so it only ever fired for an instance named seaweedfs; an instance named anything else was skipped and had its Cluster pruned as a removed resource, with CNPG taking the PVC along with it. That prune is not a one-shot: Helm computes deletions by diffing the last deployed revision against the new manifest, so a tenant whose <name>-system last succeeded on a pre-split revision recomputes the same deletion on every upgrade attempt, including attempts that fail for unrelated reasons.

v1.5.4 pins fullnameOverride: seaweedfs in system/seaweedfs, so workloads are named after the chart exactly as they were before 4.31 and upgrading adopts the running set and its volumes in place. Migration 43 is fixed to match the -system suffix, and migration 45 re-runs the hand-over for clusters that already ran the hardcoded version. Two states cannot be adopted that way, and the chart fails the render rather than guess — the enforcing guard is packages/system/seaweedfs/templates/naming-guard.yaml, with a sibling copy in extra/seaweedfs so the refusal is visible on the SeaweedFS application itself.

Step 0 — seaweedfs-db ownership (read-only, do this first). This one destroys data rather than duplicating it, so clear it before anything else.

kubectl get cluster.postgresql.cnpg.io -A \
  -o custom-columns='NS:.metadata.namespace,NAME:.metadata.name,OWNER:.metadata.annotations.meta\.helm\.sh/release-name,KEEP:.metadata.annotations.helm\.sh/resource-policy'

Read the rows where NAME is seaweedfs-db:

OWNER KEEP Meaning
<name>-db keep Handed over. Nothing to do.
<name>-db (none) Installed fresh on ≥ v1.5.0. Safe — <name>-system never rendered the Cluster, so it is not in that release's prune baseline.
<name>-system (none) At risk. Migration 45 hands it over on the next platform upgrade. Do not reconcile <name>-system before the migration runs.
(no row at all) Already lost. The metadata index is gone: that tenant's S3 returns 500 and its objects are unreachable even though the volume PVCs still hold the bytes. No migration can rebuild it — restore the seaweedfs-db Postgres from a backup, or treat that tenant's object storage as lost. Note that <name>-db may still report Ready while this is true; trust the kubectl get cluster output, not the HelmRelease status.

Step 1 — classify every SeaweedFS instance (read-only, mutates nothing).

hack/seaweedfs-naming-audit.sh                 # whole cluster
hack/seaweedfs-naming-audit.sh tenant-foo      # or named namespaces
CLASS State Action
L Only the chart-named generation is present. None. The upgrade adopts it in place.
S Only the release-named generation is present — the instance was installed fresh on 1.5.x, and its data lives on data1-seaweedfs-system-volume-* PVCs. Re-bind those volumes onto the chart-named PVC names before upgrading. Pinning the chart name without that renames the workloads away from the data, and Helm cannot move data between PVCs. Runbook Step 2.
MIXED Both generations are present. One is an empty duplicate and one holds the data, and nothing durable in the object graph says which — so the chart refuses. Classify the tenant and delete the empty generation so exactly one remains; the render then adopts the survivor with no further action. Runbook Step 1, then 2a or 3.

Read the exit code, not just the table. The audit fails closed: a kubectl call that fails, or a Helm release payload it cannot decode, prints FATAL and exits non-zero, and the partial table must not be trusted. Only exit 0 means the table is the whole answer — and an empty table with exit 0 is a genuinely clean fleet.

If you have already run this audit, run it again on v1.5.4. The version of the script that shipped in v1.6.0 silenced every kubectl failure with 2>/dev/null, so a timeout or an RBAC denial produced an empty, "all clean" table byte-identical to an honestly clean fleet — a false clean, on the script whose output gates a runbook step that deletes PVCs. v1.5.4 is the first release on the 1.5 line to carry the audit at all, and it carries the fail-closed version (v1.6.1 and later carry it too). A clean result from a v1.6.0 checkout, or from main between 2026-07-20 and 2026-07-28, is not evidence of anything.

One unrelated filer change lands on the same upgrade and is worth knowing about while you are looking at this: the filer's postgres2 connection pool to that same seaweedfs-db metadata store was unconfigured, so every metadata lookup opened a fresh PostgreSQL connection and added seconds of latency to every S3 request. That is fixed here too (see the postgres2 connection pool entry below), and it needs no operator action — but if you have been treating slow S3 as a symptom of the rename, it may well have been this instead.

Recovery for S and MIXED is docs/operations/seaweedfs-431-rename-recovery.md. Do not guess which generation holds the data — the runbook exists because a duplicate that briefly served writes and later crashed is indistinguishable, on every durable signal, from one that never scheduled. A tenant that went 1.4.x straight to 1.6 never renamed and is unaffected by any of this; duplicates exist only on tenants that passed through 1.5.x.

2. The platform migration targetVersion moves from 45 to 46 — which changes what a later upgrade to 1.6 runs.

v1.5.4 is the first 1.5.x release stamped targetVersion: 46; v1.5.0 through v1.5.3 were all stamped 45. run-migrations.sh loops seq CURRENT (TARGET - 1), so a cluster that reaches 46 and later upgrades to v1.6 (targetVersion: 54) runs slots 46 through 53 — and never executes 1.6's own slot 45. This is a skip, not an ordering problem.

Slot 45 does not hold the same thing on both branches. On release-1.5 it is the SeaweedFS seaweedfs-db hand-over repair described above. On main and release-1.6 it is the pin that stamps helm.sh/resource-policy: keep onto the CAPI KubeadmConfigTemplate objects. 1.6 drops KubeadmConfigTemplate from the tenant kubernetes chart entirely — workers move to TalosConfigTemplate — so on that upgrade Helm sees the object in the previous release manifest, absent from the new one, and deletes it while the kubeadm-backed MachineSet is still mid-rollover with its bootstrap.configRef pointing at it. controller-manager then floods with reconcile errors, and where the Talos image fetch is slow or a MachineHealthCheck remediates, workers can han...

Read more

v1.6.2-rc.1

v1.6.2-rc.1 Pre-release
Pre-release

Choose a tag to compare

@github-actions github-actions released this 18 Aug 08:09
eb11104
[Backport release-1.6] fix(backupstrategy-controller): repair lookup-…

v1.6.1

Choose a tag to compare

@cozystack-ci cozystack-ci released this 05 Aug 10:01
8f4c19d

v1.6.1 (2026-08-05)

A patch release with seven fixes covering PostgreSQL, etcd, managed Kubernetes, Keycloak, cozystack-basics, and SeaweedFS, plus release-pipeline reliability fixes and a talm update adding declarative Talos preset knobs.

Fixes

  • fix(postgres-operator): align CNPG operator and CRDs to 1.28.2 for PVC resize-deadlock fix: A simultaneous resources + size change on a single-instance PostgreSQL cluster could make the CloudNativePG operator delete the sole primary Pod, classify the PVC as resizing, and never recreate the Pod — wedging the cluster with zero instances and leaving the filesystem resize incomplete. Bumping the operator image and CRDs together to 1.28.2 (which carries upstream's fix, cloudnative-pg#9980 / cloudnative-pg#9981) resolves the deadlock (@scooby87 in #3510, backport #3542).

  • chore(etcd-operator): bump etcd-operator to v0.5.4: Rolls up four upstream controller bug fixes: the operator no longer exempts the bootstrap seed from crash-loop self-heal, self-heal now also covers memory-backed etcd members, --initial-cluster-state is derived from cluster phase instead of the seed, and each EtcdCluster's PodDisruptionBudget switches from maxUnavailable to minAvailable (existing clusters are reconciled onto the new field automatically on upgrade). Tenant etcd clusters recover more reliably from member crashes and are less likely to have their PDB block a node drain (@androndo in #3529, backport #3538).

  • fix(kubernetes): render the talos-reconcile Job for the default md0 group: On a managed Kubernetes cluster left with the default (undeclared) nodeGroups, the Job that creates each worker's TalosConfigTemplate and patches the control-plane's cert SANs only iterated the user-supplied node-group map, so it silently skipped the implicit md0 group. Nothing failed at install time, since md0 defaults to zero replicas, but the first scale-up (for example, autoscaler-driven growth after enabling ingress-nginx) left new Machines permanently blocked with no matching TalosConfigTemplate. The Job now iterates the same helper that produces the MachineDeployment, so md0 gets its reconcile Job like any explicitly declared group (@myasnikovdaniil in #3535, backport #3536).

  • fix(keycloak-configure): patch HelmRelease in release namespace on teardown: The keycloak-configure pre-delete Job cleared the Flux HelmRelease finalizer in a hardcoded namespace that did not match where the release actually installs (cozy-keycloak), so its ServiceAccount was forbidden to patch it, the Job retried forever, and the HelmRelease stuck in Terminating — blocking any uninstall or reinstall of Keycloak. The teardown Job now templates both the release name and namespace from the Helm release itself, so teardown completes correctly (@lexfrei in #3372, backport #3478).

  • fix(cozystack-basics): gate the hostname VAP policies on the VAP API: The hostname ValidatingAdmissionPolicy templates rendered unconditionally, so a first install on a cluster where the ValidatingAdmissionPolicy API is unavailable dropped the policies permanently — a later cluster upgrade that gains the API would not bring them back. The templates are now gated on .Capabilities.APIVersions.Has, so they render only where the API exists and are picked up automatically once it becomes available (@lexfrei in #3409, backport #3442).

  • fix(seaweedfs): make naming audit fail closed on kubectl and payload errors: hack/seaweedfs-naming-audit.sh, used by operators to classify SeaweedFS instances before the naming-migration cleanup, was fail-open — any kubectl failure or unreadable Helm release payload produced an empty result table indistinguishable from a genuinely clean fleet. Since the runbook uses this script as the gate before deleting old PVCs, a transient API error could have green-lit destroying live data. Every query now fails loudly on error instead of silently reporting "nothing found," and incomplete evidence now falls back to a conservative "direction cannot be established" verdict rather than a wrong deletion candidate (@myasnikovdaniil in #3436, backport #3474).

  • chore(release): don't activate kubernetes-nodes on the release-1.6 line: The kubernetes-nodes app package was not ready to ship on the 1.6 line, so its single include is removed from the iaas platform bundle — the platform stops activating the package on this line while its code, API types, and kubernetes-nodes-rd system package are all kept intact for when it is ready (@myasnikovdaniil in #3437).

Development, Testing, and CI/CD

  • ci(release): carry the finalize fixes onto the 1.6 line: release-1.6 was cut before three release-pipeline fixes landed on main: dropping persist-credentials on checkout (which had let a stale GITHUB_TOKEN silently win over the app token, so the stable tag push created no workflow run and the automated changelog/docs backstops never fired, as happened for v1.6.0), publishing the GitHub release with the merged changelog as its body instead of a placeholder, and dropping a paths-ignore filter that could drop a changelog-only promotion PR. All three are backported so v1.6.1 releases correctly with real release notes (@myasnikovdaniil in #3530).

  • fix(ci): overlay images from the PR base branch, and publish per-line artifacts: PR validation always overlaid unbuilt packages from cozystack-packages:main, so a release-1.6 PR was tested against main's controller binaries against its own line's charts — which is exactly what made #3437 above fail install deterministically with a schema-validation error. Each maintained release-X.Y branch now builds and publishes its own packages artifact, and PR validation overlays from the artifact matching the PR's own base branch (@myasnikovdaniil in #3471, backport #3514).

  • fix(release): make promote-retag digest verification media-type-agnostic: The v1.6.0 finalize run aborted partway through promoting rc images to stable tags because its post-copy digest check used skopeo inspect --format '{{.Digest}}', which prints nothing for OCI artifacts like cozystack-packages, leaving most repositories without a stable tag and skipping the installer publish. The digest is now computed as the sha256 of the raw manifest, which works identically for container images and OCI artifacts, so promotion can no longer abort mid-way on this class of artifact (@myasnikovdaniil in #3435, backport #3473).

Other repositories

talm v0.34.0

  • [talm] feat(charts): add preset value knobs: Exposes timeServers, control-plane component extraArgs, registryMirrors, per-host registryTLS, multiple Layer2 vips, network.preserveExisting, and network.extraLinks (bonds, VLANs, extra addresses and routes) as values on the cozystack, generic, and talm presets, so a node's Talos machine config can be described declaratively instead of via a template fork. Every knob defaults empty and a stock render stays byte-identical; each input Talos would reject fails fast at render time with a hinted error (@lexfrei in cozystack/talm#232).

  • [talm] chore(deps): migrate to Helm 4 and drop the cozystack/talos fork: Moves talm's vendored Helm template engine from v3 to v4 and drops the cozystack/talos fork (carried solely for a --skip-verify flag, now reimplemented locally), tracking stock upstream Talos v1.13.7. Golden render snapshots confirm the generated machine config is unchanged for users (@lexfrei in cozystack/talm#231).

Documentation

  • [website] feat(blog): add Cozystack 1.6 release and Blockstor announcement: Publishes the v1.6.0 release-announcement blog post — covering Talos Linux tenant workers, tenant-controlled OIDC, the SecurityGroup API, hierarchical quotas, and in-place etcd-operator adoption — alongside a companion post announcing the open-sourcing of Blockstor, the LINSTOR-compatible storage control plane (@tym83 in cozystack/website#641).

  • [website] chore(blog): repair front matter, links and bundle names: Follow-up cleanup on the two new blog posts, fixing front matter, internal links, and page bundle names (@tym83 in cozystack/website@4b5d1ad).

  • [website] chore: upgrade Hugo to 0.164.0 and convert HTML content to markdown: Upgrades the site generator to Hugo 0.164.0, requiring Node 22 under its new node permission model, and converts remaining raw-HTML content to markdown along the way, keeping the site buildable on current tooling (@tym83 in cozystack/website#636).

  • [website] chore(blog): make the two taxonomy axes disjoint: Cleans up the blog's tagging so its two taxonomy axes no longer overlap, making blog post categorization and filtering more consistent (@tym83 in cozystack/website#635).

  • [website] docs(talm): describe the preset value knobs for network and registries: Documents the new talm preset value knobs (cozystack/talm#232) for time servers, control-plane extra args, registry mirrors and TLS, VIPs, and network links, so operators can find the declarative equivalents to a template fork (@lexfrei in cozystack/website#633).

  • [website] feat(hack): generate docs from a pre-tag ref and fail loudly on fetch errors: Hardens the docs-generation tooling to build from a ...

Read more

v1.6.1-rc.1

v1.6.1-rc.1 Pre-release
Pre-release

Choose a tag to compare

@github-actions github-actions released this 04 Aug 18:52
76062f5
[Backport release-1.6] fix(postgres-operator): align CNPG operator an…