feat(redis): add opt-in TLS for Redis and Sentinel - #2729
Conversation
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughAdds TLS to Redis: new API types and deepcopy, Helm helpers and templates for tri-state tls.enabled and authClients, cert-manager resources and SANs, RedisFailover/dashboard bindings, chart values/schema/docs, tests, and an operator image bump to v1.4.0. ChangesRedis TLS Feature
Redis Operator Version Update
Sequence Diagram(s)sequenceDiagram
participant Values as Helm values
participant Helm as Chart templates
participant CertManager as cert-manager
participant RedisFailover as RedisFailover CR
participant Kubernetes as Kubernetes API
Values->>Helm: tls.enabled / tls.authClients
Helm->>RedisFailover: conditional spec.tls (secret refs, authClients)
Helm->>CertManager: render Issuer / CA / Certificate when tls.enabled == "true"
CertManager->>Kubernetes: Issuer/Certificate resources created
Kubernetes->>RedisFailover: secrets (cert + key + ca) become available
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~25 minutes Suggested labels
Suggested reviewers
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
Full details: Linked Issues checkExplanation The PR fully implements all coding objectives from issue Full details: Out of Scope Changes checkExplanation All changes directly support the stated objectives: operator Dockerfile/image update, chart TLS templates, cert-manager resources, schema updates, RBAC adjustments, and comprehensive TLS test coverage. No unrelated changes detected. Full details: Title checkExplanation The title clearly identifies the primary change: adding TLS support for Redis and Sentinel. The phrase "opt-in" is slightly imprecise because TLS is also implied when ✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Nitpick comments (1)
packages/apps/redis/tests/tls_test.yaml (1)
279-305: ⚡ Quick winAdd dashboard RBAC assertion for explicit TLS disable override.
Line 279 case coverage skips
dashboard-resourcemap.yamlforexternal=true+tls.enabled=false. Add anotContainscheck there to lock the tri-state behavior end-to-end.✅ Proposed test addition
- it: "redisfailover.yaml has no spec.tls when tls.enabled=false overrides external=true" templates: - templates/redisfailover.yaml set: external: true tls: enabled: false documentSelector: path: kind value: RedisFailover asserts: - notExists: path: spec.tls + - it: "dashboard-resourcemap.yaml omits TLS secrets when tls.enabled=false overrides external=true" + templates: + - templates/dashboard-resourcemap.yaml + set: + external: true + tls: + enabled: false + documentSelector: + path: kind + value: Role + asserts: + - notContains: + path: rules[1].resourceNames + content: "my-redis-tls" + - notContains: + path: rules[1].resourceNames + content: "my-redis-ca-tls"🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/apps/redis/tests/tls_test.yaml` around lines 279 - 305, Add a test case for templates/dashboard-resourcemap.yaml that mirrors the existing cases for certmanager.yaml and redisfailover.yaml when external: true and tls.enabled: false; specifically, under the same "Case 3" inputs (external: true, tls.enabled: false) add a test entry for templates/dashboard-resourcemap.yaml and assert with notContains (or notExists) that any TLS-related keys/entries are absent (e.g., no tls, no tls.enabled, no tls.* fields) so the tri-state override is validated end-to-end for dashboard-resourcemap.yaml.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@packages/apps/redis/values.yaml`:
- Around line 80-83: The values schema for TLS is missing an enum constraint for
tls.authClients, so invalid strings pass chart-value validation; update the
values.yaml typedef for TLS (the TLS struct / tls.authClients entry) to add an
enum with allowed values ["no","optional","yes"] (and a default if needed) so
validation fails early; ensure the field name tls.authClients and the TLS
typedef are the ones updated so downstream RedisFailover.spec.tls.authClients
receives only valid values.
In `@packages/system/redis-operator/images/redis-operator/Dockerfile`:
- Line 3: The Dockerfile sets ARG VERSION=v1.4.0 and later downloads
https://github.com/cozystack/redis-operator/archive/refs/tags/${VERSION}.tar.gz
which 404s because that tag isn't published; either publish the v1.4.0 tag in
the cozystack/redis-operator repo or update the Dockerfile's ARG VERSION to a
valid existing tag (or a branch like main) so the tarball URL resolves; change
the value of VERSION in the Dockerfile (or adjust the download URL) and verify
the tarball URL resolves before merging.
---
Nitpick comments:
In `@packages/apps/redis/tests/tls_test.yaml`:
- Around line 279-305: Add a test case for templates/dashboard-resourcemap.yaml
that mirrors the existing cases for certmanager.yaml and redisfailover.yaml when
external: true and tls.enabled: false; specifically, under the same "Case 3"
inputs (external: true, tls.enabled: false) add a test entry for
templates/dashboard-resourcemap.yaml and assert with notContains (or notExists)
that any TLS-related keys/entries are absent (e.g., no tls, no tls.enabled, no
tls.* fields) so the tri-state override is validated end-to-end for
dashboard-resourcemap.yaml.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: 5630a2fc-02a7-4ff2-9251-f8cd7e21fe1c
📒 Files selected for processing (14)
api/apps/v1alpha1/redis/types.gopackages/apps/redis/README.mdpackages/apps/redis/templates/_tls.tplpackages/apps/redis/templates/certmanager.yamlpackages/apps/redis/templates/dashboard-resourcemap.yamlpackages/apps/redis/templates/redisfailover.yamlpackages/apps/redis/tests/tls_test.yamlpackages/apps/redis/values.schema.jsonpackages/apps/redis/values.yamlpackages/system/redis-operator/charts/redis-operator/crds/databases.spotahome.com_redisfailovers.yamlpackages/system/redis-operator/images/redis-operator/Dockerfilepackages/system/redis-operator/images/redis-operator/patches/labels.diffpackages/system/redis-operator/values.yamlpackages/system/redis-rd/cozyrds/redis.yaml
💤 Files with no reviewable changes (1)
- packages/system/redis-operator/images/redis-operator/patches/labels.diff
Summary of ChangesHello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request introduces native, operator-managed TLS support for Redis within the Cozystack environment. By migrating to a custom redis-operator fork, the change enables secure replication and client traffic encryption. It leverages cert-manager for automated certificate lifecycle management and ensures backward compatibility by defaulting to existing behaviors while providing flexible opt-in mechanisms for TLS. Highlights
New Features🧠 You can now enable Memory (public preview) to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console. Ignored Files
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on Gemini (@gemini-code-assist) comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here. Footnotes
|
There was a problem hiding this comment.
Code Review
This pull request introduces TLS support for the Redis application, incorporating API updates, Cert-manager resource templates, and a helper for tri-state TLS logic. It also updates the redis-operator to version 1.4.0 and switches to a internal fork. Review feedback identifies critical issues in the Dockerfile where non-existent Go and Alpine versions are specified, which will cause build failures. Additionally, it is recommended to enhance the values.schema.json by adding an enumeration and default value for the authClients field to improve validation and the dashboard user experience.
Aleksei Sviridkin (lexfrei)
left a comment
There was a problem hiding this comment.
NOT LGTM — same B2/B3-class issues as mariadb #2680: the chart-managed cert-manager chain exposes both the CA and leaf private keys to tenant use access via dashboard RBAC, and the ECDSA leaf certificate is missing the digital signature usage so strict TLS clients can reject the handshake. Plus a build blocker: the Dockerfile downloads a fork tag that doesn't exist yet.
Business context: this PR shifts Redis TLS to a forked redis-operator (cozystack/redis-operator Phase 2+3 of #2662), with the chart rendering a cert-manager-issued chain and handing the leaf Secret to the operator via spec.tls.certificateSecret.secretName. The fork side is being developed in cozystack/redis-operator#1. The findings below are independent of the fork — they apply to the chart-side TLS chain as rendered.
Blockers
B1: dashboard-resources Role grants tenant use read on both <release>-tls and <release>-ca-tls — leaf AND CA private keys leaked
File: packages/apps/redis/templates/dashboard-resourcemap.yaml:24-25
Issue: when $tlsEnabled is true, the Role grants get/list/watch on <release>-tls and <release>-ca-tls. cert-manager writes both Secrets as kubernetes.io/tls with tls.crt + tls.key. <release>-tls carries the leaf server private key. <release>-ca-tls carries the CA private key.
Impact: at use access level any tenant principal in the namespace can extract:
- the Redis server's leaf private key → impersonate the Redis server, terminate TLS as
rfrm-<release>, MITM other clients, decrypt past sessions for non-PFS ciphers; - the CA's private key → mint arbitrary certificates trusted by anyone using this CA bundle (any sentinel / replication peer / external client) → full forge-and-MITM across the cluster's Redis trust domain.
Same class as the #2680 mariadb blocker. The two-Secret exposure is strictly worse than the single-CA case.
Fix: same as mariadb's: stop exposing <release>-tls and <release>-ca-tls via the chart's dashboard-resources Role. If the dashboard genuinely needs to surface connection metadata, project a ca.crt-only Secret (e.g. via cert-manager's secretTemplate with the private-key field stripped, or a tiny init job) and expose THAT. Tenants needing the leaf or CA for break-glass debugging can still use kubectl out-of-band; routine clients should resolve trust via the projected CA-only Secret.
B2: leaf ECDSA certificate is missing the digital signature usage
File: packages/apps/redis/templates/certmanager.yaml:65-68
Issue: leaf cert usages: [server auth, client auth, key encipherment]. The cert uses ECDSA (privateKey.algorithm: ECDSA, size: 256). TLS_ECDHE_ECDSA — the only meaningful TLS handshake for an ECDSA cert — requires the cert's digital signature (alias signing in cert-manager — see pkg/api/util/usages.go) bit so the server can sign the key-exchange parameters.
Impact: strict TLS clients (Go's crypto/tls, OpenSSL with purpose check) reject the cert when it lacks digitalSignature for ECDHE_ECDSA. The handshake fails with x509: certificate specifies an incompatible key usage or equivalent. Symptom: redis-cli with --tls may work depending on its OpenSSL version, but stricter clients (Go, Java with strict purpose checking, mTLS-aware tooling) fail.
Same finding as the #2680 mariadb leaf cert.
Fix: add - digital signature (or - signing) to the leaf cert's usages block. The CA cert at line 35-37 has the correct signing + cert sign already; the leaf one is the one to fix.
B3: Dockerfile downloads cozystack/redis-operator/v1.4.0 which currently returns 404 — image is unbuildable
File: packages/system/redis-operator/images/redis-operator/Dockerfile:12
Issue: RUN curl -sSL https://github.com/cozystack/redis-operator/archive/refs/tags/v1.4.0.tar.gz | tar -xzvf- --strip=1 — the tag doesn't exist. Verified: HTTP/2 404 against the tarball URL; gh api repos/cozystack/redis-operator/tags returns an empty list; the fork PR cozystack/redis-operator#1 (feat/tls-cert-manager) is still open. Out of scope per the PR body, but the chart side currently references a non-existent artifact, so anyone running make image will fail before compilation starts.
Impact: image build fails everywhere — CI for any cozystack PR that touches redis (including this one's Resolve assets / Build pipelines), local dev, fresh installs. The values.yaml:image.tag = v1.4.0 reference will also fail digest resolution when fetched.
Fix: gate this PR on the fork tag being cut (i.e. merge cozystack/redis-operator#1, tag v1.4.0, publish the image to ghcr) BEFORE merging here. Alternatively, pin to a commit SHA on the fork's feat/tls-cert-manager branch as a placeholder while the bump sequence is resolved — but the current v1.4.0 reference is dead.
Non-blocking follow-ups
-
Same schema array-type issue as #2686/#2692:
"type":["boolean","null"]fortls.enabled(andtls.authClientsif it inherits the pattern) invalues.schema.json/ cozyrdsopenAPISchemafails to unmarshal intoapiextv1.JSONSchemaProps(single-stringTypefield). cozystack-api silently disablesspecSchema-based defaulting for this app; Helm-side defaults still apply, so user-visible breakage is minimal. Worth a separate fix across the whole TLS series —nullable: true(apiextv1 idiom) or droptypefor tri-state properties. -
templates/certmanager.yaml:24and47: CA certsecretNameand CA Issuer'sca.secretNameboth reference<release>-ca-tls. Once B1 is resolved, the same change (CA private key not exposed) needs to be reflected in cozyrdssecrets.includeif it lists-ca-tls. Worth double-checking when fixing B1. -
The PR body explicitly says "digest will be repinned by
make imageonce CI publishes the release artifact". Make sure that step lands in this PR (or a follow-up) BEFORE merging — otherwise the chart references an image that doesn't exist. -
README workflow currently directs users to the dashboard /
<release>-tlsSecret for client connection. After the B1 fix lands, README needs to redirect to whatever CA-only Secret replaces it.
|
Addressed B2 and the series-wide schema cleanup. B3 (image tag) is held by an unmerged upstream change. Blockers:
Non-blocking follow-ups:
Ready for re-review on the chart-side changes; B3 remains the block for merge. |
There was a problem hiding this comment.
NOT LGTM — the chart-side TLS is solid in shape, but it depends on three operator-side artifacts that do not exist yet: an unbuildable image tag, an unmerged CA-only-secret feature, and a missing loopback-IP SAN set. Each is independently merge-blocking.
Business context: adds operator-managed Redis TLS by switching to the cozystack/redis-operator fork and rendering a cert-manager CA+leaf chain handed to the operator, matching the DBaaS-wide TLS convention.
Blockers
B1: the redis-operator image references a tag that does not exist — unbuildable
File: packages/system/redis-operator/images/redis-operator/Dockerfile:3,12, packages/system/redis-operator/values.yaml:4
Issue: the Dockerfile downloads the v1.4.0 source tarball and values.yaml pins tag: v1.4.0, but no such tag exists in the fork.
Evidence: cozystack/redis-operator has no tags, no releases; the v1.4.0 tarball URL returns HTTP 404; the ghcr package was never published. The root Makefile now wires make -C packages/system/redis-operator image into build, so the whole platform build fails on this 404 before compilation.
Impact: make image / make build fails in CI, local dev, and fresh installs; values.yaml tag also fails digest resolution.
Fix: cut and push a release tag on the fork, publish the image to ghcr, then repin values.yaml to a digest — within this PR.
B2: the chart depends on an operator field (spec.tls.caCertSecretName) absent from the operator's released/master code
File: packages/apps/redis/templates/redisfailover.yaml:34, packages/apps/redis/templates/dashboard-resourcemap.yaml:29
Issue: the RBAC private-key leak fix switches the dashboard Role to grant only <release>-ca-cert (a CA-only secret) and sets spec.tls.caCertSecretName: <release>-ca-cert so the operator publishes it. But caCertSecretName exists only in the fork's open, unmerged PR (feat/tls-ca-cert-secret), not in master.
Evidence: caCertSecretName has zero occurrences in the fork's master; it appears only on the open feature branch. The CRD vendored in this PR already carries the field — meaning it was vendored from the open branch, not master. An operator built from any released or master code will never create <release>-ca-cert.
Impact: with the real operator, <release>-ca-cert is never published — the dashboard Role grants on a non-existent secret and the operator silently ignores the unknown CRD field. The headline security fix only works against an operator that has not been merged or released; the vendored CRD diverges from the operator binary it ships with.
Fix: merge the fork's CA-secret PR, fold it into the tagged release this PR pins, and vendor the CRD from that same release point so chart + CRD + binary are consistent.
B3: the chart-issued leaf cert is missing loopback IP SANs (127.0.0.1 / ::1)
File: packages/apps/redis/templates/certmanager.yaml:70-83
Issue: when the chart supplies its own cert via spec.tls.certificateSecret, the operator stops generating the SAN set. The chart leaf lists localhost as a DNS SAN only and sets no ipAddresses, but the operator's own components dial over loopback IP with full CA verification.
Evidence: the operator's generator adds ipAddresses: ["127.0.0.1", "::1"] precisely because redis_exporter is given REDIS_ADDR=rediss://127.0.0.1:26379 with REDIS_EXPORTER_TLS_CA_CERT_FILE (verification active, not skip-verify) and the sentinel monitor targets 127.0.0.1. A localhost DNS SAN does not match an IP literal in Go/openssl cert verification.
Impact: with chart-managed TLS on, the redis_exporter metrics scrape over rediss://127.0.0.1:26379 fails cert verification and the exporter/monitoring path breaks. Probes using -h localhost match the DNS SAN, so the failure is partial and easy to miss in a smoke test.
Fix: add an ipAddresses: block with 127.0.0.1 and ::1 to the leaf Certificate, mirroring the operator's own SAN generation.
Non-blocking follow-ups
- On the
authClients=yespath the chart grants tenantsgeton<release>-tls(the leaf, which carries the private key) so they can reuse it as a client identity (packages/apps/redis/templates/dashboard-resourcemap.yaml:34). The comment acknowledges a dedicated per-tenant client cert would be the stricter long-term design — worth a follow-up issue. - README
tls.authClientsdefault renders{}instead ofno(packages/apps/redis/README.md:36) — a cozyvalues-gen artifact; cosmetic.
3e22d8f to
5d9157c
Compare
Aleksei Sviridkin (lexfrei)
left a comment
There was a problem hiding this comment.
LGTM — re-review on head 3e24931b. All three blockers from the prior review are resolved.
Business context: adds operator-managed Redis TLS by switching to the cozystack/redis-operator fork and rendering a cert-manager CA+leaf chain handed to the operator, matching the DBaaS-wide TLS convention.
Blockers — resolved
- Unbuildable image tag — RESOLVED. The fork now has a
v1.4.0tag and release; the source tarball the Dockerfile downloads returns 200, somake image/make buildno longer fails.values.yamland the DockerfileARG VERSIONboth pinv1.4.0consistently. spec.tls.caCertSecretNameabsent from the operator — RESOLVED. The fork's CA-only-secret PR is merged and folded into thev1.4.0tag (the tag's tree carriesCACertSecretNamein the API types). An operator built fromv1.4.0will publish<release>-ca-cert, so the dashboard RBAC grant now points at a secret the operator actually creates, and the vendored CRD matches the operator binary.- Missing loopback IP SANs — RESOLVED. The leaf certificate now sets
ipAddresses: ["127.0.0.1", "::1"](on the<release>-tlsleaf secret handed to the operator,isCA: false), with a comment explaining the redis_exporterrediss://127.0.0.1CA-verification path. DNS SANs still cover all service forms plus the external host.
Verified: helm-unittest passes (23/23, including the TLS suite); make generate shows no drift; the leaf cert usages include digital signature + server auth + client auth; key rotation is Always with renewBefore: 720h.
Non-blocking follow-ups
- On the
authClients=yespath the chart grants tenantsgeton<release>-tls(the leaf, which carries the private key) so they can reuse it as a client identity (packages/apps/redis/templates/dashboard-resourcemap.yaml:34). The comment acknowledges a dedicated per-tenant client cert would be the stricter long-term design — worth a follow-up issue. - README
tls.authClientsdefault renders{}instead ofno(packages/apps/redis/README.md:36) — a cozyvalues-gen artifact; cosmetic.
## What this PR does Part of #2814. Part of #2811. This is the foundational ("keystone") piece of WS3: a canonical, reusable trust-anchor object that lets a tenant obtain `ca.crt` to verify a per-app TLS endpoint **without** read access to any object that also carries a private key. It unblocks the per-app TLS series (WS4/WS5) by giving every chart one agreed, key-free delivery shape instead of each chart inventing its own. ### The problem (verified against current `main`) Per-app TLS issues a per-release, self-signed CA. The only objects that hold `ca.crt` today also hold private keys: - the cert-manager CA Secret `<release>-ca` carries the CA private key (`tls.key`) — full trust-chain compromise if leaked (`packages/apps/nats/templates/certmanager.yaml`, the `isCA` Certificate); - the cert-manager leaf Secret `<release>-tls` carries the server private key (`tls.key`) plus `ca.crt` (same file, the leaf Certificate); - CNPG surfaces `ca.crt` only inside its user-credentials Secret (`packages/apps/postgres/templates/db.yaml`, the operator-managed TLS block). So any RBAC path that hands a tenant `ca.crt` by granting read on one of those Secrets also hands over a private key. On `main` today the cert-manager apps grant tenants no access to those Secrets at all, so a tenant currently cannot obtain `ca.crt` to verify the server — the gap this object closes safely. ### Design decision: key-free Opaque object delivered through `tenantsecrets`, not trust-manager I evaluated the two candidate delivery mechanisms. **Rejected — trust-manager.** A trust-manager `Bundle` reads its sources only from trust-manager's single configured "trust namespace"; its `namespaceSelector` governs only where the bundle is *distributed to*, not where it is *sourced from*. A per-release self-signed CA lives in the tenant namespace, so trust-manager cannot use it as a source, and projecting one shared CA to all tenants would defeat per-release isolation. I re-verified this against current upstream docs (cert-manager.io trust-manager) and the open per-namespace-trust-bundle request (cert-manager/trust-manager#131, which is target-side only). The prior analysis still holds. **Chosen — a `ca.crt`-only Opaque Secret surfaced through the existing tenant-secret API.** This generalizes the reference pattern from the redis work (#2729), where the operator publishes a CA-only Opaque Secret and RBAC is scoped to it instead of to the CA-private-key Secret. The new `cozy-lib.tls.caCertSecret` helper renders that object once, identically, for every chart: - `type: Opaque`, a single `ca.crt` data key, no `tls.key`/`tls.crt`; - label `internal.cozystack.io/tenantresource: "true"`; - fails closed on empty input and refuses any PEM that contains private key material. ### How the RBAC wiring works (no new roles needed) The label is the entire mechanism, and it routes through RBAC that already exists: - `pkg/registry/core/tenantsecret/rest.go` surfaces, under the virtual resource `core.cozystack.io/tenantsecrets`, exactly the namespace Secrets bearing `internal.cozystack.io/tenantresource=true` (`buildTenantSelector`, lines 199-213; label constants `pkg/apis/core/v1alpha1/tenantresource_types.go:3-4`; full `Data` copied through by `secretToTenant`, lines 84-103). - `packages/system/cozystack-basics/templates/clusterroles.yaml` grants `get/list/watch` on `core.cozystack.io/tenantsecrets` to tenant ServiceAccounts via `cozy:tenant:base` (lines 44-47) and to `use`/`admin`/`super-admin` subjects via `cozy:tenant:use:base` (lines 179-180). None of this touches raw core `secrets`, so attaching the label to a key-free object exposes the trust anchor and nothing else. - `internal/backupcontroller/credentials_projector.go:128-133` relies on the same rule in reverse — it deliberately *omits* the label so its key-bearing projection is not promoted to a `TenantSecret`. This helper is the positive counterpart: a key-free object that is safe to promote. `view` deliberately does not receive the trust anchor through this path: `cozy:tenant:view:base` grants only `core.cozystack.io/options` (lines 120-126), and `tenantsecrets` also includes credential Secrets, so granting it to a read-only role would leak passwords. The trust anchor reaches `use` and above plus tenant ServiceAccounts — the same access level at which connection credentials are already surfaced (the per-release dashboard Role binds at `use`, e.g. `packages/apps/postgres/templates/dashboard-resourcemap.yaml` via `cozy-lib.rbac.subjectsForTenantAndAccessLevel`). A chart that must show `ca.crt` at `view` level in the dashboard can additionally name the key-free Secret in its own per-release Role; that is a per-app detail, not a base-role change. ### Why a library helper and not a controller here Populating `ca.crt` stays the responsibility of whatever owns the PKI — the app operator (as the redis fork does) or a cert-manager chain resolved at the chart level. The helper is intentionally pure and value-driven so it renders deterministically and is the single shape every per-app PR converges on. A shared key-stripping projection controller is a reasonable future generalization but is out of scope for this foundational object. ### Convergence plan for the per-app TLS series (follow-up, not this PR) I did not modify any of the open per-app branches. Each should adopt the canonical object as follows: - redis #2729 — already publishes an operator-side CA-only Opaque Secret and already drops the grant on the CA-private-key Secret; converging means emitting it through the helper / carrying the `tenantresource` label so delivery matches the rest of the catalog. - nats, qdrant (merged, cert-manager) — emit the canonical object via the helper and withhold `<release>-ca` / `<release>-tls` from tenants. - mariadb #2680, opensearch #2682, rabbitmq #2683, mongodb #2692 — same: emit the canonical object, never grant a Secret that contains `tls.key`. ### Tests `helm-unittest` in the cozy-lib test chart (`make -C packages/tests/cozy-lib-tests test`) covers: the rendered object is `Opaque` with `ca.crt` and no `tls.key`/`tls.crt`; the `tenantresource` label is present; caller labels/annotations merge while the security label always wins; the helper fails closed on empty `caCert`; and it refuses a PEM that carries private key material. ### Release note ```release-note NONE ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit ## Release Notes * **New Features** * Added a Helm helper to generate CA-only Kubernetes TLS trust-anchor secrets, including input validation (rejects private key material, requires a properly formatted certificate, and ensures required metadata). * **Tests** * Introduced a dedicated test suite covering successful rendering, metadata and content assertions, tenant label enforcement, label/annotation merging, and detailed failure cases for invalid inputs and incorrect arguments. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
1992683 to
a30b8b9
Compare
…main lookup Two defects in the generated ApplicationDefinition and the certificate template. The regenerated openAPISchema had lost the x-cozystack-options extension on storageClass that values.schema.json still carries, which is what drives the storage class picker in the dashboard form. The schema was generated with cozyvalues-gen 1.5.0 while CI pins 1.6.0, and the older release drops the extension. Regenerated with the pinned version, which restores it. The cluster-domain lookup indexed .Values._cluster directly, so a render where the platform injects no _cluster map failed with "index of untyped nil" before the documented cozy.local fallback could apply. Default the parent map so the fallback is actually reachable, and extend the test to cover an absent map as well as an empty domain — the previous test only exercised a domain set to the empty string, which never reached the failing path. Signed-off-by: Aleksei Sviridkin <[email protected]> Assisted-By: Claude <[email protected]> Signed-off-by: Aleksei Sviridkin <[email protected]>
…host The external SAN dereferenced .Values._namespace.host directly, so a render without that injected value aborted on a nil pointer with no indication of what was missing. TLS now defaults on whenever external is true, which puts that lookup on the default path for every externally reachable Redis. Wrap it in required with an explicit message, matching how the nats chart guards the same lookup in its own certificate template. Failing is the right behaviour rather than dropping the SAN: a certificate issued without the external hostname is accepted by nothing outside the cluster and the resulting handshake error points nowhere near the cause. Signed-off-by: Aleksei Sviridkin <[email protected]> Assisted-By: Claude <[email protected]> Signed-off-by: Aleksei Sviridkin <[email protected]>
…ections TLS replaces plaintext in this operator rather than running beside it — with TLS on, Redis and Sentinel are configured with port 0 and serve only on the TLS port. Since tls.enabled falls back to external, an already deployed instance with external: true switches to TLS-only on its first reconcile after the upgrade and every plaintext client breaks. That was documented as a feature and nowhere as a change of behaviour for existing instances. State it as a breaking change and name tls.enabled: false as the opt-out. Also spell out which hostname the certificate actually covers. The SAN list carries the four service names and the external tenant hostname, and its only IP entries are the loopback addresses used by the in-pod probes and the metrics sidecar, so connecting to the external LoadBalancer by IP fails for clients that verify the hostname. The TLS section marker sat above the version parameter, which put version in the TLS table in the generated README instead of Common parameters. Move the marker below it. Regenerating also reorders version ahead of tls in keysOrder, matching how postgres orders the same two fields. Pin the PackageSource CRD replacement policy the way the mongodb-operator one is pinned. The RedisFailover CRD gained spec.tls with the fork, and Helm never upgrades CRDs shipped in crds/, so losing CreateReplace would leave the live CRD behind and the apiserver would strip spec.tls from every RedisFailover the chart renders — TLS silently off, no error. Assisted-By: Claude <[email protected]> Signed-off-by: Aleksei Sviridkin <[email protected]>
…oller lands The trust anchor was selected only by the internal.cozystack.io/tenant-ca label, which nothing stamps until the CA-extraction controller merges. A label-only entry selects nothing until then, so a TLS-enabled Redis gave the tenant no way to obtain the certificate needed to verify the server — the dashboard Role grants the Secret by name, but it never became a tenant secret and so never appeared on the product surface. Name the operator's CA-only Secret directly alongside the auth secret. It holds ca.crt and no private key, so this widens tenant visibility by exactly the object the tenant needs and nothing else. The label selector stays for the converged path; once the controller publishes its projection the anchor is reachable under both names. Cover the two assertions the feature depended on and did not have: caCertSecretName, which has to agree with both the dashboard Role and the declared CA source, and the loopback IP SANs, added by their own fix commit and until now asserted nowhere. Correct the tri-state helper's comment. It claimed the kindIs "invalid" guard covers an explicitly null tls.enabled; the schema types that field as boolean and rejects null before any template runs. The guard covers the reachable shape, a tls map with no enabled key, now tested. Signed-off-by: Aleksei Sviridkin <[email protected]> Assisted-By: Claude <[email protected]>
Redis and Sentinel read tls-cert-file and tls-key-file once at startup, so a leaf renewed by cert-manager 30 days before its one-year expiry is served only after the pods restart. The operator does that restart: on every reconcile it reads the TLS Secret and stamps a hash of tls.crt and ca.crt into the pod template of the Redis StatefulSet and the Sentinel Deployment, so a renewal rolls Redis through the operator's own roller (the StatefulSet is OnDelete; the roller replaces one pod per reconcile, replicas before the master) and Sentinel through its Deployment. The TLS section presented the setup as turnkey and said nothing about renewal. Say what happens, and name the fallback: when the roll cannot run, because the operator is down or its reconcile of this RedisFailover keeps failing, the deadline is the old certificate's expiry, since past it the operator's own TLS connections to Sentinel and Redis fail along with the clients. The CA has a clock of its own: five years, reissued on the same key before the end, so a ca.crt copied once verifies new leaves only until the original CA certificate expires. Say that too, and point at re-reading the projection instead of baking the copy in. A stakater reloader annotation is not an alternative here and is not used: the operator regenerates the pod template on every reconcile, so nothing written into it from outside survives to the next one. Assisted-By: Claude <[email protected]> Signed-off-by: Aleksei Sviridkin <[email protected]>
The certificate covers <release>.<tenant-host> and the TLS section told users to connect there, but the external Service is a plain LoadBalancer with no external-dns annotation — unlike the opensearch external Service, which annotates its hostname explicitly. Nothing points that name at the LoadBalancer address, so the one documented external path does not work as written, and the IP fallback fails hostname verification because the certificate's only IP SANs are loopback. Describe what is actually true and name the manual step needed to reach the endpoint from outside. Publishing the name from the chart is the better fix, but that is a platform-integration decision about which hostname redis should claim, not a documentation change. Also disambiguate the CA source comment. Source and destination are different Secrets — the declared source is read, the projection is written — and the previous wording put both names close enough together to read as a contradiction. The hostname-verification note names which clients it applies to: redis-cli, the one the README shows, verifies the chain and not the name, so it connects to a bare IP and hides the mismatch. Assisted-By: Claude <[email protected]> Signed-off-by: Aleksei Sviridkin <[email protected]>
Requiring _namespace.host turned a nil dereference into a hard render failure for a shape that legitimately has no host. packages/apps/tenant computes an empty host whenever neither the tenant nor its parent sets one, and Helm's required rejects the empty string as readily as nil, so an external Redis running in such a tenant would have stopped templating altogether on upgrade — no object left to inspect and no error pointing at the cause. Emit the external SAN when a host exists and skip it when none does. A tenant without a host has no external name to certify, so there is nothing being silently dropped, and the in-cluster names still cover every client that verifies inside the cluster. Reading through a defaulted dict keeps an absent _namespace from dereferencing nil. Replace the test that pinned the failure with two that pin rendering, for an empty host and for an absent namespace map. Add the coupling the CA-only secret name needs. Its name has to match across the operator spec that publishes it and the tenant grant that exposes it; renaming one alone leaves the tenant without a trust anchor and produces no error anywhere. Signed-off-by: Aleksei Sviridkin <[email protected]> Assisted-By: Claude <[email protected]> Signed-off-by: Aleksei Sviridkin <[email protected]>
TLS was enabled automatically whenever external was true. In this operator TLS is not additive — the rendered config sets port 0 and moves the listener to tls-port — so that inference turned every already-running externally-reachable Redis into a TLS-only server on its first reconcile after the upgrade and cut every plaintext client with it. A platform upgrade must not sever working connections. Resolve tls.enabled to false when unset, whatever external says. Enabling TLS becomes an explicit act at creation time, and existing releases keep the behaviour they have today. Creation time, not any time: a running instance cannot change mode. A replica restarted onto the TLS listener cannot replicate from a master still on plaintext, and Sentinels already on TLS would fail over to a stale replica, so the operator's RedisFailover schema rejects a change of tls.enabled and the HelmRelease upgrade fails while the instance keeps serving what it served before. The README says so and points at creating a new instance and migrating. Pin the decision directly: external: true on its own now has to leave the cert-manager chain unrendered, the operator without spec.tls, and the tenant without TLS secrets. Every case that previously reached TLS through external says enabled: true instead, and the null and empty-map forms resolve to off alongside the unset one. With nothing inferred there is no upgrade-time behaviour change left, so the breaking-change marker and footer are gone from this branch. Say how to get out of it as well. The cozystack Redis API accepts the change, so the rejection lands on the HelmRelease, and every later edit to that instance queues behind the failure until tls.enabled goes back to what it was. Assisted-By: Claude <[email protected]> Signed-off-by: Aleksei Sviridkin <[email protected]>
a8140f4 to
4802fd8
Compare
When external access is on the chart creates <release>-external-lb itself, but the leaf certificate covered only the four Services the operator creates. That name resolves in-cluster like any other Service, so a client dialing it under TLS failed hostname verification against a Service this chart owns. Add its four DNS forms next to the operator's, gated on the same flag that creates it. Also record at the image pin how the digest moves: it is refreshed by release preparation, not by this change, so an edit to the Dockerfile under images/ ships only once the next repin lands. The README list of in-cluster names the certificate covers names the external LoadBalancer Service as well. Assisted-By: Claude <[email protected]> Signed-off-by: Aleksei Sviridkin <[email protected]>
The comment read as a justification and it was wrong. It described the backstop as guarding against any Secret in the namespace acquiring the tenant-ca label. The lineage webhook is not namespace-wide — it returns early for any object whose ownership does not resolve to this release's app CR — and the two Secrets listed do not even reach it today, because the platform runs cert-manager with enableCertificateOwnerRef false and leaves them without an ownerReference. The entries are defence in depth against a future where that changes, not a gate that fires now. State the scope rather than implying a broader guarantee. Assisted-By: Claude <[email protected]> Signed-off-by: Aleksei Sviridkin <[email protected]>
cozyvalues-gen quotes a default only when the declared type is the literal
`string` or `quantity`; a named alias, including one declared `@enum
{string}`, falls through to a YAML re-parse of the bare value where the 1.1
boolean words resolve. A chart spelling out a `"no"` default under such a
type emits `"default": false` into a string-typed, enum-constrained property
and `+kubebuilder:default:=false` onto the matching Go field. Templates read
values.yaml rather than the schema, so the chart suites stay green while the
shipped schema and CRD carry a default the field rejects.
Vendored schemas under packages/*/*/charts/ are left out of the scan:
cozyvalues-gen never writes them, they change only through `make update`,
and a violation there would have no fix inside this tree.
Assisted-By: Claude <[email protected]>
Signed-off-by: Aleksei Sviridkin <[email protected]>
The generated README renders the tls.authClients default as `{}` because the
block carries no keys. Materialising the default to fix that column corrupts
the schema and the CRD, so note the trap at the line someone would edit.
Assisted-By: Claude <[email protected]>
Signed-off-by: Aleksei Sviridkin <[email protected]>
The TLS section described the per-release cert-manager chain without saying that cert-manager has to be present to issue it. It is present on every system variant: the platform installs it in the variant-independent body of the system bundle, so isp-hosted carries it as well. What isp-hosted lacks is reloader, and the renewal section already covers that gap. The cert-manager sentence separates the two ways a cluster can lack it: without the controller but with its CRDs the resources render and nothing issues them, and without the CRDs the release fails on an unknown kind. Assisted-By: Claude <[email protected]> Signed-off-by: Aleksei Sviridkin <[email protected]>
The tenant needs the release's CA certificate to verify the server, and the platform's contract for handing one over is the key-free <release>.tenant-ca projection: the CA-extraction controller lifts a single named key out of a source Secret, refuses to publish anything that parses as PEM private key material, and stamps internal.cozystack.io/tenant-ca on what it writes. postgres, mongodb, nats and qdrant all reach the tenant this way; redis is the fifth. Source the chart's own cert-manager CA Secret, <release>-ca-tls, rather than the CA-only Secret the operator publishes at spec.tls.caCertSecretName. Both carry the same certificate, but <release>-ca-tls exists as soon as cert-manager issues the CA, so the tenant read path does not also depend on the operator having reconciled the RedisFailover and written its copy. That the source holds the CA private key beside ca.crt is the intended shape rather than a compromise — projecting one key out of a key-bearing Secret is what the controller is for, and it refuses private key material outright. Gated on tls.enabled, because the source Secret only exists when TLS is on. The test pins that the gate is tls.enabled and not external access: an externally exposed plaintext release must render no sentinel, or the controller waits forever on a Secret nothing creates. Assisted-By: Claude <[email protected]> Signed-off-by: Aleksei Sviridkin <[email protected]>
The tenant read path was two grants on Secret names: the operator's CA-only <release>-ca-cert in the dashboard Role, and the same name in the ApplicationDefinition's include list. Replace both with the projection the previous commit declares. A name grant conveys whatever occupies the name. The operator holds create and update on Secrets cluster-wide, so what sits at <release>-ca-cert is the operator's choice rather than the platform's, and RBAC has no key-level filter to catch it if that ever stops being just ca.crt. <release>.tenant-ca is written only by the CA-extraction controller, which copies one named key and refuses PEM private key material outright. That is the difference between a name that happens to hold a certificate today and provenance the platform vouches for, and it is the reason postgres, mongodb, nats and qdrant all reach the tenant this way — mongodb's operator publishes a CA Secret of its own too, and that one is excluded rather than granted. Also drop spec.caCert from the ApplicationDefinition. The ApplicationDefinition CRD does not define that field and Flux applies these server-side, so the apiserver rejects the whole object with "field not declared in schema" and the redis-rd release never goes ready. It was carried deliberately as a merge-order gate for a controller that reads it; that controller landed as the TenantProjection reconciler, which reads a sentinel in the app chart instead, so the gate has nothing left to wait for. hack/check-redis-rd-secrets.bats pins the whole rendered Secret surface rather than probing for known-bad names, alongside the existing guards for nats, qdrant and mongodb: an include with neither resourceNames nor matchLabels restricts nothing, and a second matchLabels entry selecting a label cert-manager also stamps on the key-bearing Secrets would grant the CA private key while every name in the file stayed correct. The connection example passes the generated password as well. Password generation is on by default, so the command as printed connected and then answered NOAUTH on the first real command. Assisted-By: Claude <[email protected]> Signed-off-by: Aleksei Sviridkin <[email protected]>
freshworks-oss/redis-operator v3.3.5 has no TLS support. The operator work that packages/apps/redis renders against was ported upstream as freshworks-oss/redis-operator#85 and is not in a release yet, so carry it here over the v3.3.5 tarball until one lands. Three pieces, and TLS is silently off unless all three are present: - images/redis-operator/patches/tls.diff is the source half. The Dockerfile already applies patches/*.diff, and the shell expands that to labels.diff then tls.diff; they touch disjoint files, so the order does not matter, but it is alphabetically stable either way. - patches/crd-tls.diff teaches the vendored CRD about spec.tls. Without it the apiserver prunes spec.tls from every RedisFailover the redis chart renders and the patched binary waits for a field it never sees. - patches/rbac-tls.diff grants Create, Get and Update on Secrets, so the operator can publish the CA-only Secret named by spec.tls.caCertSecretName. Those are the only Secret calls in the patched source: upstream v3.3.5 only ever reads one, and the patch adds Create and Update on a Secret it addresses by name, so nothing lists, watches, patches or deletes. tls.diff is cut from that PR's commit 2ad2ed8d1 rebased onto v3.3.5, and its header pins that SHA so the next refresh can be diffed against it. Beyond the TLS listener it carries the renewal roll: on every reconcile the operator stamps a hash of the TLS Secret's tls.crt and ca.crt into the pod template of the Redis StatefulSet and the Sentinel Deployment, so a renewed certificate rolls Redis through the operator's own roller and Sentinel through its Deployment. The effective spec.tls.authClients value is folded into the same hash, so changing it rolls the pods the same way instead of landing at the next unrelated restart, and the operator presents its client certificate whenever the Secret carries one, independent of that field, so neither direction of the change locks it out of the instance it reconciles. It also mounts the TLS volume 0440 when the pod declares an fsGroup, rejects a caCertSecretName that names the TLS Secret itself, and waits for the TLS material before it writes the ConfigMaps and the workloads, so a first TLS install starts its pods once. tls.enabled is fixed at creation. Redis replication and Sentinel cannot straddle plaintext and TLS: a replica restarted onto the other protocol cannot sync from the master, and Sentinels restarted onto it fail over to such a replica while the master still takes writes. The rule that refuses the change is a CEL transition rule on the RedisFailover spec; it lives in the CRD, so it ships here through patches/crd-tls.diff together with the schema block it references. The cut carries source only, and the Dockerfile strips the tarball's manifests/, charts/, example/ and .github/ before applying the patches, so the image builds from source alone and the two upstream tests that read the CRD YAML skip instead of failing against v3.3.5's stale copy; the CRD half is patches/crd-tls.diff. cert-manager Certificates get no grant. The patched operator manages a Certificate only under spec.tls.certManager, and packages/apps/redis pins the other mode: it always supplies spec.tls.certificateSecret, so that code path is unreachable as the platform ships it. A comment at the rule site records the decision and names the verbs the rule would need, so a re-vendor review reads the absence as deliberate. The chart patches go through the mechanism the packages that customise a vendored chart already use — reloader, grafana-operator, ingress-nginx, seaweedfs and kamaji all re-apply a patches/*.diff at the end of update: — because `rm -rf charts` discards a hand-edit and the next `make update` would take TLS back out without failing. tls.diff also carries one change that is not TLS and takes effect with TLS off: MakeSlaveOfWithPort connects to the replica on its own port rather than on the master's. It changes a dial, not a pod template, so nothing rolls when this image ships; the probes dial localhost only under TLS, and the plaintext pod templates stay byte-identical to v3.3.5. The patch header names it, so a reader of patches/ does not take the file for TLS-only. The grant is narrower than the ported operator's own chart asks for: that one also grants patch on Secrets, and create, delete, get, list, patch, update and watch on Certificates. None of that is reachable from the source as shipped here. Assisted-By: Claude <[email protected]> Signed-off-by: Aleksei Sviridkin <[email protected]>
…alue The no / optional / yes constraint on tls.authClients lives in values.schema.json, and nothing the chart renders shows whether it is enforced: a bogus value either fails validation up front or reaches the template as a string. Assert the failure, so the enum cannot be dropped from the schema without a red test. Assisted-By: Claude <[email protected]> Signed-off-by: Aleksei Sviridkin <[email protected]>
The operator side of the carried patch computes the leaf's SAN list in redisCertificateSANs. The comment in certmanager.yaml still named it redisCertificateDNSNames, a name the patched source does not contain. Assisted-By: Claude <[email protected]> Signed-off-by: Aleksei Sviridkin <[email protected]>
Everything the TLS unit tests can only render now runs on a live cluster. A second fixture with tls.enabled asserts that cert-manager issues the chain, that the patched operator brings two Redis replicas and the Sentinel Deployment to Ready through its --tls probes, that the operator publishes its CA-only Secret under the widened ClusterRole, and that the CA-extraction controller hands the tenant exactly ca.crt through core.cozystack.io/tenantsecrets while the key-bearing Secrets and the operator's copy stay off that path. The last check completes a handshake through the rfr- service name with the same CA the tenant receives. A third case applies tls.enabled to a running plaintext release and asserts the refusal end to end: the HelmRelease upgrade fails with the schema's own message, the StatefulSet keeps its revision and both replicas Ready, plaintext still answers and no TLS listener appears. The TLS case also asserts the topology the operator built rather than stopping at readyReplicas: exactly one master, a replica whose link is up, and sentinels that moved off the bootstrap address. Ready alone witnesses the bootstrap, since a pod still replicating from the initial 127.0.0.1 never gets there, but the readiness script exits 0 for anything reporting role:master, so it stops witnessing once the bootstrap is over. Both PING calls note where the pinger credentials come from: the ACL user the operator writes into every redis.conf it generates, and the one its own liveness probe uses. Nothing in this tree defines it, so a reader grepping for it here finds nothing and reads the calls as broken. Every redis-cli call runs under the container's own timeout, and both script operations carry an explicit one. The inner cap is the part that matters. The flip case dials TLS at a plaintext listener that will never send a ServerHello, and redis-cli has no handshake deadline of its own. Chainsaw does kill the shell when its exec budget runs out, but it captures the script's stdout and stderr through pipes, and Go's exec then waits for every holder of the write end to close it; the orphaned kubectl is one, so the suite would go silent until the job ceiling instead of failing. Expiry reads as the refusal for the negative check and as a named failure for the PINGs. Both engine images provide /usr/bin/timeout: busybox on the sentinel, coreutils on the data pods. The negative half of the tenant-secret pair requires the error to say NotFound. A 403, a timeout or an unserved CRD would otherwise be read as proof that the Secret is not served, which is the direction that fails silently. The dashboard Role is asserted on the deployed object, not only in the chart unit tests: it must reach the auth Secret and nothing else, which is the name-granting half of the pair the label-driven projection completes. Every kubectl in both scripts runs under a wall-clock bound, because the per-command cap inside a container cannot fire when the exec stream itself never establishes, and a kubectl left running holds the script's stdout open. The plaintext StatefulSet in the flip case waits 5m rather than 2m: it measured 98s of that budget on a real run, being the third failover the group brings up in one namespace. Assisted-By: Claude <[email protected]> Signed-off-by: Aleksei Sviridkin <[email protected]>
4802fd8 to
25a0618
Compare
IvanHunters
left a comment
There was a problem hiding this comment.
Verdict
NOT LGTM
Round two, against 25a06181 (the branch was rebased since my last round, so the commits are rewritten; I compared trees, not shas). Five of my eight findings are closed and I checked each closure by reverting it and watching a suite go red rather than by reading the change. Three are still open, all of them at two rounds now. One new defect turned up, and it is the same class as the one that closed.
I was wrong about one thing last round and want to say so plainly: my suggested redis-cli -t 5 would not have bounded the stall. -t caps the TCP connect, which succeeds against a plaintext port, so the wait is the TLS handshake after it. The inner timeout 15 in the fix is the right instrument.
Closed since my last round
- The suite stall.
hack/e2e-chainsaw/redis/chainsaw-test.yaml:551now carriestimeout: 2mon the op, everyredis-cliruns undertimeout 15and everykubectlundertimeout 30./usr/bin/timeoutis present in both engine images, which I checked by running them rather than trusting the comment that says so. CI corroborates: on the run atd5efbd228the flip step took 15.6s (CMD RUN09:23:32.455,SCRIPT DONE09:23:48.058) against 2h58m56s of silence at the same step before, and all three redis cases completed in 3m50s with the suite carrying on into qdrant and postgres. tls.authClientsno longer changes nothing. The directive is folded length-prefixed intotlsSecretContentHash, andchecker.godeletes every pod whose revision hash differs, so the roller delivers it. The reverse direction is fixed at the right layer too:check.go:44now presents the operator's client certificate whenever the Secret carries a pair, independent of the spec, so the flip back cannot lock the operator out of its own instance. Replacing the hash lines with_ = tlsAuthClients(rf)turnsTestTLSSecretHashTracksAuthClientsred on all three subtests; restoring theauthClients == yesgate turnsTestTLSConfigForPresentsClientCertificatered on three more.- The Secret-name collision, and by a better construction than I proposed. The separator is now a dot, so the suffix set
{.ca-tls, .tls, .ca-cert, .tenant-ca}is free of mutual suffixes andR1 + ".tls" == R2 + ".ca-tls"cannot hold. That does not depend on app-name validation, and it holds even for a release name containing a dot. Rendering the adversarial pair confirms it:redis-foo-ca-tlsis no longer any release's Secret name. Reverting the separator incertmanager.yamlalone fails 3 of the 48 unittests. - The
tenantsecretnegative assertion. The loop now requiresNotFoundin the message. Driven against five stubbed outcomes it separates them correctly, and reverting the guard reproduces the old fail-open exactly: forbidden, unserved and timeout all go back to exit 0. - The
authClientsdescription. The constraint reachedvalues.schema.json, the README table and the Go type with identical text, and the README goes further than I asked by stating the CA key is deliberately out of reach. I verified the underlying claim instead of accepting it: tenant ClusterRoles grantcore.cozystack.io/tenantsecretsand nothing oncert-manager.ioor rawsecrets, so neither the CA Secret nor the in-namespace Issuer is reachable.
Still open from my earlier rounds
[MAJOR] hack/e2e-chainsaw/redis/chainsaw-test.yaml:86, still nothing exercises an existing cluster (round 2)
All four Chainsaw cases still create their instances from scratch under the new operator and the new CRD:
$ grep -rn '^ name: ' hack/e2e-chainsaw/{redis,valkey}/chainsaw-test.yaml
redis/chainsaw-test.yaml:5: name: redis
redis/chainsaw-test.yaml:86: name: redis-tls
redis/chainsaw-test.yaml:463: name: redis-tls-flip-rejected
valkey/chainsaw-test.yaml:5: name: valkey
$ grep -rn 'file: redis\|file: valkey' hack/e2e-chainsaw/{redis,valkey}/chainsaw-test.yaml
redis:20 redis.yaml redis:115 redis-tls.yaml redis:486 redis-flip.yaml
redis:522 redis-flip-tls.yaml valkey:20 valkey.yaml
Every fixture is a fresh apps.cozystack.io/v1alpha1 object. redis-tls-flip-rejected does edit a running plaintext instance, but it created that instance itself under this branch, so the transition an existing customer actually takes is still untested.
Two things did materially de-risk it, and both arrived as hermetic pins rather than as arguments. The spec-level CEL rule was exercised against the real apiextensions validator with the shipped cozystack CRD staged into the fixture paths, and an object with no tls block stays updatable because the rule evaluates false == false on both sides. And the rebuilt binary renders a byte-identical plaintext StatefulSet, Sentinel Deployment and both ConfigMaps compared against a pristine v3.3.5 build, for Engine: Valkey as well; TestPlaintextProbesKeepUpstreamCommands guards that and is not inert, since making redisCLITLSFlags return --tls for a plaintext failover turns all four of its subtests red.
What is still unpinned is what needs an apiserver: SSA field ownership on the already-running rfr-* StatefulSet and rfs-* Deployment, and the CreateReplace of the CRD against stored objects. A Chainsaw case that pins the operator image to the pre-patch digest, creates the instance, then swaps to the rebuilt digest would close this permanently.
[MINOR] packages/system/redis-operator/values.yaml:4, the committed digest is still the pre-patch build (round 2)
The file is untouched by this PR, and the digest it pins was stamped by abb38ab72, whose tree carries only labels.diff under images/redis-operator/patches/. So the committed tree renders spec.tls against a binary built from a patch set that does not contain tls.diff.
The signal did improve, and it came from main through the rebase rather than from this branch: redis-operator is now in the root Makefile build: list and hack/build-matrix.sh emits it, so PR CI builds the package, make image stamps the fresh digest, and the per-package job uploads it for reassembly. docs/agents/image-refs.md is edited here to drop redis-operator from the live-violations list, consistent with that. What stays reachable is any path that builds the platform bundle without rebuilding this package first: make -C packages/core/platform image on its own, or the dev-loop package-deploy flow.
The blast radius grew since my last round, through a consumer this PR did not add. Valkey landed on main and creates RedisFailover on the same operator, CRD and widened ClusterRole. The pinned digest therefore also lacks valkey-cli-auth.diff, which switches valkeyEngine.CLIAuthEnvName() from VALKEYCLI_AUTH to REDISCLI_AUTH, and that value is exported by the generated shutdown.sh and ready.sh, so it drives the readiness probe. Checked against the pinned engine images rather than reasoned about:
$ docker run --rm --entrypoint sh valkey/valkey:8.1.8 -c '...'
-- REDISCLI_AUTH: PONG
-- VALKEYCLI_AUTH: NOAUTH Authentication required.
$ docker run --rm --entrypoint sh valkey/valkey:7.2.13 -c '...'
v7 REDISCLI_AUTH: PONG
v7 VALKEYCLI_AUTH: NOAUTH Authentication required.
On the committed digest with the default authEnabled: true, a Valkey readiness script gets NOAUTH and the pods never go Ready. Through that path the stale pin fails loudly instead of silently, and the Valkey Chainsaw case asserting readyReplicas: 2 would catch it, but only because CI rebuilds the image. Pinning the rebuilt digest here removes the whole question.
[MINOR] packages/system/redis-operator/patches/rbac-tls.diff:44, still no ownership check on the Secret write (round 2)
The only change since last round is a no-op skip when the stored content already matches. CreateOrUpdateSecret still issues a full Update on whatever Secret occupies the name, and there is no ownerReference, label or adoption check anywhere on the path. The validation is still only the inequality with the TLS Secret's own name, at api/redisfailover/v1/validate.go:134.
One correction to my own description: it is slightly worse than replacing Data. generateRedisCACertSecret sets OwnerReferences, and the write is a full Update, so overwriting a foreign Secret also reparents it to the RedisFailover and it gets garbage-collected when that object is deleted.
Still not reachable from the tenant surface, for the reasons I gave last round, and still the right destination is upstream. I am leaving it open rather than closing it because nothing in this tree changed about it.
New in this round
[MAJOR] hack/e2e-chainsaw/redis/chainsaw-test.yaml:578, the negative dial probes are fail-open, the same defect that just closed one file over
if k -n "$NAMESPACE" exec rfr-redis-flip-0 -c redis -- \
timeout 15 redis-cli --tls --insecure -h "rfr-redis-flip.$NAMESPACE.svc" -p 6379 \
--user pinger --pass pingpass --no-auth-warning ping 2>/dev/null | grep -qx PONG; then
echo "a TLS listener appeared on an instance whose flip was rejected" >&2
exit 1
fiStderr is discarded and the verdict is "did the word PONG appear". A missing pod, a 403 on exec, a wrong container name, or the outer timeout 30 firing all produce no PONG, so the if is false and the assertion reports success. It can only fail when a TLS listener genuinely answers; it cannot tell "there is no listener" from "I could not ask". set -eu does not help, because the if consumes the pipeline's status.
Driven against stubbed outcomes, four distinct failures to ask all read as the refusal being asserted:
refused exit=0 read as: no TLS listener
notimeout exit=0 read as: no TLS listener
nopod exit=0 read as: no TLS listener
rbac exit=0 read as: no TLS listener
tlsup exit=1 a TLS listener appeared on an instance whose flip was rejected
This is the security-relevant half of the flip case: it is the assertion that a rejected flip really did leave TLS off. The notimeout row is a mode the timeout fix introduced by putting a binary in the exec path; it is not live today because both images carry it, but nopod and rbac are live now.
The shape you already use twenty lines above is the fix, and you apply it correctly to the positive dial in the same script:
pong=$(k ... timeout 15 redis-cli --tls --insecure ... ping 2>&1) \
|| pong='(could not ask: exec failed or hit the cap)'
case "$pong" in
*PONG*) echo "a TLS listener appeared on an instance whose flip was rejected" >&2; exit 1 ;;
*"Connection reset"*|*"connection refused"*|*"I/O error"*|*"no reply"*) : ;;
*) echo "cannot determine whether a TLS listener is up: $pong" >&2; exit 1 ;;
esachack/e2e-chainsaw/redis/chainsaw-test.yaml:399 has the identical shape for the mirror assertion, that the plaintext listener does not answer once TLS is on, and needs the same treatment.
Caveats
- The E2E job on this exact head has not finished:
E2E (in-tree)in run 33645509783 is queued. My timing evidence for the closed stall comes from run 33608673237 ond5efbd228, a superseded rewrite of the same logical tip whose chainsaw file I compared against this head. A green run here would be the direct proof. - The two
.diff-anchored items and thevalues.yamlone are not inline comments: those files are either untouched by this PR or came from main with the rebase, so GitHub has no changed line to hang them on. - Valkey is not this PR's work and I am not asking for anything about it here beyond the digest pin. For the record and for whoever owns that package: an externally exposed Valkey is plaintext-only with no TLS option in its schema, which is an asymmetry with Redis, and there is no
check-valkey-rd-secrets.batscounterpart to the redis Secret-surface guard.
Checked and correct
- The vendored chart is not drifted from its patches, which is the failure mode where
make updatesilently reverts a hand-edit. Pulling upstream 3.3.5 fresh and applyingcrd-tls.diffandrbac-tls.diffreproduces the committedcharts/tree with no differences. - The carried patch builds and passes its own suite: upstream v3.3.5 tarball with
manifests charts example .githubremoved exactly as the Dockerfile does,git apply --checkclean for all three diffs,go build ./...exit 0,go test ./...all ok. TestGeneratedCRDRejectsTurningTLSOnOrOffskips in the image build because the Dockerfile deletesmanifests/; staging the shipped cozystack CRD into the fixture paths runs it for real, and it passes including the accepted-transition cases.
| # Expiry of that cap IS the refusal being asserted: a plaintext | ||
| # listener never sends a ServerHello, so redis-cli waits it out and | ||
| # prints no PONG for grep to find. | ||
| if k -n "$NAMESPACE" exec rfr-redis-flip-0 -c redis -- \ |
There was a problem hiding this comment.
[MAJOR] the negative dial probe is fail-open, the same defect that just closed one file over
Stderr is discarded and the verdict is "did the word PONG appear". A missing pod, a 403 on exec, a wrong container name, or the outer timeout 30 firing all produce no PONG, so the if is false and the assertion reports success. It can only fail when a TLS listener genuinely answers; it cannot tell "there is no listener" from "I could not ask". set -eu does not help, because the if consumes the pipeline's status.
Driven against stubbed outcomes, four distinct failures to ask all read as the refusal being asserted:
refused exit=0 read as: no TLS listener
notimeout exit=0 read as: no TLS listener
nopod exit=0 read as: no TLS listener
rbac exit=0 read as: no TLS listener
tlsup exit=1 a TLS listener appeared on an instance whose flip was rejected
This is the security-relevant half of the flip case: it is the assertion that a rejected flip really did leave TLS off. The notimeout row is a mode the timeout fix introduced by putting a binary in the exec path; it is not live today because both images carry it, but nopod and rbac are live now.
The shape you already use twenty lines above is the fix, and you apply it correctly to the positive dial in the same script:
pong=$(k ... timeout 15 redis-cli --tls --insecure ... ping 2>&1) \
|| pong='(could not ask: exec failed or hit the cap)'
case "$pong" in
*PONG*) echo "a TLS listener appeared on an instance whose flip was rejected" >&2; exit 1 ;;
*"Connection reset"*|*"connection refused"*|*"I/O error"*|*"no reply"*) : ;;
*) echo "cannot determine whether a TLS listener is up: $pong" >&2; exit 1 ;;
esacLine 399 has the identical shape for the mirror assertion, that the plaintext listener does not answer once TLS is on, and needs the same treatment.
There was a problem hiding this comment.
Both probes rewritten on 3cf9431 in the capture-and-classify form, with one change to your sketch: pong=$(...) || pong='(could not ask)' would overwrite the captured text on every non-zero exit, and redis-cli exits 1 on the real refusal too, so the refusal would have read as "could not ask" and failed. The exit code is kept in rc and the text stays in pong.
The accepted refusal is the one each listener actually gives, verified by running redis-cli in the shipped images rather than from the old comment. Here (flip case): a plaintext listener holds a ClientHello in silence, because processInlineBuffer scans for a newline with strchr and the handshake header carries a NUL before any newline, so redis-cli sits until the inner timeout 15 kills it, silent, exit 124. What the script accepts is kubectl's command terminated with exit code 124 trailer, which only appears when the exec reached the container and the inner cap fired; the outer cap kills kubectl without it. On line 399 (mirror) the TLS listener resets the plaintext client at AUTH and redis-cli prints I/O error, so that line is the refusal there, and an inner-cap expiry on that probe is a state the check did not read and fails.
Stubbed the k() wrapper for both: refusal signature exits 0; empty output with 124 (outer cap), pod not found, forbidden, missing container, missing timeout, unresolvable name, connection refused and an SSL negotiation failure all exit 1 with the dial's output in the message; PONG exits 1 with the security message. The old probes on the same stubs exit 0 on everything except PONG.
| # `port 0` beside the TLS port, and every check above would stay | ||
| # green if it stopped doing so. The mirror of the flip case: here the | ||
| # plaintext dial is the one that must not answer. | ||
| if k -n "$NAMESPACE" exec rfr-redis-secure-0 -c redis -- \ |
There was a problem hiding this comment.
[MAJOR] mirror of the fail-open probe: the plaintext-must-not-answer assertion cannot fail on an unaskable pod
Same shape as line 578 and the same consequence, for the mirror claim. 2>/dev/null | grep -qx PONG inside an if means a missing pod, a 403 on exec, or the outer timeout 30 firing all read as "the plaintext listener is gone", which is exactly the thing this step exists to prove.
This one carries a little more weight than its twin, because it is the only assertion that port 0 actually took effect. The comment above it says as much: every other check in the case would stay green if the operator stopped writing port 0. So if this probe cannot distinguish "no plaintext listener" from "could not reach the pod", nothing in the suite witnesses that TLS replaced plaintext rather than joining it.
Fix is the capture-and-classify form from the finding on line 578.
There was a problem hiding this comment.
Same fix on 3cf9431; the accepted refusal here is the I/O error redis-cli prints when the TLS listener resets the plaintext client at AUTH. Detail in the thread on line 578.
| apiVersion: chainsaw.kyverno.io/v1alpha1 | ||
| kind: Test | ||
| metadata: | ||
| name: redis-tls |
There was a problem hiding this comment.
[MAJOR] still open from my earlier round: nothing exercises an existing cluster (round 2)
All four Chainsaw cases still create their instances from scratch under the new operator and the new CRD:
$ grep -rn '^ name: ' hack/e2e-chainsaw/{redis,valkey}/chainsaw-test.yaml
redis/chainsaw-test.yaml:5: name: redis
redis/chainsaw-test.yaml:86: name: redis-tls
redis/chainsaw-test.yaml:463: name: redis-tls-flip-rejected
valkey/chainsaw-test.yaml:5: name: valkey
$ grep -rn 'file: redis\|file: valkey' hack/e2e-chainsaw/{redis,valkey}/chainsaw-test.yaml
redis:20 redis.yaml redis:115 redis-tls.yaml redis:486 redis-flip.yaml
redis:522 redis-flip-tls.yaml valkey:20 valkey.yaml
Every fixture is a fresh apps.cozystack.io/v1alpha1 object. redis-tls-flip-rejected does edit a running plaintext instance, but it created that instance itself under this branch, so the transition an existing customer actually takes is still untested.
Two things did materially de-risk it since my last round, and both arrived as hermetic pins rather than as arguments. The spec-level CEL rule was exercised against the real apiextensions validator with the shipped cozystack CRD staged into the fixture paths, and an object with no tls block stays updatable because the rule evaluates false == false on both sides. And the rebuilt binary renders a byte-identical plaintext StatefulSet, Sentinel Deployment and both ConfigMaps compared against a pristine v3.3.5 build, for Engine: Valkey as well; TestPlaintextProbesKeepUpstreamCommands guards that and is not inert, since making redisCLITLSFlags return --tls for a plaintext failover turns all four of its subtests red.
What is still unpinned is what needs an apiserver: SSA field ownership on the already-running rfr-* StatefulSet and rfs-* Deployment, and the CreateReplace of the CRD against stored objects. A Chainsaw case that pins the operator image to the pre-patch digest, creates the instance, then swaps to the rebuilt digest would close this permanently.
There was a problem hiding this comment.
Replayed on a live cluster today, against 25a061818. Starting point: a plaintext Redis with two replicas created under the pre-patch operator (v3.3.5, digest 551f3934), running for seven hours. The upgrade path mirrored what helm-controller does for this package: the CRD replaced first (upgradeCRDs: CreateReplace is set on the PackageSource), then the operator chart upgraded to this branch with the rebuilt image.
Results:
rfr-redis-upgrade-base:currentRevision == updateRevision(6d57f8f7f9) before and after, 2/2 Ready, zero restarts, pod start times unchanged. The StatefulSet and Sentinel Deployment specs hash identical before and after, andmanagedFieldsshows no write from the new operator at all: its lastUpdateon both objects is still the creation.- The stored
RedisFailoversurvived the CRD replace with the sameresourceVersion, took a metadata edit, and a server dry-runcustomConfigappend went through (generation 2). - The replaced schema accepts
spec.tlson a new object (server dry-run create), refuses the flip on the live one with the CEL message, and prunes an unknowntls.boguskey. - Plaintext listener:
PINGreturnsPONG, aSET/GETround trip works, the role is stillmaster.
On the Chainsaw case that pins the pre-patch digest and swaps: I agree it is the permanent form of this, and I would rather land it as its own PR. It needs the old digest reachable from the sandbox registry and an ordered CRD swap in the harness, which is e2e plumbing rather than this change. If you want it in here, say so.
| - - "get" | ||
| + - create | ||
| + - get | ||
| + - update |
There was a problem hiding this comment.
[MINOR] still open from my earlier round: no ownership check on the Secret write (round 2)
The only change since last round is a no-op skip when the stored content already matches. CreateOrUpdateSecret still issues a full Update on whatever Secret occupies the name, and there is no ownerReference, label or adoption check anywhere on the path. The validation is still only the inequality with the TLS Secret's own name, at api/redisfailover/v1/validate.go:134.
One correction to my own description from last round: it is slightly worse than replacing Data. generateRedisCACertSecret sets OwnerReferences, and the write is a full Update, so overwriting a foreign Secret also reparents it to the RedisFailover and it gets garbage-collected when that object is deleted.
Still not reachable from the tenant surface, for the reasons I gave last round, and still the right destination is upstream. I am leaving it open rather than closing it because nothing in this tree changed about it.
There was a problem hiding this comment.
The reparenting point is taken: a full Update carrying OwnerReferences adopts a foreign Secret, not just its data, so the upstream guard has to refuse the write when the stored object is not owned by the failover. It is on the freshworks-oss/redis-operator#85 follow-up list next to the CA-publish opt-out; nothing changes here by design, and this thread stays open until that lands.
|
On the pinned digest in |
The two negative dials in the chainsaw suite, the plaintext dial at a TLS instance and the TLS dial at the instance whose flip was rejected, decided by whether PONG appeared on stdout with stderr discarded. A missing pod, a 403 on exec, a wrong container name or the outer cap firing print no PONG either, so each of them passed as the refusal being asserted, and the flip case is the check that a rejected flip left TLS off. Keep what the dial printed and pass only on the refusal each listener actually gives, verified in the shipped images: a TLS listener drops a plaintext client at AUTH and redis-cli reports it as an I/O error; a plaintext listener holds a ClientHello in silence, because the inline parser's newline scan stops at a NUL in the handshake header, so the inner cap expires and kubectl's exit-code trailer carries the 124. That trailer is what separates the cap firing inside the pod from the wrapper's cap firing outside it. A listener that answers PONG fails the check, and anything else fails as a state the check did not read, with the dial's output in the message. Assisted-By: Claude <[email protected]> Signed-off-by: Aleksei Sviridkin <[email protected]>
IvanHunters
left a comment
There was a problem hiding this comment.
Verdict
LGTM with non-blocking notes
Round three, against 3cf9431f. The delta since my last round is one file, hack/e2e-chainsaw/redis/chainsaw-test.yaml, +54/-13, and it is the fix to both findings I raised. I checked the two rewrites by driving them rather than by reading them, and the E2E run on this head settles the one I raised two rounds ago.
Three items stay open and none of them blocks: one is a coverage gap whose replay has been run and reported, one is a packaging inconsistency whose published artifact is already correct, and one is an upstream hardening item we agree on. They are below with what would retire each.
One correction of mine to record: you were right to change the sketch I posted. pong=$(...) || pong='(could not ask)' overwrites the captured text on every non-zero exit, and redis-cli exits 1 on the genuine refusal too, so the refusal would have been read as "could not ask" and the case would have failed on the very state it asserts. Keeping the text in pong and the status in rc is correct.
Closed since my last round
Both negative dial probes. The fail-open is gone, and I checked it by extracting each block and driving it against stubbed kubectl outcomes rather than by reading the diff:
flip probe plaintext probe
PONG fails (listener found) fails (listener answers)
own refusal accepted accepted
other's refusal rejected rejected
missing pod rejected: cannot determine rejected: cannot determine
403 on exec rejected: cannot determine rejected: cannot determine
outer k() cap rejected: cannot determine rejected: cannot determine
wrong container rejected: cannot determine rejected: cannot determine
connection refused rejected: cannot determine rejected: cannot determine
Every "I could not ask" state now fails closed, each probe accepts only the refusal its own listener produces, and each rejects the other's. That last column is what makes this a real classification rather than a broader pattern match.
The suite stall. Settled by CI on this head rather than by argument. From the E2E job:
--- PASS: chainsaw/redis-tls-flip-rejected (75.17s)
--- PASS: chainsaw/redis-tls (60.60s)
--- PASS: chainsaw/redis (56.39s)
--- PASS: chainsaw/valkey (63.89s)
Against 2h58m56s of silence at that step and fourteen suites that never started, two rounds ago. The TLS fixture this PR's description walks through has now actually executed.
Still open, none blocking
hack/e2e-chainsaw/redis/chainsaw-test.yaml:86, no suite starts from an existing cluster
All three redis cases still create their instances from scratch under the new operator and the new CRD, so the transition an existing customer takes is not encoded anywhere that runs on every push.
I am not blocking on it, and I want to be exact about why, because the evidence is not mine. In my last round I wrote that a clean replay or an in-suite image swap would move this to a non-blocking note. You ran the replay and reported it clean with specifics: a plaintext instance on the pre-patch operator, CRD replaced first, currentRevision == updateRevision unchanged, 2/2 Ready, zero restarts, same resourceVersion, managedFields showing no writes. A hermetic review cannot reach a cluster, so that is your measurement and not my verification, and I am taking it at the value I said I would. What I did verify myself, in an earlier round and again here, is the part that does not need a cluster: the plaintext render is byte-identical across the change, the patched operator emits identical plaintext StatefulSet, Sentinel Deployment and ConfigMaps against a pristine v3.3.5 build, and the transition rule evaluates false == false on a stored object with no tls block, so it cannot fire on an upgrade that changes nothing.
What retires it: the Chainsaw case you propose as a separate PR, pinning the operator image to the pre-patch digest, creating the instance, then swapping to the rebuilt digest. Until that lands, this path is covered by a measurement nobody can re-run from the repository.
packages/system/redis-operator/patches/rbac-tls.diff:44, no ownership check on the Secret write
Unchanged this round, and by design. CreateOrUpdateSecret still issues a full Update on whatever Secret occupies the name, with no ownerReference, label or adoption check on the path, and the only validation remains the inequality with the TLS Secret's own name. Agreed that the guard belongs upstream in freshworks-oss/redis-operator#85 alongside the CA-publish opt-out, and agreed on the reparenting point: the write carries OwnerReferences, so overwriting a foreign Secret adopts it and it is garbage-collected with the failover. Not reachable from the tenant surface as the platform ships. Leaving the thread open until the upstream change lands is the right shape.
packages/system/redis-operator/values.yaml:4, the committed digest is still the pre-patch build
Third round, still untouched, still v3.3.5@sha256:551f3934..., a digest stamped by a commit whose tree carries only labels.diff. The published artifacts are fine: the package is in the root build: list, make image stamps the fresh digest, and the release and main lanes run it. What stays reachable is a tree read without a build, which is make -C packages/core/platform image on its own or the dev-loop deploy: there the chart admits and renders spec.tls against a binary that ignores it and serves plaintext, with no signal anywhere. Through Valkey the same pin also predates valkey-cli-auth.diff, so a Valkey instance built from the committed tree gets NOAUTH on its readiness script. I accept that a hand-pinned digest goes stale the moment CI stamps a new one, so I am not asking for it again; recording it because it has now survived three rounds and the next reader of this file will hit the same question.
Caveats
- The
E2E Testscheck on this head is green but readsin-tree e2e soft: node-join deadline missed, suite failed, merge not blocked. The suite did fail. The failures arekubernetes-latestandkubernetes-previous, both on the node-join deadline, and the job log says so explicitly, so nothing here is redis-adjacent. Worth knowing that a green tick on this lane is not the same as a green suite. - The refusal strings the two probes now key on,
I/O errorfor a TLS listener dropping a plaintext client and kubectl'scommand terminated with exit code 124trailer for a plaintext listener holding a ClientHello, are the load-bearing constants of both assertions. I confirmed the classification logic behaves correctly given those strings; whether the shippedredis-cliand kubectl emit exactly them is what the passing E2E run on this head demonstrates, not something I checked independently. If either string ever drifts, both probes fail closed and say "cannot determine", which is the right direction to fail in. - Not re-reviewed this round: everything outside the one changed file. The carried operator patch, the chart, the CRD and the RBAC diff were reviewed in rounds one and two and are byte-identical to what I looked at then.
| apiVersion: chainsaw.kyverno.io/v1alpha1 | ||
| kind: Test | ||
| metadata: | ||
| name: redis-tls |
There was a problem hiding this comment.
[MINOR] still open, not blocking: no suite starts from an existing cluster (round 3)
All three redis cases still create their instances from scratch under the new operator and the new CRD, so the transition an existing customer takes is not encoded anywhere that runs on every push.
Not blocking, and the evidence for that is yours rather than mine. Last round I wrote that a clean replay or an in-suite image swap would move this to a non-blocking note; you ran the replay and reported it clean with specifics (pre-patch operator, CRD replaced first, currentRevision == updateRevision unchanged, 2/2 Ready, zero restarts, same resourceVersion, managedFields showing no writes). A hermetic review cannot reach a cluster, so I am taking that at the value I said I would rather than re-deriving it.
What I did verify without a cluster, in an earlier round and again here: the plaintext render is byte-identical across the change, the patched operator emits identical plaintext StatefulSet, Sentinel Deployment and ConfigMaps against a pristine v3.3.5 build, and the transition rule evaluates false == false on a stored object with no tls block.
What retires it: the Chainsaw case you propose as a separate PR, pinning the operator image to the pre-patch digest, creating the instance, then swapping to the rebuilt digest. Until that lands, this path rests on a measurement nobody can re-run from the repository.
| - - "get" | ||
| + - create | ||
| + - get | ||
| + - update |
There was a problem hiding this comment.
[MINOR] still open, not blocking: no ownership check on the Secret write (round 3)
Unchanged this round, and by design. CreateOrUpdateSecret still issues a full Update on whatever Secret occupies the name, with no ownerReference, label or adoption check on the path, and the only validation remains the inequality with the TLS Secret's own name.
Agreed the guard belongs upstream in freshworks-oss/redis-operator#85 alongside the CA-publish opt-out, and agreed on the reparenting point: the write carries OwnerReferences, so overwriting a foreign Secret adopts it and it is garbage-collected with the failover. Not reachable from the tenant surface as the platform ships.
Leaving the thread open until the upstream change lands is the right shape; nothing is being asked of this PR.
Andrei Kvapil (kvaps)
left a comment
There was a problem hiding this comment.
LGTM. A 6500-line patch against a vendored operator is the shape I open most suspiciously, so here is what changed my mind.
It is a carry, not a fork. The patch header pins the upstream commit it was cut from (freshworks-oss/redis-operator#85, 2ad2ed8d1) and states its own retirement condition — retire when a release carrying #85 lands. The chart patches re-apply at the end of update: the same way reloader, grafana-operator, ingress-nginx, seaweedfs and kamaji carry theirs, which matters because rm -rf charts would otherwise take TLS back out on the next make update and nothing would fail.
The non-TLS change is better than the header claims. The header names MakeSlaveOfWithPort as taking effect with TLS off and argues it is safe because it changes a dial rather than a pod template. Reading the hunk, it is a latent bug fix:
- Addr: net.JoinHostPort(ip, masterPort), // this is IP and Port for the RedisFailover redis
+ Addr: net.JoinHostPort(ip, localPort),The old code dialled the replica at ip on the master's port, with a comment that read as if that were intended. It is invisible while both run the same port, which is the same-cluster failover path, and it breaks external-master mode where masterPort belongs to a Redis outside the cluster. MakeSlaveOf now passes redisPort for both and preserves the old behaviour on that path. So this is neutral for every current deployment and a fix for one that was broken.
Opt-in is typed, not just documented. Enabled *bool with no default, and the field doc says outright that TLS is not inferred from external — which is the wrong inference someone would otherwise reach for, given both concern reachability. AuthClients carries its own consequence in the doc: the directive is read once at startup, so changing it restarts Redis and Sentinel.
The RBAC delta is the part I would hold up as an example. Create, Get and Update on Secrets, which the header states are every Secret call in the patched source; no list and no watch because the Secret is addressed by name; nothing patches or deletes. Certificates are deliberately not granted, and the rationale sits at the rule site rather than only in the commit, so a future re-vendor reads the absence as a decision instead of an omission. That is narrower than the ported operator's own chart asks for.
CI is green including E2E on 3cf9431f1.
The thing to keep an eye on is the retirement condition, since it depends on an upstream PR merging and releasing — worth tracking somewhere that outlives this PR, so the carry does not quietly become permanent.
…ss fork (#3406) ## What this PR does `spotahome/redis-operator` was archived on 2026-06-11 (read-only; last release v1.2.4, 2022). This migrates the operator to the actively-maintained **freshworks-oss/redis-operator** fork (v3.3.5, Apache-2.0). ### Why freshworks-oss It keeps the same `RedisFailover` CRD and API group (`databases.spotahome.com`) — so existing managed Redis is unaffected — and adds a native `spec.engine: Redis|Valkey` switch, letting managed Valkey drop its Redis-compatibility shim image and run the official `valkey/valkey` image directly. ### Changes - Re-vendor the chart from `oci://ghcr.io/freshworks-oss/charts/redis-operator` (v3.3.5), which carries the `engine`-aware CRD. - Build the operator image from freshworks-oss v3.3.5 source. - Re-target the label/annotation-preservation patch onto the freshworks Go module (`github.com/freshworks/redis-operator`). ### Verification - freshworks-oss source + our `labels.diff` patch compiles natively with Go 1.26.3 (`service` and `cmd` packages). - `git apply --check` of the re-targeted patch against v3.3.5 succeeds. - The operator chart renders with the cozystack-registry image override. - The image is built (freshworks-oss v3.3.5 source + our patch, multi-arch amd64/arm64), pushed to `ghcr.io/cozystack/cozystack/redis-operator`, and pinned by digest in `values.yaml`. - `hack/promote-retag_test.bats` and `hack/nightly-mirror_test.bats` pass with the pinned digest. ### Note for maintainers `packages/system/redis-operator` is not part of the automated `make build`; its image is built out-of-band and the digest is pinned in `values.yaml`. That image is already built and pinned in this PR; re-run `make -C packages/system/redis-operator image` only if you bump the operator version. ### Follow-up The cozystack/redis-operator fork carries cert-manager TLS work (#2729) that freshworks-oss does not have yet; the plan is to upstream it to freshworks-oss so it lands in the shared maintained fork rather than a cozystack-only one. ### Release note ```release-note feat(redis-operator): migrate from the archived spotahome/redis-operator to the actively-maintained freshworks-oss/redis-operator fork (same RedisFailover CRD, adds native spec.engine: Redis|Valkey) ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added support for injecting additional environment variables into the Redis Operator (including a default operator group id). * Added default configurable labels for the Redis Operator. * **Bug Fixes** * Preserves and merges existing Service labels and annotations during Redis Operator updates. * **Chores** * Updated the Redis Operator and Helm chart to the Freshworks-maintained 3.3.5 release, aligned chart/image sources, and simplified Helm chart retrieval. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
What this PR does
tls.enabledputs Redis and Sentinel on a TLS-only listener with a per-release cert-manager chain, and hands the tenant the CA certificate asredis-<name>.tenant-ca. Off unless set.redis-operator v3.3.5 has no TLS support. The work is ported upstream in freshworks-oss/redis-operator#85, which is open and not in a release, so this carries it as three patches over the v3.3.5 tarball.
images/redis-operator/patches/tls.diffis the source half, cut from that PR's commit2ad2ed8d1, which the patch header pins; it is applied by the glob the image Dockerfile already runs, after the Dockerfile strips the tarball'smanifests,charts,exampleand.github, so the image builds from source alone and the upstream tests that read the CRD YAML skip instead of failing against the stale v3.3.5 copy.patches/crd-tls.diffaddsspec.tlsto the vendored CRD, and with it the CEL rule that fixestls.enabledat creation; without the schema block the apiserver prunes the field on admission and the patched binary waits for something it never sees, and the rule cannot exist without the block it references.patches/rbac-tls.diffgrants Create, Get and Update on Secrets, so the operator can publish the CA-only Secret it addresses by name. That is the whole delta, and it is narrower than the ported operator's own chart asks for: those three are the only Secret calls in the patched source, and cert-manager Certificates get no grant at all, because the operator only touches one underspec.tls.certManagerand this chart pinscertificateSecretmode instead. One change intls.diffis not TLS and takes effect with TLS off, named in the patch header:MakeSlaveOfWithPortconnects to the replica on its own port rather than on the master's. It changes a dial, not a pod template, so shipping the rebuilt image rolls nothing; the plaintext pod templates stay byte-identical to v3.3.5.The image build registration this branch used to carry (
make -C packages/system/redis-operator imagein the rootbuild:target) landed on main with the Valkey service (#3380), so the rebuilt operator image is produced by CI from main as well; the digest invalues.yamlis still refreshed by release preparation, not here.The chart patches re-apply at the end of
update:, the same way reloader, grafana-operator, ingress-nginx, seaweedfs and kamaji carry theirs. Otherwiserm -rf chartstakes TLS back out on the nextmake updateand nothing fails.make updatereproduces the committed tree byte for byte.TLS is chosen when the instance is created. The config gets
port 0and the listener moves totls-port, so TLS replaces plaintext rather than running beside it, and a running instance cannot change mode: a replica restarted onto the TLS listener cannot replicate from a master still on plaintext, and Sentinels already on TLS would fail over to a stale replica. The RedisFailover schema in the carried CRD patch rejects a change oftls.enabled, which on an existing instance surfaces as a failed HelmRelease upgrade while the instance keeps serving what it did; the README points at creating a new instance and migrating. That is also why TLS is not inferred fromexternal: an externally reachable instance is the one most likely to have clients an upgrade must not drop. Existing releases keep their plaintext listener and are not restarted by the rebuilt operator image.The tenant gets
ca.crtand nothing else, throughtenantsecrets. The chart's own Secrets take a dot between the release and the suffix (<release>.ca-tls,<release>.tls,<release>.ca-cert): the suffixes are dot-free, so the split is unique and releasefoo's CA Secret can no longer be releasefoo-ca's leaf, the collision the hyphenated form had. The chart renders aTenantProjectionpointing at<release>.ca-tls, and the CA-extraction controller re-encodes the one named key from the parsed certificate, so nothing that is not a certificate gets through. Same path as postgres, mongodb, nats and qdrant.No Secret of this release is granted to the tenant by name beyond the generated password, and that includes
<release>.ca-cert, the CA-only copy the operator publishes for itself. It holds no key today, but the operator has Create and Update on Secrets cluster-wide, so what sits at that name is the operator's call rather than the platform's, and RBAC cannot filter by key.hack/check-redis-rd-secrets.batspins the whole rendered Secret surface instead of a list of known-bad names, because an include with noresourceNamesand nomatchLabelsrestricts nothing.The chainsaw suite for redis gains two cases. A TLS fixture asserts the cert-manager chain is issued, the TLS-probed Redis replicas and Sentinel come up, the operator publishes its CA-only Secret under the widened ClusterRole, the tenant receives exactly
ca.crtthroughtenantsecretswhile the key-bearing Secrets stay off that path, and aredis-cli --tlsPING through the service name completes with the same CA. A second case appliestls.enabledto a running plaintext release and asserts the refusal: the HelmRelease upgrade fails with the schema's message, the StatefulSet keeps its revision with both replicas Ready, plaintext still answers and no TLS listener appears.Certificate renewal rolls the pods from inside the operator. Redis and Sentinel read the certificate once at startup, so the patched operator stamps a hash of the TLS Secret's
tls.crtandca.crtinto the pod template of both the Redis StatefulSet and the Sentinel Deployment on every reconcile, asredis-failover.freshworks.com/tls-secret-hash. A renewal changes the hash; the operator's own roller replaces the Redis pods one at a time and the Deployment controller rolls Sentinel. README names the fallback: if the roll cannot run, the deadline is the old certificate's expiry, past which the operator's own TLS connections fail along with the clients.tls.authClientsmaps totls-auth-clientsand defaults tono. The platform mints no client certificate for the tenant and the CA key is out of reach, soyesis only usable when the client certificate comes from whoever also controls the CA. Changing the value on a running release restarts the Redis and Sentinel pods: Redis reads the directive from its config file only at startup, so the operator folds the effective value into the certificate hash that drives the rollout, and the change lands when it is made rather than at the next unrelated restart.Follow-ups in the carried operator patch go upstream first, to freshworks-oss/redis-operator#85 or a successor there, and are picked up here afterwards; the carried patch does not grow Go that upstream has not seen.
tlsConfigForfetches the Secret on every call, tens of GETs per reconcile. The renewal roll is static analysis so far: the live check,cmctl renewon a running release while watching therfr-*andrfs-*pods roll, is still owed and belongs with the e2e follow-up.Two facts about the shipped image rather than the patch.
packages/system/redis-operator/values.yamlkeeps the digest of the image built before the patch until release preparation rebuilds it, so onmainbetween this merge and the next release-prep,tls.enabledrenders the chain andspec.tlswhile the running operator ignores them and serves plaintext. And the pinned v3.3.5 tarball carries GO-2026-5026 ingolang.org/x/netv0.53.0, reached throughPodService.UpdatePodLabels; the port neither introduces nor fixes it, upstreammasteris already on v0.55.0, so it lands with the next tag bump rather than here.Outside the patch:
cozyvalues-genrenders{}in the README Value column fortls.authClients: a child property without a default inherits the parent object's{}, andmake generatereproduces it byte for byte, so it is a generator bug rather than something to hand-edit here.Screenshots
No UI changes.
Downstream repositories
I walked the trigger map against the diff file by file. terraform-provider-cozystack is reached, tracked as cozystack/terraform-provider-cozystack#37 rather than a speculative PR:
values.schema.jsongains atlsobject, and the provider hand-writes a schema, model and expand/flatten pair per Kind. It already modelstlsfor postgres, kafka, nats and qdrant, not for redis. Redis needs different wording from those four, whose descriptions say to omit the block to followexternal, which is the one thing redis does not do.Nothing else is reached. The website regenerates this package's reference page from
README.mdon a stable tag, so the README changes need no PR there. No package added, renamed or removed.packages/core/platform/values.yaml, variant and bundle composition and release asset names are untouched, and so arehack/package.mk,hack/common-envs.mkandhack/update-crd.sh, which is what ccp and external-apps-example read. The two new files underhack/are bats unit tests. No node prerequisite, installer value, telemetry metric or cozy-proxy label changes.Release note