Skip to content

fix(redis): run the multi-arch redis_exporter so the sidecar starts on arm64 - #4522

Merged
Aleksei Sviridkin (lexfrei) merged 1 commit into
mainfrom
fix/redis-exporter-multiarch
Sep 28, 2026
Merged

Aleksei Sviridkin (lexfrei) merged 1 commit into
mainfrom
fix/redis-exporter-multiarch

Conversation

@lexfrei

@lexfrei Aleksei Sviridkin (lexfrei) commented Sep 27, 2026 •

Copy link
Copy Markdown
Contributor

What this PR does

Redis and Valkey instances, and the Redis inside Harbor, never become fully ready on arm64 nodes. Their failovers pin oliver006/redis_exporter:v1.55.0-alpine, which upstream publishes for amd64 only. On arm64 the exporter sidecar exits with exec /redis_exporter: exec format error, and the pod stays not ready.

The same release without the suffix is a multi-arch index with amd64, arm and arm64 builds. Its entrypoint, user and port match the -alpine image, and none of these charts run a shell in the exporter container. So this swaps the tag and pins it by digest, with no version bump.

I checked both tags with crane. On an arm64 test cluster a Redis and a Valkey instance hit the exec format error before the change and came up 2/2 with it. The Harbor Redis pods went to 2/2 the same way.

On upgrade the exporter image in the pod template changes, so the operator rolls every Redis and Valkey instance and Harbor's Redis, one pod at a time as it does for any failover update.

Fixes #4519

Screenshots

No UI changes.

Downstream repositories

The diff changes one image reference in three templates. No entry in the trigger map covers a sidecar image.

Release note

fix(redis): Redis, Valkey and Harbor's Redis now use the multi-arch redis_exporter image, so the exporter sidecar starts on arm64 nodes.

…n arm64

The redis, valkey and harbor Redis failovers pin
oliver006/redis_exporter:v1.55.0-alpine, which upstream publishes for
amd64 only. On an arm64 node the exporter sidecar exits with an exec
format error, so the Redis pod never becomes fully ready.

The unsuffixed v1.55.0 tag is a multi-arch index (amd64, arm, arm64)
of the same exporter release, with the same entrypoint, user and
port, and none of these charts run a shell in that container. Pin it
by digest, so the reference is reproducible.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <[email protected]>
@coderabbitai

coderabbitai Bot commented Sep 27, 2026 •

Copy link
Copy Markdown
Contributor

Review in Change Stack →

Navigate logical layers of code changes, visualize relationships, and explore their blast radius.

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Repository: cozystack/cozystack/.coderabbit.yaml

Review profile: CHILL

Plan: Advanced

Run ID: 073cbc27-3d50-4458-bd4c-6027e41df847

📥 Commits

Reviewing files that changed from the base of the PR and between 101d1d9 and 07700c7.

📒 Files selected for processing (3)
  • packages/apps/redis/templates/redisfailover.yaml
  • packages/apps/valkey/templates/valkeyfailover.yaml
  • packages/system/harbor/templates/redis.yaml

Included review availability: This review used your included allowance. Your plan provides up to 8 included reviews per hour; 6 remain after this review.


📝 Walkthrough

Walkthrough

The Redis, Valkey, and Harbor templates now use the multi-architecture Redis exporter v1.55.0 image, pinned by SHA-256 digest instead of the v1.55.0-alpine tag.

Changes

Redis exporter image

Layer / File(s) Summary
Update exporter image references
packages/apps/redis/templates/redisfailover.yaml, packages/apps/valkey/templates/valkeyfailover.yaml, packages/system/harbor/templates/redis.yaml
All three templates replace the v1.55.0-alpine image reference with v1.55.0 pinned by SHA-256 digest.

Priority: ➖ Normal

Estimated code review effort: 2 (Simple) | ~8 minutes

Change: Bug fix · Severity of issue fixed: Medium

Suggested reviewers: kvaps

Merge Risk: ⚪ Minimal · up to 07700

The Redis, Valkey, and Harbor exporter images are pinned to a v1.55.0 manifest that supports arm64, so no actionable merge-blocking risk remains from this change.

Security Architecture Review

Security architecture risk: 🔵 Low · up to 07700

All three deployments switch to the same digest-pinned exporter image without changing their declared exporter settings. No introduced security failure was established, but image compatibility and rollout behavior are not fully verified.

Retained concerns
No architecture-level concerns identified.

Security review details

Security Blast Radius

  • inferred — An image-level compatibility or integrity problem with the shared digest could affect exporter sidecars in Redis, Valkey, and Harbor; the evidence does not show a change to Redis data-store permissions or public exposure.

Trust Boundaries and Controls

  • observed — The inspected change does not modify declared exporter arguments or environment. The Redis TLS test retains assertions for TLS enablement, certificate and CA secret names, and loopback IP SANs; it does not prove the replacement image honors those controls at runtime.
🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Linked Issues check ✅ Passed The PR satisfies issue #4519. It changes the exporter image in packages/apps/redis/templates/redisfailover.yaml, packages/apps/valkey/templates/valkeyfailover.yaml, and `packages/system/harbor/tem…
Out of Scope Changes check ✅ Passed The whole-PR diff contains only the three exporter image reference changes required by issue #4519. No unrelated source, configuration, or behavior changes are present.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 0…
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly identifies the Redis exporter multi-architecture change and its arm64 startup fix. It matches the main purpose of the pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Commit to this branch
  • Create a new PR

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions github-actions Bot added area/database Issues or PRs related to managed databases (postgres, mariadb, redis, etcd, kafka, clickhouse) kind/bug Categorizes issue or PR as related to a bug size/XS This PR changes 0-9 lines, ignoring generated files labels Sep 27, 2026
@lexfrei
Aleksei Sviridkin (lexfrei) marked this pull request as ready for review September 27, 2026 01:33

@IvanHunters IvanHunters left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed statically in a hermetic clone. Confirmed via crane the old alpine tag is amd64-only and the new digest is a multi-arch index, with amd64 entrypoint, user and port unchanged, across all three sidecar call sites. LGTM. Two minor non-blocking notes.

@lexfrei
Aleksei Sviridkin (lexfrei) merged commit 597d966 into main Sep 28, 2026
20 checks passed
@lexfrei
Aleksei Sviridkin (lexfrei) deleted the fix/redis-exporter-multiarch branch September 28, 2026 16:36
Aleksei Sviridkin (lexfrei) added a commit that referenced this pull request Sep 29, 2026
…leases on it (#4505)

## What this PR does

Pre-release builds now publish every image as an amd64 and arm64 index,
and a release fails if anything it ships is single-arch. Main, release
lines and pull requests keep building amd64 only. It builds on #4498,
which made every image buildable for both architectures. Part of #1961.

**This PR is on hold until everything under "Merge after" has landed.**
Merged earlier, the new gate fails the next rc on the images those PRs
fix.

### How a pre-release is built

A pre-release tag gets a second job, for arm64, next to the existing
release job. That covers `-rc.N`, `-beta.N` and `-alpha.N` tags. It runs
on the CNCF arm64 pool (`oracle-vm-24cpu-96gb-arm64`), builds the same
images with `PLATFORM=linux/arm64`, and pushes them as `<tag>-arm64`.
`prepare-release` waits for it, and if the arm64 job fails the rc fails
with it. There is no amd64-only fallback. Stable tags do not run it:
promotion copies the rc digests with `skopeo copy --multi-arch all`, so
a stable release inherits the indexes.

After the amd64 build, the stitch joins each amd64 image with its arm64
twin into one index. It lives in `hack/stitch-multiarch.sh`. Every tag
of the amd64 image moves to the index, the component-versioned ones
included. The script then rewrites the digests in the tree and
republishes the packages artifact and the installer chart pinned on
them. It reads refs through `hack/lib/image-refs.sh`, and it fails if an
old digest survives anywhere outside `charts/`.

### The gate

A new check fails the rc when a pinned digest in the tree is not an
index with both linux/amd64 and linux/arm64. It lives in
`hack/verify-multiarch.sh`. It checks first-party and third-party refs
alike. Images that are amd64 by nature go into
`hack/multiarch-allowlist`, one repository per line, each with a
mandatory reason. Today that is only the e2e sandbox. An entry that
matches nothing is reported as stale.

The static gate only sees digest-pinned refs in the tree. A tag-only
ref, an image from a vendored chart default, or one an operator starts
at runtime is invisible to it. So the rc e2e also audits every image its
nodes pulled, with the same check and the same allowlist. Neither covers
a package the e2e never installs.

### Nightly

A new nightly workflow builds main for arm64. It keeps the arm64 build
cache warm, since the rc job only reads it. It also prints the
single-arch refs that would block the next rc. Nothing it builds is
published.

### Smaller changes

- `CACHE_TAG` in `hack/common-envs.mk` moves the default cache ref, so
the arm64 build keeps a cache of its own.
- `MATRIX_ARCH=arm64` makes `hack/build-matrix.sh` leave out the
amd64-only e2e sandbox.
- The kamaji provider image is pushed under the build's `IMAGE_TAG` like
every other image. Its Makefile set `IMAGE_TAG` itself, so concurrent
builds overwrote each other's component tag. Closes #4503.
- `hack/nightly-mirror.sh` verifies a copy against the raw manifest
digest. Plain `skopeo inspect` resolves an index to one platform's
child.
- matchbox is rebuilt for both platforms in the amd64 release job. The
arm64 leg skips the talos package, so there is no arm64 half to stitch,
and its Dockerfile only copies files, so no QEMU is needed. Releases
then network-boot arm64 machines too. Closes #4524.

The stitch also rewrites the kamaji ref inside `files/components.gz`,
recompressed with `gzip -n` so the bytes are reproducible, so the kamaji
control-plane provider ships multi-arch too. keda and kuberture now name
the repository next to their pinned digest, so the gate can resolve
them; their rendered manifests do not change.

### Merge after

- #4552 builds the Talos installer, matchbox, Harbor and the Velero
KubeVirt plugin for arm64, and adds flux-plunger, keycloak-operator,
kilo and migration-controller to the root `build:` list. It replaces
#4507, #4515, #4525 and #4528.
- #4549 moves ingress-nginx, cozy-proxy, cozystack-scheduler and
keycloak-kms-proxy to their multi-arch releases, which are already
published.

The rest of the stack is merged: #4485, #4498, #4512, #4517, #4521 and
#4522 here, and the multi-arch PRs in
ingress-nginx-with-protobuf-exporter, cozy-proxy, cozystack-scheduler
and keycloak-kms-proxy.

### Verification

The bats suites pass: stitch, verify-multiarch, build-matrix,
common-envs, nightly-mirror and the release contracts. The first three
also pass under `hack/cozytest.sh` with dash. Each new test was red
before its implementation, and the verify-multiarch and stitch tests
each have a mutation that turns them red. I ran `verify-multiarch
--report` against main. It lists the first-party refs the stitch will
fix, plus the third-party and special cases above. The workflows have
not run yet. The arm64 job's tools and duration, the stitch against real
registries, and the audit are checked for the first time on the nightly
and on the next rc.

### Screenshots

Not a UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

Nothing under `hack/` is moved or renamed, and no make target changes
its default behaviour: `CACHE_TAG` and `MATRIX_ARCH` are opt-in. The
satellite repositories listed under "Merge after" are prerequisites, not
follow-ups this change forces on them.

### Release note

```release-note
ci(release): pre-release builds publish every image as an amd64 and arm64 multi-arch index, and a release fails if any image it ships or pulls in e2e lacks either architecture. Stable releases inherit the indexes through promotion.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Release candidates now build amd64 and arm64 images in parallel and
combine eligible images into verified multi-architecture indexes before
publication.
* A nightly arm64 image build is available for testing and supplies a
build cache for release-candidate builds.
* **Bug Fixes**
* End-to-end checks audit whether pulled images support both
architectures. Audit failures block prerelease checks, while stable
releases continue with a warning.
* End-to-end test artifacts now include collected image references and
multi-architecture audit results.
* **Documentation**
* Updated release and image guidance covers multi-architecture builds,
verification checks, and troubleshooting, including arm64 build and
stitching failures.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/database Issues or PRs related to managed databases (postgres, mariadb, redis, etcd, kafka, clickhouse) kind/bug Categorizes issue or PR as related to a bug size/XS This PR changes 0-9 lines, ignoring generated files

Projects

None yet

Development

Successfully merging this pull request may close these issues.

redis, valkey, harbor: the redis_exporter sidecar is amd64-only, so it crash-loops on arm64

2 participants