fix(clickhouse): scheme the backup S3_ENDPOINT on the system-bucket flow by androndo · Pull Request #3963 · cozystack/cozystack

Andrey Kolkov (androndo) · 2026-08-26T09:27:06Z

What this PR does

On the default cozy-default / useSystemBucket backup flow, the ClickHouse chart pointed the clickhouse-backup sidecar's S3_ENDPOINT straight at the projected cozy-backups-creds.endpoint, which the backup credentials projector stores as a bare host (no scheme) by design. altinity/clickhouse-backup runs S3_ENDPOINT through the aws-sdk-go-v2 endpoint resolver, which rejects a scheme-less value with Custom endpoint 's3.host' was not a valid URI — so every ClickHouse backup and restore on the default flow failed. Reproduced live on a v1.6.2 cluster: a cozy-default BackupJob against a useSystemBucket: true ClickHouse hangs, and the sidecar logs S3 ResolveEndpoint ... was not a valid URI.

The fix prepends https:// on the useSystemBucket path via a helper env plus $(VAR) expansion (a secretKeyRef value can't be string-interpolated), mirroring the platform's backupstrategy-controller.endpoint helper which force-prepends https:// for the always-TLS external S3 ingress. The legacy and bring-your-own-Secret paths keep reading endpoint verbatim, since that value is tenant-supplied and already carries its own scheme.

ClickHouse was the only cozy-default app hit by this. I audited every default-flow app: the CNPG/postgres, etcd and velero strategies already get a scheme'd URL from the endpoint helper, and mariadb and foundationdb deliberately pair a bare host with a separate TLS flag; mongodb reads a full URL from its own chart values. So this is the single scheme-class fix needed.

Not addressed here (separate, non-scheme gaps surfaced during the audit, flagged for follow-up): MongoDB has no useSystemBucket wiring at all (its cozy-default strategy assumes a manually-configured backup.endpointURL + per-app S3 creds), and FoundationDB's self-signed-CA trust to the operator pod is a known gap.

Screenshots

Not applicable (no UI changes).

Downstream repositories

No schema or API change (only a chart template and its helm-unittest). values.yaml / values.schema.json / the CRD are untouched, so nothing downstream is affected. Leaving every box unticked per the trigger-map guidance.

Release note

fix(clickhouse): backup and restore now work on the default system-bucket (useSystemBucket) flow — the clickhouse-backup sidecar receives a scheme'd https:// S3_ENDPOINT instead of the projected bare host, which the S3 client rejected as an invalid URI.

Summary by CodeRabbit

Bug Fixes
- Corrected ClickHouse backup endpoint configuration when using the system bucket.
- Ensured backup connections consistently use the required HTTPS endpoint format.
- Improved handling of backup credentials across system bucket, legacy, and custom-secret configurations.
Tests
- Added coverage validating endpoint and credential wiring for supported backup configurations.

On the useSystemBucket / cozy-default flow the chart pointed the clickhouse-backup sidecar's S3_ENDPOINT straight at the projected cozy-backups-creds.endpoint, which the backup credentials projector stores as a bare host by design. altinity/clickhouse-backup runs S3_ENDPOINT through the aws-sdk-go-v2 endpoint resolver, which rejects a scheme-less value with "Custom endpoint 's3.host' was not a valid URI", so every backup and restore on the default flow failed (reproduced on a live v1.6.2 cluster). ClickHouse was the only cozy-default app hit by this: the CNPG/etcd/velero strategies get a scheme'd URL from the backupstrategy-controller.endpoint helper, and mariadb/foundationdb deliberately pair a bare host with a TLS flag. Prepend https:// on the useSystemBucket path via a helper env plus $(VAR) expansion (a secretKeyRef value can't be string-interpolated), mirroring the platform's endpoint helper which force-prepends https for the always-TLS external S3 ingress. The legacy and bring-your-own-Secret paths keep reading endpoint verbatim, since that value is tenant-supplied and already carries its own scheme. Signed-off-by: Andrey Kolkov <[email protected]> Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

coderabbitai · 2026-08-26T09:27:14Z

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 4d04fef8-17de-40bd-920a-da4d03a14b73

📥 Commits

Reviewing files that changed from the base of the PR and between f5b957e and d646c15.

📒 Files selected for processing (2)

packages/apps/clickhouse/templates/clickhouse.yaml
packages/apps/clickhouse/tests/backup_test.yaml

Included review availability: Your plan provides up to 8 included reviews per hour; 6 remain after this review.

📝 Walkthrough

Walkthrough

The ClickHouse backup sidecar now constructs the system-bucket endpoint from a secret-provided host. Legacy and BYO-Secret flows continue to read the endpoint directly from their secrets. Helm tests cover both configurations.

Changes

ClickHouse backup endpoint wiring

Layer / File(s)	Summary
Endpoint rendering and validation `packages/apps/clickhouse/templates/clickhouse.yaml`, `packages/apps/clickhouse/tests/backup_test.yaml`	The system-bucket flow reads `endpoint` from `cozy-backups-creds` into `S3_ENDPOINT_HOST` and sets `S3_ENDPOINT` to `https://$(S3_ENDPOINT_HOST)`. Other flows read `S3_ENDPOINT` directly from their release-specific Secret. Tests verify both behaviors.

Estimated code review effort: 2 (Simple) | ~10 minutes

Merge Risk: 🔵 Low · up to d646c

The endpoint fix is localized and mergeable with owner awareness that the system-bucket credential reference must remain optional; requiring it could prevent pods from starting before credentials are projected.

Suggested reviewers: ivanhunters

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly and concisely describes the main change: adding the URL scheme to the backup S3_ENDPOINT for the system-bucket flow.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 0…
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

Full details: Docstring Coverage

Explanation

No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 0 files. (2 skipped: 2 unsupported.)

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests
Commit unit tests in branch fix/clickhouse-backup-endpoint-scheme

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

myasnikovdaniil

Scheme fix is correct when projected Secret exists before ClickHouse pod start. Lazy projection lifecycle predates this PR, tracked in #3986.

github-actions · 2026-08-28T11:05:44Z

Successfully created backport PR for release-1.6:

[Backport release-1.6] fix(clickhouse): scheme the backup S3_ENDPOINT on the system-bucket flow #3987

… on the system-bucket flow (#3987) # Description Backport of #3963 to `release-1.6`.

github-actions Bot added size/M This PR changes 30-99 lines, ignoring generated files area/database Issues or PRs related to managed databases (postgres, mariadb, redis, etcd, kafka, clickhouse) kind/bug Categorizes issue or PR as related to a bug labels Aug 26, 2026

Andrey Kolkov (androndo) added the kind/backport Categorizes issue or PR as requiring a backport to the current release line label Aug 26, 2026

Andrey Kolkov (androndo) marked this pull request as ready for review August 27, 2026 07:25

Andrey Kolkov (androndo) requested review from IvanHunters, Andrei Kvapil (kvaps), Aleksei Sviridkin (lexfrei), Timofei Larkin (lllamnyp), myasnikovdaniil and scooby87 as code owners August 27, 2026 07:25

myasnikovdaniil mentioned this pull request Aug 28, 2026

ClickHouse system-bucket backup needs pod restart after first credential projection #3986

Open

myasnikovdaniil approved these changes Aug 28, 2026

View reviewed changes

Andrey Kolkov (androndo) merged commit ac00577 into main Aug 28, 2026
30 checks passed

Andrey Kolkov (androndo) deleted the fix/clickhouse-backup-endpoint-scheme branch August 28, 2026 11:05

github-actions Bot mentioned this pull request Aug 28, 2026

[Backport release-1.6] fix(clickhouse): scheme the backup S3_ENDPOINT on the system-bucket flow #3987

Merged

myasnikovdaniil added a commit that referenced this pull request Sep 3, 2026

[Backport release-1.6] fix(clickhouse): scheme the backup S3_ENDPOINT…

3a4e173

… on the system-bucket flow (#3987) # Description Backport of #3963 to `release-1.6`.

Navigation Menu

fix(clickhouse): scheme the backup S3_ENDPOINT on the system-bucket flow - #3963

fix(clickhouse): scheme the backup S3_ENDPOINT on the system-bucket flow#3963
Andrey Kolkov (androndo) merged 1 commit into
mainfrom
fix/clickhouse-backup-endpoint-scheme

Andrey Kolkov (androndo) commented Aug 26, 2026 •

edited by coderabbitai Bot

Loading

Uh oh!

coderabbitai Bot commented Aug 26, 2026 •

edited

Loading

Walkthrough

Changes

Uh oh!

myasnikovdaniil left a comment

Uh oh!

Uh oh!

github-actions Bot commented Aug 28, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

Andrey Kolkov (androndo) commented Aug 26, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What this PR does

Screenshots

Downstream repositories

Release note

Summary by CodeRabbit

Uh oh!

coderabbitai Bot commented Aug 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Uh oh!

myasnikovdaniil left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

github-actions Bot commented Aug 28, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Andrey Kolkov (androndo) commented Aug 26, 2026 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented Aug 26, 2026 •

edited

Loading