feat(clickhouse): let the backup sidecar trust a private S3 CA, unblocking the backup e2e round-trip - #3385

hack/select-install.sh --validate (run in the unit-test job) requires every
Chainsaw suite dir to resolve to a PackageSource via suite_to_source(); the
new clickhouse-backup suite has no <suite>-application fallback, so map it to
cozystack.clickhouse-application, in lockstep with the select-e2e.sh override.

Assisted-By: Claude <[email protected]>
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Signed-off-by: Andrey Kolkov <[email protected]>

- Add a select-e2e_test.bats case asserting a packages/apps/clickhouse diff
  selects BOTH the clickhouse and clickhouse-backup suites (mirrors the
  kubernetes-application multi-suite test). Fails without the src_to_suites
  override that makes the backup contracts suite run on ClickHouse chart PRs.
- Correct the backup-api-auth Secret assertion's prose to match what it proves
  (present with both keys, not "populated").
- Capture the to-copy restore target's Pod logs in the round-trip catch so a
  failure isolated to step 07 keeps its diagnostics.

Assisted-By: Claude <[email protected]>
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Signed-off-by: Andrey Kolkov <[email protected]>

Extend verify-backup-sidecar to assert the sidecar's API_USERNAME/API_PASSWORD
env resolve valueFrom the backup-api-auth Secret's username/password keys, not
just that the Secret and the :7171 port exist independently. A chart change
that repointed or dropped those secretKeyRefs while keeping both would slip
through and surface only as 401s at BackupJob time — the silent break this
suite exists to catch.

Assisted-By: Claude <[email protected]>
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Signed-off-by: Andrey Kolkov <[email protected]>

The clickhouse-backup sidecar had no TLS/CA surface at all: bucket coordinates
and credentials arrive as S3_* env, and nothing let it verify an endpoint whose
certificate is signed by a private CA. Cozystack's own in-cluster SeaweedFS is
exactly that case — it serves :8333 behind the self-signed "SeaweedFS CA" — so a
ClickHouse release could only ever back up to a publicly-trusted endpoint, while
MariaDB (storage.s3.tls.caSecretKeyRef) and Postgres (barman endpointCA) have had
a CA field all along.

Add backup.endpointCA {name, key}. When set, the chart mounts that Secret into
the sidecar and points SSL_CERT_DIR at it. Empty (the default) renders no env, no
mount and no volume, so existing releases are untouched.

SSL_CERT_DIR rather than AWS_CA_BUNDLE, measured against
altinity/clickhouse-backup:2.7.4 with a self-signed endpoint and public AWS S3 in
turn: Go reads SSL_CERT_DIR in addition to the system bundle, so both verify;
AWS_CA_BUNDLE replaces the SDK trust pool, so the private endpoint verifies and
public S3 starts failing `x509: certificate signed by unknown authority`. That
would silently break a release that also talks to a public endpoint.
S3_DISABLE_CERT_VERIFICATION was rejected outright — it drops verification
instead of trusting one more CA.

Covered by helm-unittest: the unset default, the mount/env/volume wiring, a
custom key still projected as ca.crt, composition with useSystemBucket, and that
backup.enabled=false renders nothing.

Assisted-By: Claude <[email protected]>
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Signed-off-by: Andrey Kolkov <[email protected]>

The round-trip Test was skipping: it exited 0 behind CLICKHOUSE_E2E_S3_ROUNDTRIP,
so "PASS (1.01s)" meant the gate fired, not that backup and restore worked. Only
the contracts Test ran, which proves the chart still exposes the sidecar and its
auth Secret — not that a byte ever reaches S3 and comes back.

The gate existed because the sidecar could not verify the in-cluster SeaweedFS
certificate. backup.endpointCA closes that, so adopt the shape mariadb and
postgres already use to run their round-trips un-gated:

- examples: resolve the S3 endpoint from S3_ENDPOINT when set (BucketInfo
  advertises the external ingress URL, an unroutable placeholder in CI), copy the
  seaweedfs CA (auto-discovered from its cert-manager Certificate when the
  fullname default is absent) into a per-release Secret, and reference it from
  backup.endpointCA on both the source and the to-copy target. NAMESPACE now
  defaults to tenant-root, matching the other backup demos.
- suite: drop the gate, run in tenant-root — an isolated tenant's Cilium egress
  allowlist blocks its Pods from reaching tenant-root's seaweedfs — and pre-clean
  with cleanup.sh, because tenant-root is shared and a leftover Succeeded
  BackupJob would satisfy the harness's wait without a backup running.

cleanup.sh also drops the copied CA Secret; the platform's own is never touched.

Assisted-By: Claude <[email protected]>
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Signed-off-by: Andrey Kolkov <[email protected]>

Review findings on the previous two commits.

1. TIA no longer selected the suite that runs the harness. select-e2e.sh maps
   examples/backups/<app>/** to suite <app>, so with the round-trip living in a
   separate clickhouse-backup/ dir an edit to the harness selected `clickhouse`
   — which did not execute those scripts — while the suite that does was left
   out. Fixed the way postgres/mariadb/etcd already do it: both Tests move into
   hack/e2e-chainsaw/clickhouse/ as clickhouse-1-backup-contracts and
   clickhouse-2-backup-roundtrip. That also deletes every special case the
   earlier commits added (src_to_suites, suite_to_source, and the bats case that
   guarded them — both scripts are now byte-identical to main), and the suite
   list in hack/e2e-chainsaw/README.md stays correct because no dir is added.
   The bats case is replaced by one asserting the invariant that actually
   matters: an examples/backups/clickhouse edit selects the suite that runs it.

2. The docs claimed a fail-fast the harness did not have, and the budget was
   sized as if it did. wait_for_field polled to the full timeout with no
   terminal-value escape, and the HelmRelease/StatefulSet waits were bare
   `kubectl wait` with no existence backstop and no Stalled fail-fast. Ported
   wait_for_field's fail_value parameter and wait_hr_ready from the postgres
   helpers, added wait_sts_ready, and routed all eight waits through them.
   Verified against a fake kubectl: a Failed BackupJob now bails in 3s instead
   of 600s, a Stalled HelmRelease immediately. The inner budgets sum to 3900s,
   which the old 30m op timeout did not cover — a hang would have been SIGKILLed
   with no diagnostics, the failure mode .chainsaw.yaml calls strictly worse — so
   the timeout is now 65m with the arithmetic written out.

3. The strategy Pod ran `apk add --no-cache curl jq` on every backup and
   restore. Harmless while nothing in CI ran it; a per-PR dependency on the
   Alpine CDN once it does, and broken outright air-gapped — which is exactly
   why the shipped strategy bakes curl+jq into its image. Step 01 now reads that
   image off cozy-default-altinity and drops the apk add, falling back to
   alpine only when the platform strategy is absent. Also bounded the two
   polling curls with --max-time; a hung connection previously stalled the loop
   until the outer timeout.

4. 91-scenario-user-backup.md still documented the old tenant-test default.

5. .chainsaw.yaml claimed all app tests run in tenant-test, which the postgres
   and mariadb round-trips already contradicted before this branch.

The round-trip header now also states its scope plainly: it exercises the
example's strategy, not the shipped cozy-default-altinity.

Assisted-By: Claude <[email protected]>
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Signed-off-by: Andrey Kolkov <[email protected]>

Second review pass on this branch.

1. The op timeout equalled the inner sum instead of exceeding it: the comment
   enumerated 3900s of inner waits and then set 65m, and 3900s IS 65m — so there
   was zero headroom for the applies and SQL calls between those waits, and the
   claim of "room" was false. Sized the inner budgets against what CI actually
   takes instead (the bucket suite completes in ~37s and a backup-enabled
   ClickHouse plus Keeper reaches Ready in ~98s, measured on run 30353894332):
   bucket claim/access 300->180 and the three job waits 600->480, summing to
   3300s, under a 60m op timeout. Shrinking the inner budgets rather than
   raising the ceiling also keeps this suite closer to its postgres and mariadb
   siblings on a 180-minute job cap.

2. The strategy Pod's securityContext comment still justified root with "Pod
   runs apk add at startup", which the previous commit made conditional — the
   default path now runs the platform's pre-baked image and installs nothing.
   Scoped the note to the alpine fallback and said explicitly that the field set
   matches cozy-default-altinity, runAsNonRoot deliberately unset in both.

3. clickhouse-backup.yaml still referenced clickhouse-backup-{1-contracts,
   2-roundtrip}: the file moved unchanged, so the Test rename never reached its
   own header and a reader following either name found nothing.

4. wait_sts_ready's timeout diagnostic could not print anything. It selected
   clickhouse.altinity.com/chi=${name#chi-}, which yields
   clickhouse-<app>-clickhouse-0-0 while the label carries the CHI name
   (clickhouse-<app>), and 2>/dev/null then swallowed kubectl's "No resources
   found". Ask for the StatefulSet's own first replica by name instead and let
   both streams through; verified it now surfaces the Pod events that explain
   why the replica never came up.

Also dropped the clickhouse bats case added in the previous commit: it asserted
what the generic examples/backups/<app> -> <app> mapping already guarantees, so
it did not lock the invariant its comment claimed. That invariant is now
recorded where it belongs — in the mapping's own comment in select-e2e.sh.

The dedicated ch-backup-client image this example now inherits is tracked in
#3420; noted there that the command: override is load-bearing in two files.

Assisted-By: Claude <[email protected]>
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Signed-off-by: Andrey Kolkov <[email protected]>

Recommendations from the third review pass (which returned LGTM; none of these
blocked, all three are cheap and make the failure paths honest).

1. The round-trip's catch named nothing. An unnamed `describe HelmRelease` in
   tenant-root dumps every release on the root tenant of a live cluster and
   buries the two that matter, so name them (mariadb-2-backup-roundtrip names
   its release for the same reason). Raised podLogs tail 100 -> 200, since the
   sidecar logs every action and an S3 error sits well back from the tail by the
   time a step fails, and added the Keeper Pods: chkeeper.yaml labels them
   `app: <release>-keeper`, so the existing app.kubernetes.io/instance selectors
   missed them entirely — and a source ClickHouse that never reaches a ready
   replica is usually Keeper failing to form a quorum, which explains itself
   only there.

2. SSL_CERT_DIR is now `/etc/ssl/cozy-s3-ca:/etc/ssl/certs`. The single-directory
   form works — Go reads the default bundle FILE regardless, which is what the
   earlier measurement showed — but that left the public-CA half of the contract
   resting on a file this image happens to ship. Naming the system directory
   costs nothing and survives a base-image bump that moves the bundle; verified
   against public AWS S3 that both forms still verify (403 auth, no x509). The
   mount path stays the bare directory, and the unit test now pins both halves
   separately — it caught the over-broad edit that had briefly made mountPath a
   colon list.

3. cleanup.sh returned while the uninstalls it triggered were still running, so
   the next suite (etcd) could start with helm-controller still draining this
   one, and the pre-clean could hand step 04 an application whose predecessor
   was still going away. It now waits for both ClickHouse releases and the
   Bucket to actually disappear, and — since the script deliberately runs
   without `set -e` so one absent resource cannot abandon the rest — records
   those failures and exits non-zero, rather than reporting a teardown that did
   not settle as success. wait_deleted polls `get` instead of using
   `kubectl wait --for=delete`, which errors on an already-absent resource: the
   normal case here, and indistinguishable from a real failure.

Sized the two cleanup step budgets to 12m for the same reason the run step is
60m: cleanup.sh's waits now sum to 540s, which the previous 8m did not clear.

The Keeper-sized-from-the-server-resources bug the review found outside this
diff is filed as #3487.

Assisted-By: Claude <[email protected]>
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Signed-off-by: Andrey Kolkov <[email protected]>