fix(tenant): retry failed monitoring and gateway installs in place - #3633

What this PR does

The tenant chart renders monitoring and gateway with no install strategy, so they get Flux's default, and install remediation is an uninstall in every case Flux offers. Neither sets spec.uninstall.disableHooks, which defaults to false. So a release that missed nothing but a readiness budget gets uninstalled, and the app chart's post-delete cleanup Job runs against it.

Those Jobs delete on purpose, and they are right to. The monitoring one removes PVCs labelled apps.cozystack.io/application.name=<release>-system. That label is not incidental either: the metrics and logs claim templates stamp it so the operator copies it onto the storage PVCs, and the Job selects on it to stay release-scoped. Both ends were built to meet. The result is that a slow first install deletes the tenant's metrics and logs, and nothing reconstructs them.

The gateway Job deletes the cozystack-acme-account Secret. That key regenerates, so this one costs no data. It costs a re-registration against the ACME provider, and repeated cycles run into its rate limits, which can hold certificate issuance for the tenant down for hours. Smaller stake, same mechanism, and the two are not one class: only one of them owns storage.

RetryOnFailure keeps the manifests applied and retries the failed release, so no uninstall happens and neither Job runs. Both actions carry the strategy because the controller performs that retry as an upgrade, so with install alone the retry's own failure falls through to upgrade remediation. The remediation blocks stay: an absent release consults install, an out-of-sync one consults upgrade, both by retry count rather than by strategy.

Two things are traded away. A real upgrade failure now retries instead of rolling back, which gives up no stable safety net, since with retries: -1 the default strategy was already rolling back and re-upgrading without end. And the uninstall was the only automatic clean slate, so an install wedged on a bad resource state retries against that state until an operator removes the release. For a release that owns data that is the right exchange.

This removes the consequence rather than the trigger, and for monitoring the trigger is worth naming separately: the parent HelmRelease here waits 10m, while the child monitoring-system release sets 15m deliberately, with a comment explaining that 10m is too short for the OIDC users-Job inside its activeDeadlineSeconds: 900. The parent's wait covers the child, so the parent can fail while the child is behaving exactly as designed. That mismatch is untouched here and wants its own change.

The chart's etcd and seaweedfs releases lose storage the same way and are outside this change. Beyond those four, computeplane, ingress and info stay on the default strategy, but their charts ship no destructive post-delete hook, so remediation costs them a wasteful teardown rather than data.

Same shape as #3552 and #3581. Relates to #3624.

Screenshots

Downstream repositories

• No downstream repository is affected by this change
• cozystack/website - follow-up:
• cozystack/terraform-provider-cozystack - follow-up:
• cozystack/ansible-cozystack - follow-up:
• cozystack/ccp - follow-up:
• cozystack/talm - follow-up:
• cozystack/cozyhr - follow-up:
• cozystack/cozy-proxy - follow-up:
• cozystack/cozystack-telemetry-server - follow-up:
• cozystack/external-apps-example - follow-up:
• cozystack/examples - follow-up:

Release note

fix(tenant): retry a failed monitoring or gateway install instead of uninstalling the release, so a slow first install no longer runs the cleanup hooks that delete the tenant's metrics and logs volumes or its ACME account key

Summary by CodeRabbit

• Reliability Improvements
  • Monitoring and gateway deployments now automatically retry failed installs and upgrades every 30 seconds.
  • Existing unlimited remediation retries remain available, improving recovery from deployment issues.
• Validation
  • Added automated coverage to verify retry behavior for both monitoring and gateway deployments.