Skip to content

fix(cert-manager): raise cainjector memory limit to unblock caBundle injection - #3199

Merged
Aleksei Sviridkin (lexfrei) merged 1 commit into
mainfrom
fix/cert-manager-cainjector-oom
Jul 4, 2026
Merged

Aleksei Sviridkin (lexfrei) merged 1 commit into
mainfrom
fix/cert-manager-cainjector-oom

Conversation

@IvanHunters

@IvanHunters IvanHunters commented Jul 4, 2026 •

Copy link
Copy Markdown
Collaborator

What this PR does

cert-manager's cainjector loads every ValidatingWebhookConfiguration,
MutatingWebhookConfiguration, APIService and CRD into its informer caches at
startup. On a full cozystack install that working set exceeds the 128Mi memory
limit the package sets today, so the single leader-elected cainjector Pod is
OOMKilled (exit 137) during cache population and enters CrashLoopBackOff before
it ever runs a reconcile.

Because cainjector is the component that injects the issuing CA into webhook
clientConfig.caBundle (via cert-manager.io/inject-ca-from), a cainjector
that never completes a reconcile leaves those caBundles empty. With
failurePolicy: Fail, the apiserver then rejects every call to an affected
webhook with x509: certificate signed by unknown authority — even though the
cert-manager PKI chain is healthy and the webhook Pods serve a valid cert. In
practice this intermittently blocks Ingress creation in tenant namespaces
(the ingress-nginx admission webhook), which stalls dependent HelmReleases and
fails installs.

It behaves as a flake because it is a memory race: on a lighter run cainjector
fits under 128Mi, injects the caBundle within seconds and everything works; on
a heavier run (more CRDs/webhooks resident, slower leader election) peak startup
memory crosses 128Mi and it is OOMKilled before the first injection, stalling CA
injection cluster-wide for as long as it CrashLoops.

Fix: raise the cainjector memory limit to 512Mi (matching the cert-manager
controller in the same values file) and lift the request from 32Mi to 128Mi so
injection completes deterministically instead of racing the OOM floor. A
helm-unittest pins the sizing so a blanket resource change or a make update
cannot silently lower it again. The webhook and controller resources are left
unchanged — only cainjector exhibited the OOM.

Screenshots

N/A — no UI changes.

Release note

fix(cert-manager): raise cainjector memory limit so CA injection into webhook caBundles no longer fails under memory pressure (previously surfaced as intermittent "x509: certificate signed by unknown authority" admission errors)

Summary by CodeRabbit

  • New Features

    • Increased cainjector memory settings for cert-manager, helping it run more reliably during startup and cache population.
    • Added automated chart tests to verify the cainjector resource settings stay within the expected range.
  • Bug Fixes

    • Reduced the risk of cainjector being OOM-killed, improving stability and consistency of CA injection.

…injection

cainjector loads all ValidatingWebhookConfigurations, APIServices and CRDs
into its informer caches at startup. On a full cozystack install that working
set exceeds the 128Mi limit, so the single leader-elected cainjector Pod is
OOMKilled during cache population and CrashLoops before it injects any
caBundle. Webhook ValidatingWebhookConfigurations then keep an empty caBundle
and every admission call fails with "x509: certificate signed by unknown
authority" (failurePolicy: Fail), intermittently blocking Ingress creation
and stalling tenant installs.

Raise the cainjector memory limit to 512Mi (matching the controller) and the
request to 128Mi, and add a helm-unittest that pins the sizing so a blanket
resource change or a `make update` cannot silently lower it again.

Signed-off-by: Ivan Okhotnikov <[email protected]>
@IvanHunters IvanHunters added the kind/backport Categorizes issue or PR as requiring a backport to the current release line label Jul 4, 2026
@IvanHunters IvanHunters added the kind/backport Categorizes issue or PR as requiring a backport to the current release line label Jul 4, 2026
@github-actions github-actions Bot added area/uncategorized PR auto-labeler could not map title scope to a known area/*; please review kind/bug Categorizes issue or PR as related to a bug size/M This PR changes 30-99 lines, ignoring generated files labels Jul 4, 2026
@dosubot dosubot Bot added the area/platform Issues or PRs related to platform infrastructure (bundle, flux, talos, installer) label Jul 4, 2026
@coderabbitai

coderabbitai Bot commented Jul 4, 2026 •

Copy link
Copy Markdown
Contributor

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 6e4932a9-0e36-4dcb-b126-9acadfe177b7

📥 Commits

Reviewing files that changed from the base of the PR and between 10f8e2d and 223c0ef.

📒 Files selected for processing (3)
  • packages/system/cert-manager/Makefile
  • packages/system/cert-manager/tests/cainjector_resources_test.yaml
  • packages/system/cert-manager/values.yaml

📝 Walkthrough

Walkthrough

This PR raises the cert-manager cainjector memory limit from 128Mi to 512Mi and request from 32Mi to 128Mi in values.yaml, adds a helm unittest suite verifying these resource values, and introduces a Makefile test target running helm unittest ..

Changes

Cainjector Resource Sizing and Test Coverage

Layer / File(s) Summary
Memory limit/request increase
packages/system/cert-manager/values.yaml
Cainjector memory limit raised from 128Mi to 512Mi and request from 32Mi to 128Mi, with expanded comments explaining the OOM-prevention rationale.
Test suite and target for resource sizing
packages/system/cert-manager/tests/cainjector_resources_test.yaml, packages/system/cert-manager/Makefile
Adds a helm unittest suite asserting cainjector memory limit/request values against cainjector-deployment.yaml, plus a Makefile test target running helm unittest ..

Estimated code review effort: 1 (Trivial) | ~5 minutes

Possibly related PRs

  • cozystack/cozystack#2616: Also modifies packages/system/cert-manager/values.yaml cainjector memory request/limit settings.

Suggested labels: area/testing

Suggested reviewers: kvaps, lllamnyp, androndo, sircthulhu, myasnikovdaniil, lexfrei

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title accurately summarizes the main change: increasing cert-manager cainjector memory to prevent OOMs and restore CA bundle injection.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/cert-manager-cainjector-oom

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@gemini-code-assist

Copy link
Copy Markdown
Contributor

Summary of Changes

Hello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request addresses intermittent OOMKilled errors in the cert-manager cainjector component. By increasing the allocated memory resources, the cainjector can successfully populate its informer caches on larger clusters, ensuring that CA injection into webhook caBundles completes reliably and preventing subsequent admission failures.

Highlights

  • Resource Adjustment: Increased the cainjector memory limit to 512Mi and the request to 128Mi to prevent OOMKilled errors during cache population.
  • Regression Testing: Added a helm-unittest to pin the cainjector resource configuration and prevent accidental future downgrades.
  • Makefile Update: Updated the Makefile to include a test target for running helm unit tests.
New Features

🧠 You can now enable Memory (public preview) to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature Command Description
Code Review /gemini review Performs a code review for the current pull request in its current state.
Pull Request Summary /gemini summary Provides a summary of the current pull request in its current state.
Comment Gemini (@gemini-code-assist) Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help /gemini help Displays a list of available commands.

Customization

To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on Gemini (@gemini-code-assist) comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Footnotes

  1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request adds a test target to the cert-manager Makefile, introduces a Helm unit test to verify the cainjector resource limits, and increases the cainjector memory limits and requests in values.yaml to prevent OOMKills on large clusters. The reviewer suggested declaring the test target as .PHONY in the Makefile to avoid potential conflicts with files or directories named test.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.


include ../../../hack/package.mk

test:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

low

It is a best practice to declare the test target as .PHONY to prevent conflicts with any files or directories named test that might be created in the future.

.PHONY: test
test:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM — correct root-cause fix for the cluster-wide CA-injection stall; verified against the failing install signature.

cainjector's 128Mi limit is exceeded while it populates its informer caches (all ValidatingWebhookConfigurations/APIServices/CRDs) on a full install, so the single leader-elected replica is OOMKilled and CrashLoops before injecting any caBundle. With failurePolicy: Fail every affected webhook then rejects calls with "x509: certificate signed by unknown authority" — which blocks tenant Ingress creation via validate.nginx.ingress.kubernetes.io and fails the install. Raising the limit to 512Mi (matching the controller) and the request to 128Mi removes the memory race.

The pinning unit test targets the correct container (the cainjector Deployment has a single container) and asserts the 512Mi/128Mi sizing, guarding against a future make update silently lowering it.

Verified: renders correctly; CI fully green including E2E; an independent second-model pass found no regressions or actionable bugs. The 128Mi cap is not present on release branches, so no backport is required.

@lexfrei
Aleksei Sviridkin (lexfrei) merged commit 2cfcdc2 into main Jul 4, 2026
24 checks passed
@lexfrei
Aleksei Sviridkin (lexfrei) deleted the fix/cert-manager-cainjector-oom branch July 4, 2026 19:36
@github-actions

github-actions Bot commented Jul 4, 2026

Copy link
Copy Markdown

Created backport PR for release-1.5:

Please cherry-pick the changes locally and resolve any conflicts.

git fetch origin backport-3199-to-release-1.5
git worktree add --checkout .worktree/backport-3199-to-release-1.5 backport-3199-to-release-1.5
cd .worktree/backport-3199-to-release-1.5
git reset --hard HEAD^
git cherry-pick -x 223c0ef4fb97fb4a904ca75cf9478a9149481109
git push --force-with-lease

myasnikovdaniil added a commit that referenced this pull request Jul 31, 2026
Backport of #3359 onto release-1.5.

The cert-manager webhook listened on 10250, the port the kubelet also
serves. When a connection to the webhook Service resolved to a node IP
rather than a Pod IP it reached the kubelet, which completed the TLS
handshake with its own node serving certificate, and the API server then
rejected every cert-manager admission call cluster-wide:

  failed calling webhook "webhook.cert-manager.io": ... x509: certificate
  is valid for srv3, not cert-manager-webhook.cozy-cert-manager.svc

The certificate is valid — it is simply the wrong server's — so the error
reads as a cert-manager fault and hides the misroute that caused it.
Moving the webhook to 10260 is upstream's own remedy for this signature.
It does not fix the misroute; it removes the collision that turns a rare
transient one into a cluster-wide outage attributed to the wrong
component.

The override lives in the package-root values file, which `make update`
leaves alone, so it survives re-vendoring.

Stacked on the #3199 backport (#3202): release-1.5 ships an EMPTY
packages/system/cert-manager/values.yaml, so every resource override in
that file arrived after v1.5.2. The bot's cherry-pick of this fix
conflicted for that reason alone — nothing to merge into — and applies
cleanly once the cainjector backport is in place.

Upgrade safety is unchanged from the original: one replica, no explicit
strategy, so maxUnavailable 0 / maxSurge 1 apply and the replacement Pod
must be Ready before the old one goes. The Service targets a NAMED port,
so during the overlap each Pod is reached on the port it actually serves
and there is no window with zero ready endpoints.

Closes #3355

(cherry picked from commit 3affec9)

Assisted-By: Claude <[email protected]>
Signed-off-by: Myasnikov Daniil <[email protected]>
myasnikovdaniil added a commit that referenced this pull request Aug 4, 2026
…it to unblock caBundle injection (#3202)

# Description
Backport of #3199 to `release-1.5`.
Aleksei Sviridkin (lexfrei) added a commit that referenced this pull request Sep 18, 2026
## What this PR does

The barman-cloud plugin injects its sidecar with no resource requests or
limits. What that means depends on the namespace. A tenant with
`resourceQuotas` set ships a `LimitRange` that defaults containers to
128Mi (`packages/apps/tenant/templates/quota.yaml`), so there the
sidecar inherits that default and is OOMKilled mid-backup. A namespace
without a `LimitRange`, which is `cozy-keycloak` and any tenant that
leaves `resourceQuotas` empty, gave the sidecar no requests and no limit
at all, so nothing reserved memory for it and nothing bounded it. The
failure on the tenant path is quiet from the control plane: the
`ObjectStore` stays healthy, the `Cluster` stays `Ready`, and only the
backup fails.

Observed on a 1.6 cluster while backing up a 38 MB database:
`lastState.terminated.reason: OOMKilled`, `exit 137`, four restarts on
the sidecar, and a cgroup high-water mark of 254 MiB.
`container_memory_working_set_bytes` peaked at only 64 MiB over the same
window — it is sampled, and it misses the spike that actually triggers
the kill, which is why the working-set number does not explain the
failure on its own.

Both paths that build an ObjectStore now carry resources:

- `cozy-lib.barman.sidecarConfiguration`, used by
`packages/apps/postgres` for the backup and recovery ObjectStores and by
`packages/system/keycloak`;
- `barmanSidecarConfiguration()` in
`internal/backupcontroller/cnpgstrategy_controller.go`, which builds the
platform's own ObjectStore on the `useSystemBucket=true` path.

Requests are 100m/256Mi with a 1Gi memory limit on every caller: 256Mi
holds the measured working set and 1Gi is four times the measurement,
which is also the ceiling a caller without a LimitRange gets where it
had none. No CPU limit is set, following the `entityOperator` precedent
in `packages/apps/kafka/templates/kafka.yaml`: a throttled sidecar
stalls WAL archiving instead of failing it, which is harder to notice
than an outright failure. Measured CPU peak was 50 mCPU. Inside a tenant
the limit is charged against the `ResourceQuota` `limits.memory` budget,
1Gi per instance pod; a 4Gi tenant spends a quarter of it per Postgres
instance.

The helper is renamed from `checksumSidecarConfiguration` to
`sidecarConfiguration`, because it no longer carries only the checksum
pin and its name and doc comment would otherwise be wrong.

Tests cover both paths and both render sites: a new helm suite in
`packages/apps/postgres`, extra assertions in the existing keycloak
suite, an assertion in the controller test, and a deepcopy test for the
new field. Each of them fails against the unfixed sources.

One thing this PR deliberately does not do. This is the fifth time the
128Mi tenant default has been worked around per component — kafka and
zookeeper presets (#2537), the kafka entity-operator (#2934), the
cert-manager cainjector (#3199), and the note carried in the
etcd-operator values. Whether the default itself should move looks like
your call rather than something to fold into a fix for one sidecar, so
it is left alone here.

### Screenshots

Not a UI change.

### Downstream repositories

Walked the trigger map in `docs/agents/contributing.md` against the
diff, entry by entry. The change touches one cozy-lib helper, the two
chart templates that include it, one Go function, and tests. It adds,
renames or removes no package; changes no `values.yaml`,
`values.schema.json`, `Chart.yaml` or `README.md`, so no version enum,
default or generated artifact moves; changes no `ApplicationDefinition`
semantics, no `release.prefix`, no output Secret or Service name;
touches no CRD, no namespace, no `hack/` file, no telemetry metric or
label, no `cozy-proxy` annotation, no node or network requirement, and
no commit or PR convention. No entry matches.

- [x] No downstream repository is affected by this change

### Release note

```release-note
fix(backups): give the barman-cloud sidecar its own resources
The barman-cloud sidecar was injected without resource requests or limits, so in tenant namespaces it inherited the 128Mi `LimitRange` default and was OOMKilled during backups while the `ObjectStore` and the `Cluster` both stayed healthy. It now requests 100m/256Mi and caps memory at 1Gi on the chart path and on the platform's Go path alike; in namespaces without a LimitRange the sidecar had no limit before and gets the same 1Gi ceiling.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

- **New Features**
- Added explicit CPU and memory resource settings for barman-cloud
backup and recovery sidecars.
- Backup sidecars request 100m CPU and 256Mi memory, with a 1Gi memory
limit.
  - Recovery sidecars receive a 1Gi memory limit.
  - Preserved the S3 checksum configuration.

- **Bug Fixes**
- Prevented sidecars from inheriting unsuitable tenant resource limits.
- Ensured resource settings are preserved when backup configuration is
copied.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/platform Issues or PRs related to platform infrastructure (bundle, flux, talos, installer) area/uncategorized PR auto-labeler could not map title scope to a known area/*; please review kind/backport Categorizes issue or PR as requiring a backport to the current release line kind/bug Categorizes issue or PR as related to a bug size/M This PR changes 30-99 lines, ignoring generated files

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants