Skip to content

[Backport release-1.5] fix(cert-manager): raise cainjector memory limit to unblock caBundle injection - #3202

Merged
myasnikovdaniil merged 1 commit into
release-1.5from
backport-3199-to-release-1.5
Aug 4, 2026
Merged

myasnikovdaniil merged 1 commit into
release-1.5from
backport-3199-to-release-1.5

Conversation

@github-actions

@github-actions github-actions Bot commented Jul 4, 2026

Copy link
Copy Markdown

Description

Backport of #3199 to release-1.5.

@github-actions

github-actions Bot commented Jul 4, 2026

Copy link
Copy Markdown
Author

Please cherry-pick the changes locally and resolve any conflicts.

git fetch origin backport-3199-to-release-1.5
git worktree add --checkout .worktree/backport-3199-to-release-1.5 backport-3199-to-release-1.5
cd .worktree/backport-3199-to-release-1.5
git reset --hard HEAD^
git cherry-pick -x 223c0ef4fb97fb4a904ca75cf9478a9149481109
git push --force-with-lease

@myasnikovdaniil
myasnikovdaniil force-pushed the backport-3199-to-release-1.5 branch from 489144d to c3afc63 Compare July 8, 2026 04:39
@github-actions github-actions Bot added area/release Issues or PRs related to release tooling (changelog, backport, release pipeline) kind/bug Categorizes issue or PR as related to a bug size/M This PR changes 30-99 lines, ignoring generated files labels Jul 8, 2026
@myasnikovdaniil
myasnikovdaniil marked this pull request as ready for review July 15, 2026 09:34
@gemini-code-assist

Copy link
Copy Markdown
Contributor

Summary of Changes

Hello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request addresses a critical issue where the cert-manager cainjector component would consistently crash due to OOMKills on large clusters. By increasing the memory limit and adding a regression test, this change ensures stable CA injection and prevents admission failures caused by empty caBundles.

Highlights

  • Memory Limit Increase: Increased the cainjector memory limit to 512Mi to prevent OOMKills during cache population on large clusters.
  • Regression Testing: Added a new helm unittest to ensure the cainjector memory configuration is maintained and protected against future regressions.
  • Configuration Update: Updated values.yaml to explicitly define resource limits and requests for cert-manager components.
New Features

🧠 You can now enable Memory (public preview) to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature Command Description
Code Review /gemini review Performs a code review for the current pull request in its current state.
Pull Request Summary /gemini summary Provides a summary of the current pull request in its current state.
Comment Gemini (@gemini-code-assist) Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help /gemini help Displays a list of available commands.

Customization

To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on Gemini (@gemini-code-assist) comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Footnotes

  1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

@dosubot dosubot Bot added kind/backport Categorizes issue or PR as requiring a backport to the current release line kind/backport-previous Categorizes issue or PR as requiring a backport to the previous release line labels Jul 15, 2026

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request configures resource limits and requests for cert-manager components in values.yaml, notably increasing the cainjector memory limit to 512Mi to prevent OOMKills during startup. It also adds a test target to the Makefile and a Helm unit test to assert these resource limits. The reviewer pointed out that the PR description is missing the required release-note block and reminded the author to follow Conventional Commits and include sign-offs.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

Comment on lines +6 to +7
test:
helm unittest .

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The pull request description is missing the required release-note block. According to the repository guidelines, every PR body must contain a release note block in the following format:

```release-note
type(scope): human-readable changelog entry

Additionally, please ensure that all commits in this PR follow the Conventional Commits format and have a `Signed-off-by:` trailer.

<details>
<summary>References</summary>

1. PR body must contain a release note block in the specified format, and any PR lacking it should be flagged. <sup>([link](https://github.com/cozystack/cozystack/blob/main/.gemini/styleguide.md))</sup>
</details>

…injection

cainjector loads all ValidatingWebhookConfigurations, APIServices and CRDs
into its informer caches at startup. On a full cozystack install that working
set exceeds the 128Mi limit, so the single leader-elected cainjector Pod is
OOMKilled during cache population and CrashLoops before it injects any
caBundle. Webhook ValidatingWebhookConfigurations then keep an empty caBundle
and every admission call fails with "x509: certificate signed by unknown
authority" (failurePolicy: Fail), intermittently blocking Ingress creation
and stalling tenant installs.

Raise the cainjector memory limit to 512Mi (matching the controller) and the
request to 128Mi, and add a helm-unittest that pins the sizing so a blanket
resource change or a `make update` cannot silently lower it again.

Signed-off-by: Ivan Okhotnikov <[email protected]>
(cherry picked from commit c3afc63)
(cherry picked from commit 9f312d7a216c34724848142415d2de7c086a711e)
@myasnikovdaniil
myasnikovdaniil force-pushed the backport-3199-to-release-1.5 branch from c3afc63 to 711f09f Compare July 31, 2026 05:26
@github-actions github-actions Bot added the area/platform Issues or PRs related to platform infrastructure (bundle, flux, talos, installer) label Jul 31, 2026
myasnikovdaniil added a commit that referenced this pull request Jul 31, 2026
Backport of #3359 onto release-1.5.

The cert-manager webhook listened on 10250, the port the kubelet also
serves. When a connection to the webhook Service resolved to a node IP
rather than a Pod IP it reached the kubelet, which completed the TLS
handshake with its own node serving certificate, and the API server then
rejected every cert-manager admission call cluster-wide:

  failed calling webhook "webhook.cert-manager.io": ... x509: certificate
  is valid for srv3, not cert-manager-webhook.cozy-cert-manager.svc

The certificate is valid — it is simply the wrong server's — so the error
reads as a cert-manager fault and hides the misroute that caused it.
Moving the webhook to 10260 is upstream's own remedy for this signature.
It does not fix the misroute; it removes the collision that turns a rare
transient one into a cluster-wide outage attributed to the wrong
component.

The override lives in the package-root values file, which `make update`
leaves alone, so it survives re-vendoring.

Stacked on the #3199 backport (#3202): release-1.5 ships an EMPTY
packages/system/cert-manager/values.yaml, so every resource override in
that file arrived after v1.5.2. The bot's cherry-pick of this fix
conflicted for that reason alone — nothing to merge into — and applies
cleanly once the cainjector backport is in place.

Upgrade safety is unchanged from the original: one replica, no explicit
strategy, so maxUnavailable 0 / maxSurge 1 apply and the replacement Pod
must be Ready before the old one goes. The Service targets a NAMED port,
so during the overlap each Pod is reached on the port it actually serves
and there is no window with zero ready endpoints.

Closes #3355

(cherry picked from commit 3affec9)

Assisted-By: Claude <[email protected]>
Signed-off-by: Myasnikov Daniil <[email protected]>
@myasnikovdaniil
myasnikovdaniil merged commit 3c164d8 into release-1.5 Aug 4, 2026
8 checks passed
@myasnikovdaniil
myasnikovdaniil deleted the backport-3199-to-release-1.5 branch August 4, 2026 08:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/platform Issues or PRs related to platform infrastructure (bundle, flux, talos, installer) area/release Issues or PRs related to release tooling (changelog, backport, release pipeline) kind/backport Categorizes issue or PR as requiring a backport to the current release line kind/backport-previous Categorizes issue or PR as requiring a backport to the previous release line kind/bug Categorizes issue or PR as related to a bug size/M This PR changes 30-99 lines, ignoring generated files

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants