Skip to content

[etcd] Add VPA for etcd - #1489

Merged
Timofei Larkin (lllamnyp) merged 1 commit into
mainfrom
feat/vpa-for-etcd
Oct 6, 2025
Merged

Timofei Larkin (lllamnyp) merged 1 commit into
mainfrom
feat/vpa-for-etcd

Conversation

@lllamnyp

@lllamnyp Timofei Larkin (lllamnyp) commented Oct 6, 2025 •

Copy link
Copy Markdown
Member

What this PR does

The etcd tenant module deploys by default with a large resource limit/request and these values are not exposed at deploy time. This patch lowers the default resources and adds a VPA to autoconfigure them according to the real needs.

Release note

[etcd] Attach VPA to etcd and lower initial default resource requests.

Summary by CodeRabbit

  • New Features

    • Enabled automatic resource autoscaling for etcd with a Vertical Pod Autoscaler (VPA).
  • Chores

    • Updated default etcd resource requests to CPU 1000m and memory 512Mi (previously 4 and 1Gi), reflected across chart values and API schema.
    • Changed the output location for generated CRDs.
  • Documentation

    • Revised README to document the new default CPU and memory values for etcd.

@coderabbitai

coderabbitai Bot commented Oct 6, 2025 •

Copy link
Copy Markdown
Contributor

Note

Other AI code review bot(s) detected

CodeRabbit has detected other AI code review bot(s) in this pull request and will avoid duplicating their findings in the review comments. This may lead to a less comprehensive review.

Walkthrough

Adds a VerticalPodAutoscaler for etcd, updates etcd default CPU/memory values across values, schema, README, and generated OpenAPI schema, and changes the CRD generation output path in a helper script.

Changes

Cohort / File(s) Summary
Etcd VPA manifest
packages/extra/etcd/templates/vpa.yaml
Introduces a VerticalPodAutoscaler targeting the etcd StatefulSet with Auto updatePolicy and resourcePolicy (minAllowed: CPU 250m, Mem 256Mi; maxAllowed: CPU 5000m, Mem 8Gi).
Etcd resource defaults update
packages/extra/etcd/values.yaml, packages/extra/etcd/values.schema.json, packages/extra/etcd/README.md, packages/system/cozystack-api/cozyrds/etcd.yaml
Changes default CPU from 4 to 1000m and memory from 1Gi to 512Mi in values, schema defaults, README table, and generated OpenAPI schema. No validation/structure changes.
CRD generation path
hack/update-crd.sh
Updates CRD output directory from ../../system/cozystack-api/templates/cozystack-resource-definitions to ../../system/cozystack-api/cozyrds.

Sequence Diagram(s)

sequenceDiagram
  autonumber
  participant U as User
  participant H as Helm Chart
  participant K as Kubernetes API
  participant SS as etcd StatefulSet
  participant V as VPA (Recommender/Updater)

  U->>H: Install/Upgrade etcd chart
  H->>K: Apply StatefulSet + VPA manifests
  K-->>SS: Create/Update etcd Pods
  K-->>V: Register VPA targeting etcd Pods

  rect rgba(230,245,255,0.6)
  note over V: New/Changed interaction
  V->>K: Recommend resource requests
  V->>SS: Update Pod resource requests (policy=Auto)
  end

  alt Resource change needed
    SS->>K: Rolling update Pods with new requests
  else No change
    SS-->>U: Pods unchanged
  end
Loading

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~10 minutes

Suggested labels

ok-to-test

Suggested reviewers

  • kvaps
  • klinch0

Poem

I hop through YAML fields at night,
Tuning CPU to m’s delight,
A VPA hums, “I’ve got your back,”
While pods adjust their snacky stack.
CRDs waltz to a new abode—
Hippity-hop, we ship this load! 🐇✨

Pre-merge checks and finishing touches

✅ Passed checks (3 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title Check ✅ Passed The title directly communicates the primary purpose of the pull request—adding a Vertical Pod Autoscaler for etcd—using concise, clear language without extraneous details. It accurately reflects the main change in the diff and omits unnecessary noise. As titles need not cover every minor adjustment, it sufficiently informs reviewers about the key update.
Docstring Coverage ✅ Passed No functions found in the changes. Docstring coverage check skipped.
✨ Finishing touches
  • 📝 Generate docstrings
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Post copyable unit tests in a comment
  • Commit unit tests in branch feat/vpa-for-etcd

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@gemini-code-assist

Copy link
Copy Markdown
Contributor

Summary of Changes

Hello Timofei Larkin (@lllamnyp), I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly enhances the resource management of the etcd tenant module by integrating a Vertical Pod Autoscaler (VPA) and optimizing its default resource requests. The VPA will dynamically adjust etcd's CPU and memory allocations, preventing over-provisioning while ensuring performance. Concurrently, the initial default resource requests have been lowered to provide a more efficient baseline deployment, addressing the issue of fixed, high resource consumption and promoting better resource utilization.

Highlights

  • Vertical Pod Autoscaler (VPA) Integration: Introduced a Vertical Pod Autoscaler (VPA) for the etcd StatefulSet, enabling automatic adjustment of resource requests and limits based on actual usage patterns within defined minimum and maximum bounds.
  • Optimized Default Resource Requests: Reduced the default CPU request for etcd from 4 cores to 1 core (1000m) and memory from 1Gi to 512Mi in the values.yaml file, aiming for more efficient initial resource allocation and preventing over-provisioning.
Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature Command Description
Code Review /gemini review Performs a code review for the current pull request in its current state.
Pull Request Summary /gemini summary Provides a summary of the current pull request in its current state.
Comment Gemini (@gemini-code-assist) Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help /gemini help Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on Gemini (@gemini-code-assist) comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

  1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a Vertical Pod Autoscaler (VPA) for etcd to automatically manage its resources, and lowers the default resource requests. While adding a VPA is a good step towards optimizing resource usage, I've found a critical issue with the VPA configuration where the maximum allowed CPU is set lower than the new default request. I've also raised a concern about using the Auto update mode for a critical component like etcd due to potential availability risks during pod evictions.

Comment thread packages/extra/etcd/templates/vpa.yaml Outdated
cpu: 25m
memory: 256Mi
maxAllowed:
cpu: 500m

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

critical

The maxAllowed CPU is set to 500m, but the default CPU request in packages/extra/etcd/values.yaml is being set to 1000m. This means the VPA will cap the CPU at a value lower than the default request, which is likely not the intention. The maxAllowed value should be greater than or equal to the default request to allow for proper scaling. I'd recommend increasing maxAllowed.cpu to a more reasonable value, for example 4 (which was the previous default request) to give it room to scale.

        cpu: "4"

Comment on lines +11 to +12
updatePolicy:
updateMode: Auto

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

The updateMode is set to Auto, which will cause VPA to evict etcd pods to apply new resource recommendations. For a critical stateful component like etcd, this can lead to leader elections and potential brief interruptions. While etcd is designed to handle this, it's a risk, especially if a Pod Disruption Budget is not in place to ensure quorum is maintained. A safer approach would be to set updateMode: "Off". This will allow VPA to generate recommendations without automatically applying them, giving you a chance to review them first. Once you are confident with the recommendations, you can apply them manually or switch to an automatic mode.

  updatePolicy:
    updateMode: "Off"

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 50b8dda and fc65ff3.

📒 Files selected for processing (2)
  • packages/extra/etcd/templates/vpa.yaml (1 hunks)
  • packages/extra/etcd/values.yaml (1 hunks)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)
  • GitHub Check: Build

Comment thread packages/extra/etcd/templates/vpa.yaml Outdated
Comment on lines +1 to +21
---
apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
name: etcd
spec:
targetRef:
apiVersion: apps/v1
kind: StatefulSet
name: etcd
updatePolicy:
updateMode: Auto
resourcePolicy:
containerPolicies:
- containerName: etcd
minAllowed:
cpu: 25m
memory: 256Mi
maxAllowed:
cpu: 500m
memory: 8Gi

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🔴 Critical

Template the VPA names to match the StatefulSet

Both metadata.name and spec.targetRef.name are hard-coded to etcd, but the charted StatefulSet uses the templated fullname. On any release where the generated name differs, this VPA either conflicts with other installs or never binds to the intended target. Reuse the same helper the StatefulSet employs, e.g.:

-  name: etcd
+  name: {{ include "etcd.fullname" . }}
…
-    name: etcd
+    name: {{ include "etcd.fullname" . }}

This keeps the VPA aligned with the StatefulSet across releases.

📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
---
apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
name: etcd
spec:
targetRef:
apiVersion: apps/v1
kind: StatefulSet
name: etcd
updatePolicy:
updateMode: Auto
resourcePolicy:
containerPolicies:
- containerName: etcd
minAllowed:
cpu: 25m
memory: 256Mi
maxAllowed:
cpu: 500m
memory: 8Gi
---
apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
name: {{ include "etcd.fullname" . }}
spec:
targetRef:
apiVersion: apps/v1
kind: StatefulSet
name: {{ include "etcd.fullname" . }}
updatePolicy:
updateMode: Auto
resourcePolicy:
containerPolicies:
- containerName: etcd
minAllowed:
cpu: 25m
memory: 256Mi
maxAllowed:
cpu: 500m
memory: 8Gi
🤖 Prompt for AI Agents
In packages/extra/etcd/templates/vpa.yaml around lines 1 to 21, both
metadata.name and spec.targetRef.name are hard-coded to "etcd" but must use the
chart's fullname template used by the StatefulSet; change both fields to
reference the same fullname helper (the chart helper the StatefulSet uses) so
the VPA name and the targetRef.name are generated identically to the StatefulSet
across releases, ensuring no conflicts and correct binding.

Comment on lines +15 to +16
cpu: 1000m
memory: 512Mi

@coderabbitai coderabbitai Bot Oct 6, 2025 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

Prevent the VPA from clamping baseline CPU

The new default request is 1000m, but the VPA caps recommendations at 500m. As soon as VPA applies, it will evict the pods and drop them to 500m, halving the CPU headroom that was just configured. Raise the maxAllowed (or lower the default) so the baseline is inside the permitted range and the VPA retains freedom to scale up rather than down.

🤖 Prompt for AI Agents
In packages/extra/etcd/values.yaml around lines 15-16, the container CPU request
is set to 1000m which is above the VPA's maxAllowed (500m), causing VPA to clamp
recommendations down and trigger evictions; fix by making the baseline request
fall within the VPA permitted range or expanding the VPA cap — either lower the
default cpu request to ≤500m (e.g., 500m) or update the VPA configuration to
raise maxAllowed.cpu to at least 1000m so the VPA can scale up instead of
forcing pods down.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in the VPA config.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# Check the VPA configuration to verify the fix
cat packages/extra/etcd/templates/vpa.yaml

Length of output: 657


Timofei Larkin (@lllamnyp) Perfect! ✅ I can confirm the fix is in place. The VPA maxAllowed CPU is now set to 5000m, which comfortably accommodates the 1000m baseline request. This gives the VPA room to scale from 1000m up to 5000m based on actual usage, rather than being forced to clamp down immediately upon applying recommendations.

The configuration looks good:

  • minAllowed: 250m CPU / 256Mi memory
  • Default request: 1000m CPU / 512Mi memory
  • maxAllowed: 5000m CPU / 8Gi memory

If you found this review helpful, would you consider giving us a shout-out on X?

Thank you for using CodeRabbit!

@lllamnyp
Timofei Larkin (lllamnyp) force-pushed the feat/vpa-for-etcd branch 2 times, most recently from 35b5c40 to 2c18809 Compare October 6, 2025 18:39
The etcd tenant module deploys by default with a large resource
limit/request and these values are not exposed at deploy time. This
patch lowers the default resources and adds a VPA to autoconfigure them
according to the real needs.

```release-note
[etcd] Attach VPA to etcd and lower initial default resource requests.
```

Signed-off-by: Timofei Larkin <[email protected]>

@kvaps Andrei Kvapil (kvaps) left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

♻️ Duplicate comments (1)
packages/extra/etcd/templates/vpa.yaml (1)

5-10: Use the chart fullname for VPA binding.

Both metadata.name and spec.targetRef.name are hard-coded to etcd, so the VPA misses the StatefulSet whenever the chart renders a different fullname (e.g., any non-default release). Switch both fields to the same fullname helper the StatefulSet uses to avoid mis-binding.

📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 2c18809 and 42c9d65.

📒 Files selected for processing (6)
  • hack/update-crd.sh (1 hunks)
  • packages/extra/etcd/README.md (1 hunks)
  • packages/extra/etcd/templates/vpa.yaml (1 hunks)
  • packages/extra/etcd/values.schema.json (2 hunks)
  • packages/extra/etcd/values.yaml (1 hunks)
  • packages/system/cozystack-api/cozyrds/etcd.yaml (1 hunks)
🚧 Files skipped from review as they are similar to previous changes (1)
  • packages/extra/etcd/README.md
🧰 Additional context used
🪛 YAMLlint (1.37.1)
packages/extra/etcd/templates/vpa.yaml

[error] 17-17: syntax error: could not find expected ':'

(syntax)

Comment on lines +11 to +12
updatePolicy:
updateMode: Auto

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

Avoid automatic VPA evictions for etcd.

updateMode: Auto lets VPA evict etcd pods to apply new requests, risking avoidable leader churn and quorum loss if a PDB doesn’t stop back-to-back evictions. Default to "Off" (or at least "Initial") so you can inspect recommendations and roll them out deliberately.

🤖 Prompt for AI Agents
packages/extra/etcd/templates/vpa.yaml around lines 11 to 12: the VPA is
configured with updateMode: Auto which allows automated evictions of etcd pods;
change updateMode to Off (or at minimum Initial) so VPA only reports resource
recommendations and does not evict pods automatically, allowing manual review
and controlled rollout to avoid leader churn and quorum loss.

@lllamnyp
Timofei Larkin (lllamnyp) merged commit 1ee3d00 into main Oct 6, 2025
19 checks passed
@lllamnyp
Timofei Larkin (lllamnyp) deleted the feat/vpa-for-etcd branch October 6, 2025 19:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants