[etcd] Add VPA for etcd - #1489
Conversation
|
Note Other AI code review bot(s) detectedCodeRabbit has detected other AI code review bot(s) in this pull request and will avoid duplicating their findings in the review comments. This may lead to a less comprehensive review. WalkthroughAdds a VerticalPodAutoscaler for etcd, updates etcd default CPU/memory values across values, schema, README, and generated OpenAPI schema, and changes the CRD generation output path in a helper script. Changes
Sequence Diagram(s)sequenceDiagram
autonumber
participant U as User
participant H as Helm Chart
participant K as Kubernetes API
participant SS as etcd StatefulSet
participant V as VPA (Recommender/Updater)
U->>H: Install/Upgrade etcd chart
H->>K: Apply StatefulSet + VPA manifests
K-->>SS: Create/Update etcd Pods
K-->>V: Register VPA targeting etcd Pods
rect rgba(230,245,255,0.6)
note over V: New/Changed interaction
V->>K: Recommend resource requests
V->>SS: Update Pod resource requests (policy=Auto)
end
alt Resource change needed
SS->>K: Rolling update Pods with new requests
else No change
SS-->>U: Pods unchanged
end
Estimated code review effort🎯 2 (Simple) | ⏱️ ~10 minutes Suggested labels
Suggested reviewers
Poem
Pre-merge checks and finishing touches✅ Passed checks (3 passed)
✨ Finishing touches
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Summary of ChangesHello Timofei Larkin (@lllamnyp), I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request significantly enhances the resource management of the etcd tenant module by integrating a Vertical Pod Autoscaler (VPA) and optimizing its default resource requests. The VPA will dynamically adjust etcd's CPU and memory allocations, preventing over-provisioning while ensuring performance. Concurrently, the initial default resource requests have been lowered to provide a more efficient baseline deployment, addressing the issue of fixed, high resource consumption and promoting better resource utilization. Highlights
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on Gemini (@gemini-code-assist) comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here. You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension. Footnotes
|
There was a problem hiding this comment.
Code Review
This pull request introduces a Vertical Pod Autoscaler (VPA) for etcd to automatically manage its resources, and lowers the default resource requests. While adding a VPA is a good step towards optimizing resource usage, I've found a critical issue with the VPA configuration where the maximum allowed CPU is set lower than the new default request. I've also raised a concern about using the Auto update mode for a critical component like etcd due to potential availability risks during pod evictions.
| cpu: 25m | ||
| memory: 256Mi | ||
| maxAllowed: | ||
| cpu: 500m |
There was a problem hiding this comment.
The maxAllowed CPU is set to 500m, but the default CPU request in packages/extra/etcd/values.yaml is being set to 1000m. This means the VPA will cap the CPU at a value lower than the default request, which is likely not the intention. The maxAllowed value should be greater than or equal to the default request to allow for proper scaling. I'd recommend increasing maxAllowed.cpu to a more reasonable value, for example 4 (which was the previous default request) to give it room to scale.
cpu: "4"| updatePolicy: | ||
| updateMode: Auto |
There was a problem hiding this comment.
The updateMode is set to Auto, which will cause VPA to evict etcd pods to apply new resource recommendations. For a critical stateful component like etcd, this can lead to leader elections and potential brief interruptions. While etcd is designed to handle this, it's a risk, especially if a Pod Disruption Budget is not in place to ensure quorum is maintained. A safer approach would be to set updateMode: "Off". This will allow VPA to generate recommendations without automatically applying them, giving you a chance to review them first. Once you are confident with the recommendations, you can apply them manually or switch to an automatic mode.
updatePolicy:
updateMode: "Off"There was a problem hiding this comment.
Actionable comments posted: 2
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (2)
packages/extra/etcd/templates/vpa.yaml(1 hunks)packages/extra/etcd/values.yaml(1 hunks)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)
- GitHub Check: Build
| --- | ||
| apiVersion: autoscaling.k8s.io/v1 | ||
| kind: VerticalPodAutoscaler | ||
| metadata: | ||
| name: etcd | ||
| spec: | ||
| targetRef: | ||
| apiVersion: apps/v1 | ||
| kind: StatefulSet | ||
| name: etcd | ||
| updatePolicy: | ||
| updateMode: Auto | ||
| resourcePolicy: | ||
| containerPolicies: | ||
| - containerName: etcd | ||
| minAllowed: | ||
| cpu: 25m | ||
| memory: 256Mi | ||
| maxAllowed: | ||
| cpu: 500m | ||
| memory: 8Gi |
There was a problem hiding this comment.
Template the VPA names to match the StatefulSet
Both metadata.name and spec.targetRef.name are hard-coded to etcd, but the charted StatefulSet uses the templated fullname. On any release where the generated name differs, this VPA either conflicts with other installs or never binds to the intended target. Reuse the same helper the StatefulSet employs, e.g.:
- name: etcd
+ name: {{ include "etcd.fullname" . }}
…
- name: etcd
+ name: {{ include "etcd.fullname" . }}This keeps the VPA aligned with the StatefulSet across releases.
📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| --- | |
| apiVersion: autoscaling.k8s.io/v1 | |
| kind: VerticalPodAutoscaler | |
| metadata: | |
| name: etcd | |
| spec: | |
| targetRef: | |
| apiVersion: apps/v1 | |
| kind: StatefulSet | |
| name: etcd | |
| updatePolicy: | |
| updateMode: Auto | |
| resourcePolicy: | |
| containerPolicies: | |
| - containerName: etcd | |
| minAllowed: | |
| cpu: 25m | |
| memory: 256Mi | |
| maxAllowed: | |
| cpu: 500m | |
| memory: 8Gi | |
| --- | |
| apiVersion: autoscaling.k8s.io/v1 | |
| kind: VerticalPodAutoscaler | |
| metadata: | |
| name: {{ include "etcd.fullname" . }} | |
| spec: | |
| targetRef: | |
| apiVersion: apps/v1 | |
| kind: StatefulSet | |
| name: {{ include "etcd.fullname" . }} | |
| updatePolicy: | |
| updateMode: Auto | |
| resourcePolicy: | |
| containerPolicies: | |
| - containerName: etcd | |
| minAllowed: | |
| cpu: 25m | |
| memory: 256Mi | |
| maxAllowed: | |
| cpu: 500m | |
| memory: 8Gi |
🤖 Prompt for AI Agents
In packages/extra/etcd/templates/vpa.yaml around lines 1 to 21, both
metadata.name and spec.targetRef.name are hard-coded to "etcd" but must use the
chart's fullname template used by the StatefulSet; change both fields to
reference the same fullname helper (the chart helper the StatefulSet uses) so
the VPA name and the targetRef.name are generated identically to the StatefulSet
across releases, ensuring no conflicts and correct binding.
| cpu: 1000m | ||
| memory: 512Mi |
There was a problem hiding this comment.
Prevent the VPA from clamping baseline CPU
The new default request is 1000m, but the VPA caps recommendations at 500m. As soon as VPA applies, it will evict the pods and drop them to 500m, halving the CPU headroom that was just configured. Raise the maxAllowed (or lower the default) so the baseline is inside the permitted range and the VPA retains freedom to scale up rather than down.
🤖 Prompt for AI Agents
In packages/extra/etcd/values.yaml around lines 15-16, the container CPU request
is set to 1000m which is above the VPA's maxAllowed (500m), causing VPA to clamp
recommendations down and trigger evictions; fix by making the baseline request
fall within the VPA permitted range or expanding the VPA cap — either lower the
default cpu request to ≤500m (e.g., 500m) or update the VPA configuration to
raise maxAllowed.cpu to at least 1000m so the VPA can scale up instead of
forcing pods down.
There was a problem hiding this comment.
Fixed in the VPA config.
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
# Check the VPA configuration to verify the fix
cat packages/extra/etcd/templates/vpa.yaml
Length of output: 657
Timofei Larkin (@lllamnyp) Perfect! ✅ I can confirm the fix is in place. The VPA maxAllowed CPU is now set to 5000m, which comfortably accommodates the 1000m baseline request. This gives the VPA room to scale from 1000m up to 5000m based on actual usage, rather than being forced to clamp down immediately upon applying recommendations.
The configuration looks good:
- minAllowed: 250m CPU / 256Mi memory
- Default request: 1000m CPU / 512Mi memory
- maxAllowed: 5000m CPU / 8Gi memory
If you found this review helpful, would you consider giving us a shout-out on X?
Thank you for using CodeRabbit!
35b5c40 to
2c18809
Compare
The etcd tenant module deploys by default with a large resource limit/request and these values are not exposed at deploy time. This patch lowers the default resources and adds a VPA to autoconfigure them according to the real needs. ```release-note [etcd] Attach VPA to etcd and lower initial default resource requests. ``` Signed-off-by: Timofei Larkin <[email protected]>
2c18809 to
42c9d65
Compare
There was a problem hiding this comment.
Actionable comments posted: 1
♻️ Duplicate comments (1)
packages/extra/etcd/templates/vpa.yaml (1)
5-10: Use the chart fullname for VPA binding.Both
metadata.nameandspec.targetRef.nameare hard-coded toetcd, so the VPA misses the StatefulSet whenever the chart renders a different fullname (e.g., any non-default release). Switch both fields to the same fullname helper the StatefulSet uses to avoid mis-binding.
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (6)
hack/update-crd.sh(1 hunks)packages/extra/etcd/README.md(1 hunks)packages/extra/etcd/templates/vpa.yaml(1 hunks)packages/extra/etcd/values.schema.json(2 hunks)packages/extra/etcd/values.yaml(1 hunks)packages/system/cozystack-api/cozyrds/etcd.yaml(1 hunks)
🚧 Files skipped from review as they are similar to previous changes (1)
- packages/extra/etcd/README.md
🧰 Additional context used
🪛 YAMLlint (1.37.1)
packages/extra/etcd/templates/vpa.yaml
[error] 17-17: syntax error: could not find expected ':'
(syntax)
| updatePolicy: | ||
| updateMode: Auto |
There was a problem hiding this comment.
Avoid automatic VPA evictions for etcd.
updateMode: Auto lets VPA evict etcd pods to apply new requests, risking avoidable leader churn and quorum loss if a PDB doesn’t stop back-to-back evictions. Default to "Off" (or at least "Initial") so you can inspect recommendations and roll them out deliberately.
🤖 Prompt for AI Agents
packages/extra/etcd/templates/vpa.yaml around lines 11 to 12: the VPA is
configured with updateMode: Auto which allows automated evictions of etcd pods;
change updateMode to Off (or at minimum Initial) so VPA only reports resource
recommendations and does not evict pods automatically, allowing manual review
and controlled rollout to avoid leader churn and quorum loss.
What this PR does
The etcd tenant module deploys by default with a large resource limit/request and these values are not exposed at deploy time. This patch lowers the default resources and adds a VPA to autoconfigure them according to the real needs.
Release note
Summary by CodeRabbit
New Features
Chores
Documentation