feat(kubernetes): add kubelet resource reservations for worker nodes - #2420

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request addresses recurring node stability issues in tenant Kubernetes clusters caused by memory pressure. By configuring kubelet with strict resource reservations and hard cgroup enforcement, the changes ensure that system and kubelet processes are protected from being terminated by the Linux OOM killer. While this introduces a minor reduction in total allocatable pod capacity, it provides a robust mechanism for maintaining node health and graceful pod eviction.

Highlights

• Resource Reservations: Implemented explicit system and kube-reserved resource allocations to protect critical node processes.
• Hard Cgroup Enforcement: Enabled enforce-node-allocatable to prevent system processes from exceeding reserved limits, effectively stopping OOM-related kubelet crashes.
• Eviction Policies: Configured tiered memory eviction thresholds (soft and hard) with minimum reclaim guarantees to improve node stability under pressure.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

sequenceDiagram
    participant Helm as "Helm template"
    participant Lookup as "InstanceType lookup"
    participant Compute as "Reservation calculator"
    participant Renderer as "KubeadmConfigTemplate"
    participant K8s as "MachineDeployment / Cluster Autoscaler"
    Helm->>Lookup: resolve instanceType per nodeGroup
    Lookup-->>Helm: instanceType spec (cpu/memory)
    Helm->>Compute: compute effective CPU/memory, derive kubelet reservations & evictions
    Compute-->>Helm: reservations, eviction args, validations
    Helm->>Renderer: render KubeadmConfigTemplate with kubeletExtraArgs
    Renderer-->>K8s: emit MachineDeployment, capacity annotations

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

A few readability suggestions for the new templating block in packages/apps/kubernetes/templates/cluster.yaml (most impactful first):

1. Collapse the four regex validation blocks (lines 215–226). They're near-duplicates across system/kube × memory/cpu. A single range over a list of (fieldName, value, regex) tuples halves the line count and keeps the error-message format in one place.

2. Use sprig min/max for the $autoReservedMi clamp (lines 200–206) instead of two if gt branches:

{{- $autoReservedMi = min 1024 (max 64 (mulf ($memMi | float64) 0.05 | int)) }}

3. Compute $effectiveCpu alongside $effectiveMemory and reuse both in the capacity annotations. The and $instanceType $instanceType.spec $instanceType.spec.memory $instanceType.spec.memory.guest chain is currently repeated in the annotation block (lines 356–365); sprig's dig makes the initial resolution a one-liner, and the annotation block collapses to two plain if $effectiveX branches.

4. Simplify eviction validation (lines 227–241). Three branches with duplicated error-message strings can become: classify each value once ($hardIsPct := hasSuffix \"%\" ...), fail on type mismatch, then run one numeric comparison. Cuts the block roughly in half and DRYs the failure strings.

5. Minor: the $kubeletOverride.systemReservedMemory-style guards are intentional (you want to validate the user-supplied value, not the resolved default). Worth a one-line comment saying so — it's easy to misread as a bug and "fix" it into validating $systemReservedMemory.

None of these change behavior; they're pure readability/maintainability wins.

A few additional issues from a fresh pass over the branch (ones not yet raised):

Blocking

1. Missing format validation for evictionHardMemory / evictionSoftMemory. Commit 90656aa added regex validation for the four reservation fields but not the two eviction fields, so malformed values are silently accepted. Two failure modes:

• evictionHardMemory: "banana", evictionSoftMemory: "garbage" — both miss the % suffix so we fall into the absolute-units branch. cozy-lib.resources.toFloat returns 0 for unparseable strings (via sprig float64 → cast.ToFloat64), so both compute to 0, ge 0 0 is true, and the user sees evictionHardMemory (banana) must be strictly less than evictionSoftMemory (garbage) — the wrong error.

• evictionHardMemory: "banana", evictionSoftMemory: "1Gi" — 0 < 2^30, validation passes entirely, and the rendered KubeadmConfigTemplate ships memory.available<banana to kubelet. helm template succeeds; the failure surfaces only at node boot as a silent provisioning failure.

  Fix is symmetric to the reservation validation: accept ^[0-9]+(\.[0-9]+)?(%|Ki|Mi|Gi|Ti|Pi|Ei)?$ (or similar) and emit the same error shape. Tests should mirror the existing rejects invalid memory format for systemReservedMemory cases.

2. README version range is stale. packages/apps/kubernetes/README.md:15 says "Users can select the latest patch versions from 1.28 to 1.33." The Version enum in values.yaml (and the schema/types) declares v1.30–v1.35. Pre-existing drift, but the README is being actively edited in this PR — worth fixing the sentence or dropping the concrete numbers and pointing at the version field.

Non-blocking

3. No cross-field sanity check on reservation totals. A user can set systemReservedMemory: 4Gi + kubeReservedMemory: 4Gi on a 4Gi node and get zero allocatable memory. kubelet will eventually surface this, but a template-side check like "system + kube + evictionHard ≤ effectiveMemory × 0.9" would catch the footgun at helm template time. If not adding the check, a docs note putting the burden on the operator would help.

4. Reservation memory regex rejects valid Kubernetes forms. ^[0-9]+(\.[0-9]+)?(Ki|Mi|Gi|Ti|Pi|Ei)?$ rejects base-10 (1G, 500M) and exponent (1e9) quantities that resource.ParseQuantity accepts. Not common, but the error message locks users into Mi/Gi. Either broaden the regex or leave the message as-is (it's accurate to what you accept).

5. CPU regex accepts 0m and 0. Cosmetic. Could be ^([1-9][0-9]*|[0-9]+\.[0-9]+)m?$ if you want to reject zero.