test(e2e): self-heal Cilium orphaned-endpoint leak during install - #2874
myasnikovdaniil wants to merge 1 commit into
Conversation
📝 WalkthroughWalkthroughIntroduces a Cilium endpoint leak healer watchdog for e2e CI that detects stale in-memory endpoints and disconnects confirmed orphans. Integrates via Kubernetes manifest with RBAC and test suite hooks. Separately, enhances SeaweedFS Helm chart with COSI lock and read-only access classes, updates S3 service naming, reduces master volume sizing, and hardens the bucket e2e test with direct port-forwarding. Adds startup probe to CAPI providers core component. ChangesCilium endpoint leak healer CI integration
SeaweedFS chart and e2e improvements
CAPI providers core startup probe
Sequence DiagramsequenceDiagram
participant Bats as e2e-install-cozystack.bats
participant ConfigMap as cilium-leak-healer ConfigMap
participant Job as Job cilium-leak-healer
participant KubeAPI as Kubernetes API
participant CiliumAgent as cilium-agent pod
Bats->>ConfigMap: create/apply from hack/e2e-cilium-endpoint-leak-healer.sh (setup_file)
Bats->>Job: Job starts, runs heal.sh from ConfigMap volume
loop Every CILIUM_HEALER_POLL_SEC
Job->>KubeAPI: list events (FailedCreatePodSandBox, "IP ipv4:X in use")
KubeAPI-->>Job: matching events with pod refs
Job->>KubeAPI: get pod phase and metadata
KubeAPI-->>Job: pod state (not Running or empty)
Job->>KubeAPI: resolve pod's node, find cilium-agent on that node
KubeAPI-->>Job: cilium-agent pod name
Job->>CiliumAgent: cilium-dbg endpoint get ipv4:X -o json
CiliumAgent-->>Job: endpoint JSON (owning k8s-namespace/k8s-pod-name)
Job->>KubeAPI: get owning pod phase and podIP
KubeAPI-->>Job: owning pod state
alt owning pod Running with matching IP
Job-->>Job: log REFUSE
else endpoint orphaned
Job->>CiliumAgent: cilium-dbg endpoint disconnect ipv4:X
CiliumAgent-->>Job: success/failure
Job-->>Job: log HEAL
end
end
Bats->>Job: teardown_file reads job/cilium-leak-healer logs
Bats->>Bats: echo HEAL/REFUSE lines to FD 3 for trace
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~22 minutes Possibly related PRs
Suggested labels
Suggested reviewers
Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Summary of ChangesHello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request implements an interim CI-only mitigation for a known Cilium in-memory endpoint leak that causes pods to get stuck in a 'ContainerCreating' state. By deploying a background watchdog during e2e tests, the system can now detect and surgically remove stale endpoints that block IP reuse, preventing unrelated test failures while waiting for an upstream fix. Highlights
New Features🧠 You can now enable Memory (public preview) to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console. Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on Gemini (@gemini-code-assist) comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here. Footnotes
|
dosubot
Bot
added
area/ci
github-actions
Bot
added
the
size/L
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces an interim CI self-heal watchdog script (hack/e2e-cilium-endpoint-leak-healer.sh) and integrates it into the E2E test suite (hack/e2e-install-cozystack.bats) to mitigate the Cilium in-memory endpoint leak issue. The watchdog identifies orphaned endpoints and evicts them safely. Feedback on the script includes setting up a trap for reliable PID file cleanup, optimizing multiple jq invocations into a single call, and using a bash here-string (<<<) to simplify the events loop.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
| MAX_RUNTIME="${CILIUM_HEALER_MAX_SEC:-2700}" # 45m: covers install + tenant + app suite | ||
| PIDFILE="${CILIUM_HEALER_PIDFILE:-/tmp/cilium-leak-healer.pid}" | ||
|
|
||
| echo $$ > "$PIDFILE" 2>/dev/null || true |
There was a problem hiding this comment.
To ensure the PID file is reliably cleaned up even when the script is terminated (e.g., by the kill command in teardown_file), it is best practice to set up a trap on EXIT, INT, and TERM.
| echo $$ > "$PIDFILE" 2>/dev/null || true | |
| echo $$ > "$PIDFILE" 2>/dev/null || true | |
| trap 'rm -f "$PIDFILE" 2>/dev/null' EXIT INT TERM |
| epid=$(printf '%s' "$ep_json" | jq -r '.[0].id // empty' 2>/dev/null) | ||
| epns=$(printf '%s' "$ep_json" | jq -r '.[0].status."external-identifiers"."k8s-namespace" // empty' 2>/dev/null) | ||
| eppod=$(printf '%s' "$ep_json" | jq -r '.[0].status."external-identifiers"."k8s-pod-name" // empty' 2>/dev/null) |
There was a problem hiding this comment.
Instead of invoking jq three separate times per event, we can optimize this by querying all three fields in a single jq call and parsing them using read with a custom delimiter. This significantly reduces process execution overhead in the polling loop.
| epid=$(printf '%s' "$ep_json" | jq -r '.[0].id // empty' 2>/dev/null) | |
| epns=$(printf '%s' "$ep_json" | jq -r '.[0].status."external-identifiers"."k8s-namespace" // empty' 2>/dev/null) | |
| eppod=$(printf '%s' "$ep_json" | jq -r '.[0].status."external-identifiers"."k8s-pod-name" // empty' 2>/dev/null) | |
| IFS='|' read -r epid epns eppod < <(printf '%s' "$ep_json" | jq -r '.[0] | [ (.id // ""), (.status."external-identifiers"."k8s-namespace" // ""), (.status."external-identifiers"."k8s-pod-name" // "") ] | join("|")' 2>/dev/null) |
| -o jsonpath='{range .items[*]}{.involvedObject.namespace}{"|"}{.involvedObject.name}{"|"}{.message}{"\n"}{end}' 2>/dev/null \ | ||
| | grep -i "is already in use" | sort -u) | ||
|
|
||
| [ -n "$events" ] && while IFS='|' read -r ns pod msg; do |
There was a problem hiding this comment.
When using a here-string (<<<), we can simplify the loop condition and remove the redundant [ -n "$events" ] check. Any empty event list will be safely skipped by the inner validation checks.
| [ -n "$events" ] && while IFS='|' read -r ns pod msg; do | |
| while IFS='|' read -r ns pod msg; do |
| done <<EOF | ||
| $events | ||
| EOF |
There was a problem hiding this comment.
Using a here-string (<<< "$events") is cleaner and more idiomatic in Bash than a multi-line here-doc (<<EOF) when feeding a variable into a loop.
| done <<EOF | |
| $events | |
| EOF | |
| done <<< "$events" |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Nitpick comments (1)
hack/e2e-cilium-endpoint-leak-healer.sh (1)
64-64: 💤 Low valueIP extraction regex doesn't validate octet ranges.
The regex
ipv4:[0-9]{1,3}(\.[0-9]{1,3}){3}matches 1-3 digits per octet without validating the [0-255] range. Input like "ipv4:999.999.999.999" would match. Since the source is Kubernetes event messages (Cilium agent errors), invalid IPs are unlikely, and cilium-dbg will reject them downstream at line 79-82, causing a silent no-op. This is safe but not technically correct.♻️ More robust IP extraction
- ip=$(printf '%s' "$msg" | grep -oE 'ipv4:[0-9]{1,3}(\.[0-9]{1,3}){3}' | head -1 | cut -d: -f2) + ip=$(printf '%s' "$msg" | grep -oP 'ipv4:\K((25[0-5]|2[0-4][0-9]|1[0-9]{2}|[1-9]?[0-9])\.){3}(25[0-5]|2[0-4][0-9]|1[0-9]{2}|[1-9]?[0-9])' | head -1)🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@hack/e2e-cilium-endpoint-leak-healer.sh` at line 64, The current IP extraction using grep -oE 'ipv4:[0-9]{1,3}(\.[0-9]{1,3}){3}' (assigned to variable ip) accepts invalid octets like 999; update the extraction to validate each octet (0-255) so only real IPv4 addresses are captured and passed to cilium-dbg. Replace the simple [0-9]{1,3} pattern with a stricter octet pattern (e.g. (25[0-5]|2[0-4][0-9]|1?[0-9]{1,2})) applied to each of the four octets in the grep/egrep call that produces ip, and ensure that if no valid match is found ip remains empty or the script skips the cilium-dbg invocation to avoid silent no-ops.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@hack/e2e-cilium-endpoint-leak-healer.sh`:
- Around line 84-86: Add an explicit availability check for the jq binary before
the main loop so the script fails fast instead of silently producing empty
epid/epns/eppod values; specifically, verify `jq` exists (e.g., via command -v
jq) after initialization and exit non‑zero with a clear error if missing,
ensuring the subsequent freshness gate that tests `[ -n "$epns" ] && [ -n
"$eppod" ]` cannot be bypassed and the disconnect path (the code that
disconnects endpoints) will never run when jq is not present.
In `@hack/e2e-install-cozystack.bats`:
- Around line 3-28: The file-scoped hooks setup_file/teardown_file won't be
executed by hack/cozytest.sh because it only runs `@test` blocks, so start/stop of
the cilium leak healer never happens; rename or duplicate that logic into Bats'
per-test hooks that cozytest executes (e.g., move the healer invocation into
setup() and the PID-kill/log-grepping into teardown()), keeping the same
invocation of hack/e2e-cilium-endpoint-leak-healer.sh and preserving use of
/tmp/cilium-leak-healer.pid and /tmp/cilium-leak-healer.log so the healer is
actually launched and its cleanup/trace output runs under cozytest.sh.
---
Nitpick comments:
In `@hack/e2e-cilium-endpoint-leak-healer.sh`:
- Line 64: The current IP extraction using grep -oE
'ipv4:[0-9]{1,3}(\.[0-9]{1,3}){3}' (assigned to variable ip) accepts invalid
octets like 999; update the extraction to validate each octet (0-255) so only
real IPv4 addresses are captured and passed to cilium-dbg. Replace the simple
[0-9]{1,3} pattern with a stricter octet pattern (e.g.
(25[0-5]|2[0-4][0-9]|1?[0-9]{1,2})) applied to each of the four octets in the
grep/egrep call that produces ip, and ensure that if no valid match is found ip
remains empty or the script skips the cilium-dbg invocation to avoid silent
no-ops.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: 4f71307f-d17e-44e1-a718-56afd13306fd
📒 Files selected for processing (2)
hack/e2e-cilium-endpoint-leak-healer.shhack/e2e-install-cozystack.bats
| epid=$(printf '%s' "$ep_json" | jq -r '.[0].id // empty' 2>/dev/null) | ||
| epns=$(printf '%s' "$ep_json" | jq -r '.[0].status."external-identifiers"."k8s-namespace" // empty' 2>/dev/null) | ||
| eppod=$(printf '%s' "$ep_json" | jq -r '.[0].status."external-identifiers"."k8s-pod-name" // empty' 2>/dev/null) |
There was a problem hiding this comment.
Missing jq dependency check bypasses critical safety gate.
Lines 84-86 use jq to extract endpoint metadata without verifying that jq is installed. If jq is unavailable, the 2>/dev/null suppression causes epid, epns, and eppod to silently become empty strings. The freshness gate at lines 91-98 then short-circuits (because [ -n "$epns" ] && [ -n "$eppod" ] evaluates to false), and the script proceeds directly to disconnect the endpoint at line 104 without confirming the endpoint is not backing a live Running pod. This bypasses the core safety check that prevents the script from disconnecting live endpoints.
🛡️ Add jq availability check at startup
After line 43, before the main loop starts:
log() { echo "[cilium-leak-healer $(date -u +%H:%M:%S)] $*"; }
+
+# Verify required tools are available
+if ! command -v jq >/dev/null 2>&1; then
+ log "FATAL: jq is not installed. The freshness gate requires jq to prevent disconnecting live endpoints."
+ exit 1
+fi
# cilium-agent pod running on a given node (DaemonSet selector k8s-app=cilium).🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@hack/e2e-cilium-endpoint-leak-healer.sh` around lines 84 - 86, Add an
explicit availability check for the jq binary before the main loop so the script
fails fast instead of silently producing empty epid/epns/eppod values;
specifically, verify `jq` exists (e.g., via command -v jq) after initialization
and exit non‑zero with a clear error if missing, ensuring the subsequent
freshness gate that tests `[ -n "$epns" ] && [ -n "$eppod" ]` cannot be bypassed
and the disconnect path (the code that disconnects endpoints) will never run
when jq is not present.
57cb3f5 to
c8f0597
Compare
… apps Run the Cilium endpoint-leak watchdog as an in-cluster Job instead of a background process started in the install bats setup_file. bats *_file hooks fire per file, so the old process was killed at the install file's teardown_file — before any app test ran — and the leak recurred unhealed in the app phase (e.g. the tenant-Kubernetes CVMI importer). The Job is bound to the cluster, not to one bats file, so it covers install and the whole app suite. The heal logic stays in hack/e2e-cilium-endpoint-leak-healer.sh (single source of truth) and is shipped to the pod via a ConfigMap built from it at apply time. The pod uses hostNetwork so it needs no Cilium endpoint and cannot be wedged by the very leak it repairs, with broad tolerations so it schedules before nodes are Ready. RBAC is scoped to read events/pods and exec into the cilium-agent. Image reuses the in-tree digest-pinned alpine/k8s already shipped by the installer. Interim CI band-aid for cilium/cilium#38313; remove with a fixed Cilium. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> Signed-off-by: Myasnikov Daniil <[email protected]>
c8f0597 to
b825c31
Compare
There was a problem hiding this comment.
Actionable comments posted: 2
♻️ Duplicate comments (1)
hack/e2e-cilium-endpoint-leak-healer.sh (1)
88-95:⚠️ Potential issue | 🔴 Critical | ⚡ Quick winFail closed if endpoint JSON parsing fails before disconnect.
If
endpoint get -o jsonreturns malformed/non-JSON output (orjqfails),epns/eppodbecome empty and the live-owner check is bypassed, but Line 112 still disconnects by IP. This can evict a live endpoint.Suggested hardening
log() { echo "[cilium-leak-healer $(date -u +%H:%M:%S)] $*"; } + +if ! command -v jq >/dev/null 2>&1; then + log "FATAL: jq is required for orphan-safety checks" + exit 1 +fi @@ - epid=$(printf '%s' "$ep_json" | jq -r '.[0].id // empty' 2>/dev/null) + if ! printf '%s' "$ep_json" | jq -e 'type=="array" and length>0 and .[0].id != null' >/dev/null 2>&1; then + log "REFUSE ip=$ip on node=$node: unexpected endpoint payload; refusing unsafe disconnect" + continue + fi + epid=$(printf '%s' "$ep_json" | jq -r '.[0].id // empty' 2>/dev/null)Also applies to: 100-113
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@hack/e2e-cilium-endpoint-leak-healer.sh` around lines 88 - 95, The issue is that if the endpoint JSON parsing fails or returns malformed data, the variables epns and eppod extracted via jq become empty, which bypasses the live-owner check but still allows the disconnect operation at line 112 to proceed based on IP alone, potentially evicting a live endpoint. Add validation after the jq extraction commands for epid, epns, and eppod to ensure they were successfully parsed; if any of these variables are empty (indicating parsing failure), log an error and skip the disconnect operation for that endpoint by using continue or similar flow control to fail closed rather than proceeding with a potentially unsafe disconnection.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@hack/e2e-cilium-leak-healer.yaml`:
- Around line 31-34: The pods/exec resource with create verb is currently
defined in a ClusterRole, which grants cluster-wide exec privileges into any pod
in any namespace. Remove the apiGroups, resources, and verbs entry for pods/exec
from the ClusterRole, then create a separate namespace-scoped Role in the
cozy-cilium namespace containing only the pods/exec:create permission. Bind this
Role to the appropriate service account within the cozy-cilium namespace using a
RoleBinding to ensure exec privileges are limited to that namespace only.
In `@hack/e2e-install-cozystack.bats`:
- Around line 18-21: The kubectl create configmap and kubectl apply commands for
the cilium leak healer in the setup_file function use `|| true` which silently
masks any deployment failures, allowing the test suite to proceed without proper
protection. Remove the `|| true` operators from both the kubectl create
configmap cilium-leak-healer command and the kubectl apply command for
hack/e2e-cilium-leak-healer.yaml so that failures in setting up the leak healer
will cause the setup to fail and alert users of the problem rather than silently
continuing with an unprotected environment.
---
Duplicate comments:
In `@hack/e2e-cilium-endpoint-leak-healer.sh`:
- Around line 88-95: The issue is that if the endpoint JSON parsing fails or
returns malformed data, the variables epns and eppod extracted via jq become
empty, which bypasses the live-owner check but still allows the disconnect
operation at line 112 to proceed based on IP alone, potentially evicting a live
endpoint. Add validation after the jq extraction commands for epid, epns, and
eppod to ensure they were successfully parsed; if any of these variables are
empty (indicating parsing failure), log an error and skip the disconnect
operation for that endpoint by using continue or similar flow control to fail
closed rather than proceeding with a potentially unsafe disconnection.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: fe4fa927-8faa-4ee3-a7c7-72654e9add67
📒 Files selected for processing (8)
hack/e2e-apps/bucket.batshack/e2e-cilium-endpoint-leak-healer.shhack/e2e-cilium-leak-healer.yamlhack/e2e-install-cozystack.batspackages/system/capi-providers-core/files/core-components.yamlpackages/system/seaweedfs/charts/seaweedfs/templates/cosi/cosi-bucket-class.yamlpackages/system/seaweedfs/charts/seaweedfs/templates/s3/s3-service.yamlpackages/system/seaweedfs/values.yaml
| # run `cilium-dbg endpoint ...` inside the owning node's cilium-agent | ||
| - apiGroups: [""] | ||
| resources: ["pods/exec"] | ||
| verbs: ["create"] |
There was a problem hiding this comment.
Scope pods/exec to the Cilium namespace instead of cluster-wide.
The healer only execs into cilium-agent pods, but current ClusterRole permits exec into any pod in any namespace. Move pods/exec:create into a namespace-scoped Role (bound in cozy-cilium) to reduce privilege blast radius.
🧰 Tools
🪛 Trivy (0.69.3)
[error] 31-33: Exec into Pods
ClusterRole 'cilium-leak-healer' should not have access to resource '["pods/exec"]' for verbs ["create", "update", "patch", "delete", "deletecollection", "impersonate", "*"]
Rule: KSV-0053
(IaC/Kubernetes)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@hack/e2e-cilium-leak-healer.yaml` around lines 31 - 34, The pods/exec
resource with create verb is currently defined in a ClusterRole, which grants
cluster-wide exec privileges into any pod in any namespace. Remove the
apiGroups, resources, and verbs entry for pods/exec from the ClusterRole, then
create a separate namespace-scoped Role in the cozy-cilium namespace containing
only the pods/exec:create permission. Bind this Role to the appropriate service
account within the cozy-cilium namespace using a RoleBinding to ensure exec
privileges are limited to that namespace only.
Source: Linters/SAST tools
| kubectl create configmap cilium-leak-healer -n kube-system \ | ||
| --from-file=heal.sh=hack/e2e-cilium-endpoint-leak-healer.sh \ | ||
| --dry-run=client -o yaml | kubectl apply -f - || true | ||
| kubectl apply -f hack/e2e-cilium-leak-healer.yaml || true |
There was a problem hiding this comment.
Do not swallow healer setup failures in setup_file.
|| true here can silently skip ConfigMap/Job deployment, leaving the suite unprotected while still appearing healthy.
Suggested change
- kubectl create configmap cilium-leak-healer -n kube-system \
- --from-file=heal.sh=hack/e2e-cilium-endpoint-leak-healer.sh \
- --dry-run=client -o yaml | kubectl apply -f - || true
- kubectl apply -f hack/e2e-cilium-leak-healer.yaml || true
+ kubectl create configmap cilium-leak-healer -n kube-system \
+ --from-file=heal.sh=hack/e2e-cilium-endpoint-leak-healer.sh \
+ --dry-run=client -o yaml | kubectl apply -f -
+ kubectl apply -f hack/e2e-cilium-leak-healer.yaml📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| kubectl create configmap cilium-leak-healer -n kube-system \ | |
| --from-file=heal.sh=hack/e2e-cilium-endpoint-leak-healer.sh \ | |
| --dry-run=client -o yaml | kubectl apply -f - || true | |
| kubectl apply -f hack/e2e-cilium-leak-healer.yaml || true | |
| kubectl create configmap cilium-leak-healer -n kube-system \ | |
| --from-file=heal.sh=hack/e2e-cilium-endpoint-leak-healer.sh \ | |
| --dry-run=client -o yaml | kubectl apply -f - | |
| kubectl apply -f hack/e2e-cilium-leak-healer.yaml |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@hack/e2e-install-cozystack.bats` around lines 18 - 21, The kubectl create
configmap and kubectl apply commands for the cilium leak healer in the
setup_file function use `|| true` which silently masks any deployment failures,
allowing the test suite to proceed without proper protection. Remove the `||
true` operators from both the kubectl create configmap cilium-leak-healer
command and the kubectl apply command for hack/e2e-cilium-leak-healer.yaml so
that failures in setting up the leak healer will cause the setup to fail and
alert users of the problem rather than silently continuing with an unprotected
environment.
## What Consolidated landing branch for a set of interdependent platform/CI stabilization fixes. It began as the source PRs in the table below (in dependency order) and has since grown with review-driven fixes and a few production-behavior changes — the full current inventory is under **Scope** below. Landing them as one unit lets CI run against the real combined state instead of a fragile bottom-up merge train. ## Source PRs (dependency order) | # | Commit | Source PR | Addresses | |---|--------|-----------|-----------| | 1 | fix(capi): startupProbe on capi-controller-manager | #2946 | capi crashloop during cert provisioning | | 2 | test(e2e): bucket.bats port-forward + S3 client reliability | #2944 | flaky bucket test | | 3 | fix(seaweedfs): restore -lock BucketClass, s3 svc name, drop volumeSizeLimitMB | #2943 | missing bucket/access classes | | 4 | test(e2e): in-cluster Cilium endpoint-leak healer (install + apps) | #2874 | cilium "IP already in use" leak | | 5 | refactor(build): mode=max registry cache + main-only warmer | #2938 | build cache | | 6 | ci(build): isolate each PR build on its own ephemeral runner VM | #2939 | shared-buildkit contention | | 7 | fix(e2e): LINSTOR post-install waits on a single 15m deadline | #2928 | LINSTOR provisioning timeouts | | 8 | test(metallb): assert digest-pinned image form, not version literal | #2873 | brittle metallb assertion | ## Why consolidated These form a dependency DAG (verified from CI logs): e.g. 2943 needs 2946, 2938 needs 2874, 2928 needs 2938/2939, 2873 needs 2928. Tested bottom-up, each lower PR runs with none of the fixes above it and can't go green alone. This branch carries all of them, so CI runs against the real combined state. ## Verification status CI is green on the latest head (`5653c30`): full-suite E2E passes end-to-end and is reproducible (2 of 3 runs on this SHA green). The one red run was a LINSTOR tie-breaker / DRBD-metadata infra flake on the sandbox, unrelated to the diff — a different environmental subsystem failed each run (details in the comments below). The earlier 3-hour crust-gather snapshot hang is fixed in `5653c30`. ## Scope beyond the original 8 The branch has grown past the 8 source PRs above with review-driven fixes and a few production-behavior changes. For reviewer transparency, the full set: **Review fixes (@lexfrei review):** - **B1 (blocker)** — `8f41910` converts the SeaweedFS `-lock`/`-readonly` BucketClasses and the s3 service-name override into `patches/`, wired into `make update`, so a re-vendor no longer drops them. - **FU1** — `cc99230` grants the cilium leak-healer `delete` on pods (both delete remedies were RBAC-forbidden). - **FU3** — `014de3f` adds a unit test for the HelmRelease update conflict-retry path. - **FU4** — `6deddc6` makes the leak-healer refuse a disconnect for any non-terminal owner phase, not just `Running`. - **FU7** — `f3914d2` sets `persist-credentials: false` on the build-cache checkout (SHA-pinning is handled repo-wide in #2849). **Production-behavior changes (not in the table above):** - VPA `updateMode: Auto → Initial` for etcd (`6e9ff90`) and monitoring (`0997105`), plus `vmselect`/`vmstorage` `minAllowed` floors — stops install-time eviction churn. - cozystack-api: `retry.RetryOnConflict` on the Application→HelmRelease update path (`728f676`). **E2E robustness follow-ups:** per-test crust-gather snapshots (`50e5f94`), EtcdBackupSchedule wait (`763b85e`), tenant-node single-deadline wait (`9e43d45`), harbor BucketClaim 10m budget (`d7f244f`), tenant API via LoadBalancer (`9beda4e`), crust-gather pin (`95f42aa`). **Not addressed (by decision):** FU2 — the s3 Service keeps `seaweedfs.name`-based naming via patch (rendered output unchanged; cozystack renders `fullname == seaweedfs`, so it matches the `componentName`/`fullname` siblings in practice). FU6 — fork PR build push is handled by the existing mirror-to-same-repo-branch workflow. ## Relates to #2946 #2944 #2943 #2874 #2938 #2939 #2928 #2873 — if this lands, those can be closed; otherwise they remain the granular per-PR review path. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added CI build cache warming for main branch builds. * Added a best-effort in-cluster Cilium endpoint self-heal watchdog for e2e installs. * Extended SeaweedFS COSI with object-lock support plus separate readonly access. * **Improvements** * Reduced e2e install flakiness with a shared readiness deadline, safer waits, and improved cleanup/diagnostics capture. * Prevented install-time churn by switching VPA update mode to **Initial**. * Added container startup probes and updated SeaweedFS volume sizing. * **Tests** * Improved e2e robustness (etcd backup schedule waiting, digest-pinned image checks, and corrected S3 port-forwarding). <!-- end of auto-generated comment: release notes by coderabbit.ai -->
… apps (cozystack#2874) Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> Signed-off-by: Myasnikov Daniil <[email protected]>
|
superseded by #2948 |
1 participant
What this PR does
Adds an interim e2e self-heal for a Cilium in-memory endpoint leak that intermittently fails PRs unrelated to the change under test.
Symptom. On a churn-heavy platform install a pod sticks in
ContainerCreating/Initbecause cilium-agent rejects its sandbox with[PUT /endpoint/{id}][400] putEndpointIdInvalid "IP ipv4:<X> is already in use", repeated until the agent restarts or the install times out. The disputed IP has no live owner — IPAM and the ipcache released it — but the agent's in-memoryendpointManagerstill holds a staleEndpointfrom a previously-deleted pod whose CNIDELwas skipped or raced. Neither the operator's CiliumEndpoint-CRD GC nor the agent's veth-based endpoint GC reaps it. This surfaced as several of the e2e failures after #2558 dropped the 3× install retry, each on a different victim pod (velero node-agent, fluent-bit, fdb-operator, cdi-operator, capi bootstrap). Tracked upstream at cilium/cilium#38313 (closed stale); no released or pre-release Cilium fixes the leak and no flag disables it. A more precise upstream report is being filed.What this adds.
hack/e2e-cilium-endpoint-leak-healer.sh, launched as a file-scoped background watchdog viasetup_file/teardown_fileinhack/e2e-install-cozystack.batsso it covers the whole install + tenant + app window. It:is already in usesignature and extracts the disputed IP and owning node;Runningpod with that IP, and a different pod is currently wedged requesting it;cilium-dbg endpoint disconnect ipv4:<X>(DELETE /endpoint/{id}→unexpose→removeReferencesLocked, clearing the stale IP-keyed entry). Blast radius is one dead endpoint, not an agent restart, so other endpoints on the node are untouched.It is read-only until an orphan is positively confirmed, and it refuses (logging loudly) to disconnect an endpoint that backs a live pod — so a genuine duplicate-IP bug stays visible rather than being masked. Any
HEAL/REFUSEaction is surfaced into the test trace on teardown.Scope / lifetime. CI-only; no product or runtime code is touched. This is a band-aid to stop the leak from failing unrelated PRs while the upstream fix lands — remove the script and the bats hooks once a fixed Cilium ships (called out in the script header and commit message).
Screenshots
N/A — no UI changes.
Release note
Summary by CodeRabbit