fix(controller): cascade Resource deletions and surface precise finalize statuses - #3917
Conversation
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughThe PR fixes finalization gaps in the Project and Resource controllers. The Project controller now cascades deletion to owned ChangesController Finalization Cascade Fixes
Sequence Diagram(s)sequenceDiagram
rect rgba(173, 216, 230, 0.5)
Note over finalize,deleteResourcesAndWait: Project deletion cascade
finalize->>finalize: Conditionally set ConditionFinalizing = True
finalize->>deleteChildAndLinkedResources: invoke
deleteChildAndLinkedResources->>deleteComponentsAndWait: list & delete owned Components
deleteChildAndLinkedResources->>deleteResourcesAndWait: list & delete owned Resources (IndexKeyResourceOwnerProjectName)
alt Components or Resources remain
deleteChildAndLinkedResources-->>finalize: return true
finalize->>finalize: MarkTrueCondition(Finalizing) + requeue after 5s
else All gone
deleteChildAndLinkedResources-->>finalize: return false
finalize->>finalize: proceed to deleteExternalResourcesAndWait
end
end
rect rgba(144, 238, 144, 0.5)
Note over finalize,deleteOwnedResourceReleasesAndWait: Resource deletion cascade
finalize->>finalize: Conditionally set ConditionFinalizing = True
finalize->>deleteOwnedResourceReleaseBindingsAndWait: list & delete owned ResourceReleaseBindings
alt Bindings existed
deleteOwnedResourceReleaseBindingsAndWait-->>finalize: return true
finalize->>finalize: MarkTrueCondition(Finalizing) + requeue
else No bindings
deleteOwnedResourceReleaseBindingsAndWait-->>finalize: return false
finalize->>deleteOwnedResourceReleasesAndWait: list & delete owned ResourceReleases
alt Releases existed
deleteOwnedResourceReleasesAndWait-->>finalize: return true
finalize->>finalize: MarkTrueCondition(Finalizing) + requeue
else No releases
deleteOwnedResourceReleasesAndWait-->>finalize: return false
finalize->>finalize: remove finalizer
end
end
end
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~25 minutes Suggested reviewers
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
🧹 Nitpick comments (2)
internal/controller/resource/controller_finalize_test.go (1)
127-140: ⚡ Quick winAssert that binding deletion was actually triggered in the blocking-path test.
The test proves the
Resourcefinalizer is held, but it does not verify that cascade deletion was initiated on the ownedResourceReleaseBinding. Adding aDeletionTimestampassertion on the binding makes this regression test stricter for the new active-delete behavior.As per coding guidelines, "Add tests for critical logic or regressions."
Suggested test assertion
// 2nd reconcile: binding still present, requeue without acting. result, err := r.Reconcile(finCtx, req) Expect(err).NotTo(HaveOccurred()) Expect(result.RequeueAfter).To(BeNumerically(">", 0)) + + updatedBinding := &openchoreov1alpha1.ResourceReleaseBinding{} + Expect(cli.Get(finCtx, client.ObjectKeyFromObject(binding), updatedBinding)).To(Succeed()) + Expect(updatedBinding.DeletionTimestamp).NotTo(BeNil(), + "binding should be marked for deletion by resource finalizer") updated := &openchoreov1alpha1.Resource{} Expect(cli.Get(finCtx, client.ObjectKeyFromObject(res), updated)).To(Succeed())🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@internal/controller/resource/controller_finalize_test.go` around lines 127 - 140, The test verifies that the Resource finalizer is held but does not verify that cascade deletion was actually initiated on the owned ResourceReleaseBinding. Add an assertion after the existing Resource status checks to fetch the ResourceReleaseBinding object associated with the Resource using cli.Get and the appropriate object key, then assert that the binding's DeletionTimestamp is not nil to prove that cascade deletion was triggered on the owned binding resource.Source: Coding guidelines
internal/controller/resource/controller_finalize.go (1)
82-84: ⚡ Quick winGuard status writes behind
MarkTrueConditionchange checks.Line 82 and Line 95 call
controller.UpdateStatusConditionseven when the condition is unchanged. This can create unnecessary status-write traffic during every requeue loop.As per coding guidelines, "Ensure status updates handle conflicts and avoid API spam."
Suggested patch
- controller.MarkTrueCondition(res, ConditionFinalizing, ReasonFinalizing, "Waiting for ResourceReleaseBindings to be deleted") - if err := controller.UpdateStatusConditions(ctx, r.Client, old, res); err != nil { - return ctrl.Result{}, err - } + if controller.MarkTrueCondition(res, ConditionFinalizing, ReasonFinalizing, "Waiting for ResourceReleaseBindings to be deleted") { + if err := controller.UpdateStatusConditions(ctx, r.Client, old, res); err != nil { + return ctrl.Result{}, err + } + } return ctrl.Result{RequeueAfter: requeueWaitForChildren}, nil } @@ - controller.MarkTrueCondition(res, ConditionFinalizing, ReasonFinalizing, "Waiting for ResourceReleases to be deleted") - if err := controller.UpdateStatusConditions(ctx, r.Client, old, res); err != nil { - return ctrl.Result{}, err - } + if controller.MarkTrueCondition(res, ConditionFinalizing, ReasonFinalizing, "Waiting for ResourceReleases to be deleted") { + if err := controller.UpdateStatusConditions(ctx, r.Client, old, res); err != nil { + return ctrl.Result{}, err + } + } return ctrl.Result{RequeueAfter: requeueWaitForChildren}, nil }Also applies to: 95-97
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@internal/controller/resource/controller_finalize.go` around lines 82 - 84, Guard the `controller.UpdateStatusConditions` calls on lines 82-84 and 95-97 behind condition change checks to avoid unnecessary API writes. Before calling `UpdateStatusConditions` after the `MarkTrueCondition` call for `ConditionFinalizing`, verify that the condition on the resource object actually differs from the previous state. Only proceed with the status update if the condition has genuinely changed, preventing redundant API calls during requeue loops.Source: Coding guidelines
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@internal/controller/resource/controller_finalize_test.go`:
- Around line 127-140: The test verifies that the Resource finalizer is held but
does not verify that cascade deletion was actually initiated on the owned
ResourceReleaseBinding. Add an assertion after the existing Resource status
checks to fetch the ResourceReleaseBinding object associated with the Resource
using cli.Get and the appropriate object key, then assert that the binding's
DeletionTimestamp is not nil to prove that cascade deletion was triggered on the
owned binding resource.
In `@internal/controller/resource/controller_finalize.go`:
- Around line 82-84: Guard the `controller.UpdateStatusConditions` calls on
lines 82-84 and 95-97 behind condition change checks to avoid unnecessary API
writes. Before calling `UpdateStatusConditions` after the `MarkTrueCondition`
call for `ConditionFinalizing`, verify that the condition on the resource object
actually differs from the previous state. Only proceed with the status update if
the condition has genuinely changed, preventing redundant API calls during
requeue loops.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: 425b627e-815d-4e5b-87a0-2dd81acaf847
📒 Files selected for processing (11)
install/helm/openchoreo-control-plane/templates/gateway/gateway.yamlinstall/helm/openchoreo-control-plane/values.schema.jsoninstall/helm/openchoreo-control-plane/values.yamlinstall/helm/openchoreo-observability-plane/templates/gateway/gateway.yamlinstall/helm/openchoreo-observability-plane/values.schema.jsoninstall/helm/openchoreo-observability-plane/values.yamlinternal/controller/project/controller_finalize.gointernal/controller/project/suite_test.gointernal/controller/resource/controller_finalize.gointernal/controller/resource/controller_finalize_test.gointernal/controller/watch.go
Codecov Report❌ Patch coverage is 📢 Thoughts on this report? Let us know! |
|
Hi @kavix, thanks for opening the PR. |
| {{- end }} | ||
| spec: | ||
| gatewayClassName: kgateway | ||
| gatewayClassName: {{ .Values.gateway.gatewayClassName | default "kgateway" }} |
There was a problem hiding this comment.
This looks irrelevant. Why do we need this change for this PR?
There was a problem hiding this comment.
I accidentally included those commits initially, but I've just force-pushed to remove them. The PR is clean now!
Sure! |
dc39aca to
86aa052
Compare
| openchoreov1alpha1 "github.com/openchoreo/openchoreo/api/v1alpha1" | ||
| ) | ||
|
|
||
| func TestIndexResourceReleaseBindingOwnerEnv(t *testing.T) { |
There was a problem hiding this comment.
Any reason for deleting this test case?
| // deleteOwnedResourceReleaseBindingsAndWait triggers deletion of every ResourceReleaseBinding | ||
| // owned by the given Resource. Returns true if any bindings still exist; the caller | ||
| // should requeue to wait for them to be deleted. | ||
| func (r *Reconciler) deleteOwnedResourceReleaseBindingsAndWait(ctx context.Context, res *openchoreov1alpha1.Resource) (bool, error) { |
There was a problem hiding this comment.
Won't this need to check the retention policy?
| } | ||
|
|
||
| newBinding := func(name, ownerResource string) *openchoreov1alpha1.ResourceReleaseBinding { | ||
| newBinding := func(name, ownerResource string, withFinalizer bool) *openchoreov1alpha1.ResourceReleaseBinding { |
There was a problem hiding this comment.
Shall we add more test cases to cover the deletion with different retention policies?
There was a problem hiding this comment.
Let's add a test to verify the resource deletion during the project deletion
051c06c to
1eb0990
Compare
…ize statuses Signed-off-by: kavix <[email protected]>
|
Backport failed. See the workflow run for details. |
|
The PR is merged. Thank you for the contribution. |
|
Backport failed. See the workflow run for details. |
…ize statuses (openchoreo#3917) Signed-off-by: Kavindu Sachinthe <[email protected]> (cherry picked from commit 0802819) Signed-off-by: Miraj Abeysekara <[email protected]>
…ize statuses (openchoreo#3917) Signed-off-by: Kavindu Sachinthe <[email protected]> (cherry picked from commit 0802819) Signed-off-by: Miraj Abeysekara <[email protected]>
…ize statuses (backport to release-v1.1) (#4047) * fix(controller): cascade Resource deletions and surface precise finalize statuses (#3917) Signed-off-by: Kavindu Sachinthe <[email protected]> (cherry picked from commit 0802819) Signed-off-by: Miraj Abeysekara <[email protected]> * test(controller): drop ProjectSpec.Type from backported finalize test release-v1.1 ProjectSpec has no Type field (ProjectTypeRef was added on main after this branch was cut), so the backported project finalize integration test failed to compile. Remove the Type from the test's Project literal to match the release-v1.1 API. No change to the fix. Signed-off-by: Miraj Abeysekara <[email protected]> --------- Signed-off-by: Kavindu Sachinthe <[email protected]> Signed-off-by: Miraj Abeysekara <[email protected]> Co-authored-by: Kavindu Sachinthe <[email protected]>
Purpose
Currently, deleting a
ResourcewithretainPolicy: Deletehangs in theTerminatingstate, as its finalizer fails to cascade the deletion to itsResourceReleaseBindings. Additionally, deleting aProjectdoes not cascade to itsResourceobjects, leaving them orphaned. This PR fixes the finalizer logic for both controllers to correctly trigger cleanup of dependent artifacts and prevents infinite loops when updating block statuses.Approach
hasOwnedResourceReleaseBindingswithdeleteOwnedResourceReleaseBindingsAndWaitto explicitly issueDeletecalls on child bindings during finalization.IndexKeyResourceOwnerProjectName) in the watch logic to enable rapid lookup ofResourceobjects owned by aProject. AddeddeleteResourcesAndWaitto initiate deletion for all owned Resources alongside Components.ResourceandProjectfinalizers to ensure specific blocked statuses (e.g.,"Waiting for child resources to be deleted") are persisted correctly without being continuously overwritten by generic finalizing statuses, avoiding an infinite requeue loop.internal/controller/resource/controller_finalize_test.goandinternal/controller/project/suite_test.goto support the new cascades and test environments.Related Issues
Checklist
backport/<release-branch>label if this should be backported (e.g.,backport/release-v1.0)Remarks