Skip to content

fix(controller): cascade Resource deletions and surface precise finalize statuses - #3917

Merged
Mirage20 merged 1 commit into
openchoreo:mainfrom
kavix:fix/issue-3909
Jun 25, 2026
Merged

Mirage20 merged 1 commit into
openchoreo:mainfrom
kavix:fix/issue-3909

Conversation

@kavix

@kavix kavix commented Jun 19, 2026

Copy link
Copy Markdown
Contributor

Purpose

Briefly describe the problem or need driving this PR and how it resolves the issue. Include links to related issues if applicable.

Currently, deleting a Resource with retainPolicy: Delete hangs in the Terminating state, as its finalizer fails to cascade the deletion to its ResourceReleaseBindings. Additionally, deleting a Project does not cascade to its Resource objects, leaving them orphaned. This PR fixes the finalizer logic for both controllers to correctly trigger cleanup of dependent artifacts and prevents infinite loops when updating block statuses.

Approach

Summarize the solution and implementation details.

  • Resource Controller: Replaced hasOwnedResourceReleaseBindings with deleteOwnedResourceReleaseBindingsAndWait to explicitly issue Delete calls on child bindings during finalization.
  • Project Controller: Introduced a shared index (IndexKeyResourceOwnerProjectName) in the watch logic to enable rapid lookup of Resource objects owned by a Project. Added deleteResourcesAndWait to initiate deletion for all owned Resources alongside Components.
  • Status Conditions Fix: Updated Resource and Project finalizers to ensure specific blocked statuses (e.g., "Waiting for child resources to be deleted") are persisted correctly without being continuously overwritten by generic finalizing statuses, avoiding an infinite requeue loop.
  • Testing: Updated internal/controller/resource/controller_finalize_test.go and internal/controller/project/suite_test.go to support the new cascades and test environments.

Related Issues

Include any related issues that are resolved by this PR.
Closes #3909

Checklist

  • Tests added or updated (unit, integration, etc.)
  • Samples updated (if applicable)
  • Added backport/<release-branch> label if this should be backported (e.g., backport/release-v1.0)

Remarks

Add any additional context, known issues, or TODOs related to this PR.
N/A

@coderabbitai

coderabbitai Bot commented Jun 19, 2026 •

Copy link
Copy Markdown

Review Change Stack

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review
📝 Walkthrough

Walkthrough

The PR fixes finalization gaps in the Project and Resource controllers. The Project controller now cascades deletion to owned Resource objects (in addition to Components), and the Resource controller replaces a passive binding-existence check with active cascade deletion of owned ResourceReleaseBinding objects. Both controllers explicitly surface the Finalizing status condition and requeue while waiting. A shared field index for Resource by owner project name is added to support efficient Resource lookup during Project finalization.

Changes

Controller Finalization Cascade Fixes

Layer / File(s) Summary
Shared index key and SetupSharedIndexes wiring
internal/controller/watch.go, internal/controller/watch_test.go
Exports IndexKeyResourceOwnerProjectName constant and registers a controller-runtime field index for Resource objects keyed by resource.spec.owner.projectName in SetupSharedIndexes. Unit test validates the indexer function via mocked manager for Resource ownership lookup cases.
Project finalizer: cascade Resource deletion and Finalizing condition
internal/controller/project/controller_finalize.go, internal/controller/project/suite_test.go, internal/controller/project/controller_integration_test.go
Conditionally sets ConditionFinalizing on first observation; adds deleteResourcesAndWait to list and delete owned Resource objects via the new index; gates progression to external cleanup until both components and resources are gone. Test suite adds Resource to cache and registers the new field index; integration test creates an owned Resource before project deletion and refactors post-deletion verification to use Eventually with reconcile polling.
Resource finalizer: cascade ResourceReleaseBinding deletion and Finalizing condition
internal/controller/resource/controller_finalize.go, internal/controller/resource/controller_finalize_test.go
Replaces hasOwnedResourceReleaseBindings with deleteOwnedResourceReleaseBindingsAndWait, which lists, deletes, and reports remaining owned ResourceReleaseBinding objects. Sets Finalizing condition while bindings or releases remain; updates status only when condition sets semantically change. Tests updated to create bindings with their own finalizer and assert DeletionTimestamp is set after reconciliation.

Sequence Diagram(s)

sequenceDiagram
  rect rgba(173, 216, 230, 0.5)
    Note over finalize,deleteResourcesAndWait: Project deletion cascade
    finalize->>finalize: Conditionally set ConditionFinalizing = True
    finalize->>deleteChildAndLinkedResources: invoke
    deleteChildAndLinkedResources->>deleteComponentsAndWait: list & delete owned Components
    deleteChildAndLinkedResources->>deleteResourcesAndWait: list & delete owned Resources (IndexKeyResourceOwnerProjectName)
    alt Components or Resources remain
      deleteChildAndLinkedResources-->>finalize: return true
      finalize->>finalize: MarkTrueCondition(Finalizing) + requeue after 5s
    else All gone
      deleteChildAndLinkedResources-->>finalize: return false
      finalize->>finalize: proceed to deleteExternalResourcesAndWait
    end
  end
  rect rgba(144, 238, 144, 0.5)
    Note over finalize,deleteOwnedResourceReleasesAndWait: Resource deletion cascade
    finalize->>finalize: Conditionally set ConditionFinalizing = True
    finalize->>deleteOwnedResourceReleaseBindingsAndWait: list & delete owned ResourceReleaseBindings
    alt Bindings existed
      deleteOwnedResourceReleaseBindingsAndWait-->>finalize: return true
      finalize->>finalize: MarkTrueCondition(Finalizing) + requeue
    else No bindings
      deleteOwnedResourceReleaseBindingsAndWait-->>finalize: return false
      finalize->>deleteOwnedResourceReleasesAndWait: list & delete owned ResourceReleases
      alt Releases existed
        deleteOwnedResourceReleasesAndWait-->>finalize: return true
        finalize->>finalize: MarkTrueCondition(Finalizing) + requeue
      else No releases
        deleteOwnedResourceReleasesAndWait-->>finalize: return false
        finalize->>finalize: remove finalizer
      end
    end
  end
Loading

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

Suggested reviewers

  • akila-i
  • binoyPeries
  • NomadXD
  • sameerajayasoma
  • mevan-karu
🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 50.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly summarizes the main changes: cascading Resource deletions and surfacing finalize status conditions.
Description check ✅ Passed The PR description covers Purpose, Approach, Related Issues, Checklist, and Remarks sections as required by the template.
Linked Issues check ✅ Passed All requirements from issue #3909 are met: Resource finalizer cascades to ResourceReleaseBindings, Project cascades to Resources, and status conditions surface blocker reasons.
Out of Scope Changes check ✅ Passed The lint.mk change excluding node_modules is a minor housekeeping improvement unrelated to the main cascading deletion fix but reasonable to include.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (2)
internal/controller/resource/controller_finalize_test.go (1)

127-140: ⚡ Quick win

Assert that binding deletion was actually triggered in the blocking-path test.

The test proves the Resource finalizer is held, but it does not verify that cascade deletion was initiated on the owned ResourceReleaseBinding. Adding a DeletionTimestamp assertion on the binding makes this regression test stricter for the new active-delete behavior.

As per coding guidelines, "Add tests for critical logic or regressions."

Suggested test assertion
 		// 2nd reconcile: binding still present, requeue without acting.
 		result, err := r.Reconcile(finCtx, req)
 		Expect(err).NotTo(HaveOccurred())
 		Expect(result.RequeueAfter).To(BeNumerically(">", 0))
+
+		updatedBinding := &openchoreov1alpha1.ResourceReleaseBinding{}
+		Expect(cli.Get(finCtx, client.ObjectKeyFromObject(binding), updatedBinding)).To(Succeed())
+		Expect(updatedBinding.DeletionTimestamp).NotTo(BeNil(),
+			"binding should be marked for deletion by resource finalizer")
 
 		updated := &openchoreov1alpha1.Resource{}
 		Expect(cli.Get(finCtx, client.ObjectKeyFromObject(res), updated)).To(Succeed())
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@internal/controller/resource/controller_finalize_test.go` around lines 127 -
140, The test verifies that the Resource finalizer is held but does not verify
that cascade deletion was actually initiated on the owned
ResourceReleaseBinding. Add an assertion after the existing Resource status
checks to fetch the ResourceReleaseBinding object associated with the Resource
using cli.Get and the appropriate object key, then assert that the binding's
DeletionTimestamp is not nil to prove that cascade deletion was triggered on the
owned binding resource.

Source: Coding guidelines

internal/controller/resource/controller_finalize.go (1)

82-84: ⚡ Quick win

Guard status writes behind MarkTrueCondition change checks.

Line 82 and Line 95 call controller.UpdateStatusConditions even when the condition is unchanged. This can create unnecessary status-write traffic during every requeue loop.

As per coding guidelines, "Ensure status updates handle conflicts and avoid API spam."

Suggested patch
-		controller.MarkTrueCondition(res, ConditionFinalizing, ReasonFinalizing, "Waiting for ResourceReleaseBindings to be deleted")
-		if err := controller.UpdateStatusConditions(ctx, r.Client, old, res); err != nil {
-			return ctrl.Result{}, err
-		}
+		if controller.MarkTrueCondition(res, ConditionFinalizing, ReasonFinalizing, "Waiting for ResourceReleaseBindings to be deleted") {
+			if err := controller.UpdateStatusConditions(ctx, r.Client, old, res); err != nil {
+				return ctrl.Result{}, err
+			}
+		}
 		return ctrl.Result{RequeueAfter: requeueWaitForChildren}, nil
 	}
@@
-		controller.MarkTrueCondition(res, ConditionFinalizing, ReasonFinalizing, "Waiting for ResourceReleases to be deleted")
-		if err := controller.UpdateStatusConditions(ctx, r.Client, old, res); err != nil {
-			return ctrl.Result{}, err
-		}
+		if controller.MarkTrueCondition(res, ConditionFinalizing, ReasonFinalizing, "Waiting for ResourceReleases to be deleted") {
+			if err := controller.UpdateStatusConditions(ctx, r.Client, old, res); err != nil {
+				return ctrl.Result{}, err
+			}
+		}
 		return ctrl.Result{RequeueAfter: requeueWaitForChildren}, nil
 	}

Also applies to: 95-97

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@internal/controller/resource/controller_finalize.go` around lines 82 - 84,
Guard the `controller.UpdateStatusConditions` calls on lines 82-84 and 95-97
behind condition change checks to avoid unnecessary API writes. Before calling
`UpdateStatusConditions` after the `MarkTrueCondition` call for
`ConditionFinalizing`, verify that the condition on the resource object actually
differs from the previous state. Only proceed with the status update if the
condition has genuinely changed, preventing redundant API calls during requeue
loops.

Source: Coding guidelines

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@internal/controller/resource/controller_finalize_test.go`:
- Around line 127-140: The test verifies that the Resource finalizer is held but
does not verify that cascade deletion was actually initiated on the owned
ResourceReleaseBinding. Add an assertion after the existing Resource status
checks to fetch the ResourceReleaseBinding object associated with the Resource
using cli.Get and the appropriate object key, then assert that the binding's
DeletionTimestamp is not nil to prove that cascade deletion was triggered on the
owned binding resource.

In `@internal/controller/resource/controller_finalize.go`:
- Around line 82-84: Guard the `controller.UpdateStatusConditions` calls on
lines 82-84 and 95-97 behind condition change checks to avoid unnecessary API
writes. Before calling `UpdateStatusConditions` after the `MarkTrueCondition`
call for `ConditionFinalizing`, verify that the condition on the resource object
actually differs from the previous state. Only proceed with the status update if
the condition has genuinely changed, preventing redundant API calls during
requeue loops.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 425b627e-815d-4e5b-87a0-2dd81acaf847

📥 Commits

Reviewing files that changed from the base of the PR and between 7503a25 and 887f8a7.

📒 Files selected for processing (11)
  • install/helm/openchoreo-control-plane/templates/gateway/gateway.yaml
  • install/helm/openchoreo-control-plane/values.schema.json
  • install/helm/openchoreo-control-plane/values.yaml
  • install/helm/openchoreo-observability-plane/templates/gateway/gateway.yaml
  • install/helm/openchoreo-observability-plane/values.schema.json
  • install/helm/openchoreo-observability-plane/values.yaml
  • internal/controller/project/controller_finalize.go
  • internal/controller/project/suite_test.go
  • internal/controller/resource/controller_finalize.go
  • internal/controller/resource/controller_finalize_test.go
  • internal/controller/watch.go

@codecov

codecov Bot commented Jun 19, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 62.96296% with 30 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
...nternal/controller/resource/controller_finalize.go 53.33% 16 Missing and 5 partials ⚠️
internal/controller/project/controller_finalize.go 75.00% 4 Missing and 3 partials ⚠️
internal/controller/watch.go 75.00% 2 Missing ⚠️

📢 Thoughts on this report? Let us know!

@akila-i

akila-i commented Jun 19, 2026

Copy link
Copy Markdown
Contributor

Hi @kavix, thanks for opening the PR.
Could you fix the failing PR checks by modifying the pr title properly and signing your commits (ex: git rebase upstream/main --signoff -i)

{{- end }}
spec:
gatewayClassName: kgateway
gatewayClassName: {{ .Values.gateway.gatewayClassName | default "kgateway" }}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks irrelevant. Why do we need this change for this PR?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I accidentally included those commits initially, but I've just force-pushed to remove them. The PR is clean now!

@kavix kavix changed the title Fix/issue 3909 fix(controller): cascade Resource deletions and surface precise finalize statuses Jun 19, 2026
@kavix
kavix requested a review from Mirage20 June 19, 2026 05:06
@kavix

kavix commented Jun 19, 2026

Copy link
Copy Markdown
Contributor Author

Hi @kavix, thanks for opening the PR. Could you fix the failing PR checks by modifying the pr title properly and signing your commits (ex: git rebase upstream/main --signoff -i)

Sure!

@kavix
kavix force-pushed the fix/issue-3909 branch 4 times, most recently from dc39aca to 86aa052 Compare June 19, 2026 05:53
openchoreov1alpha1 "github.com/openchoreo/openchoreo/api/v1alpha1"
)

func TestIndexResourceReleaseBindingOwnerEnv(t *testing.T) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any reason for deleting this test case?

// deleteOwnedResourceReleaseBindingsAndWait triggers deletion of every ResourceReleaseBinding
// owned by the given Resource. Returns true if any bindings still exist; the caller
// should requeue to wait for them to be deleted.
func (r *Reconciler) deleteOwnedResourceReleaseBindingsAndWait(ctx context.Context, res *openchoreov1alpha1.Resource) (bool, error) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Won't this need to check the retention policy?

}

newBinding := func(name, ownerResource string) *openchoreov1alpha1.ResourceReleaseBinding {
newBinding := func(name, ownerResource string, withFinalizer bool) *openchoreov1alpha1.ResourceReleaseBinding {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shall we add more test cases to cover the deletion with different retention policies?

Comment thread internal/controller/watch_test.go Outdated
Comment thread .golangci.yml Outdated
Comment thread make/lint.mk Outdated

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's add a test to verify the resource deletion during the project deletion

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added.

kavix

This comment was marked as low quality.

@kavix
kavix force-pushed the fix/issue-3909 branch 4 times, most recently from 051c06c to 1eb0990 Compare June 21, 2026 17:39
@kavix
kavix requested a review from Mirage20 June 25, 2026 06:19
@Mirage20 Mirage20 added the backport/release-v1.1 Backport to release-v1.1 branch label Jun 25, 2026
@Mirage20
Mirage20 merged commit 0802819 into openchoreo:main Jun 25, 2026
15 checks passed
@github-actions

Copy link
Copy Markdown

Backport failed. See the workflow run for details.

@Mirage20

Copy link
Copy Markdown
Contributor

The PR is merged. Thank you for the contribution.

@Mirage20 Mirage20 added backport/release-v1.1 Backport to release-v1.1 branch and removed backport/release-v1.1 Backport to release-v1.1 branch labels Jun 25, 2026
@github-actions

Copy link
Copy Markdown

Backport failed. See the workflow run for details.

Mirage20 pushed a commit to Mirage20/openchoreo that referenced this pull request Jul 1, 2026
…ize statuses (openchoreo#3917)

Signed-off-by: Kavindu Sachinthe <[email protected]>
(cherry picked from commit 0802819)
Signed-off-by: Miraj Abeysekara <[email protected]>
Mirage20 pushed a commit to Mirage20/openchoreo that referenced this pull request Jul 2, 2026
…ize statuses (openchoreo#3917)

Signed-off-by: Kavindu Sachinthe <[email protected]>
(cherry picked from commit 0802819)
Signed-off-by: Miraj Abeysekara <[email protected]>
Mirage20 added a commit that referenced this pull request Jul 2, 2026
…ize statuses (backport to release-v1.1) (#4047)

* fix(controller): cascade Resource deletions and surface precise finalize statuses (#3917)

Signed-off-by: Kavindu Sachinthe <[email protected]>
(cherry picked from commit 0802819)
Signed-off-by: Miraj Abeysekara <[email protected]>

* test(controller): drop ProjectSpec.Type from backported finalize test

release-v1.1 ProjectSpec has no Type field (ProjectTypeRef was added on
main after this branch was cut), so the backported project finalize
integration test failed to compile. Remove the Type from the test's
Project literal to match the release-v1.1 API. No change to the fix.

Signed-off-by: Miraj Abeysekara <[email protected]>

---------

Signed-off-by: Kavindu Sachinthe <[email protected]>
Signed-off-by: Miraj Abeysekara <[email protected]>
Co-authored-by: Kavindu Sachinthe <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport/release-v1.1 Backport to release-v1.1 branch reportedBy/community

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Resource deletion with retainPolicy=Delete does not delete the resource

3 participants