Repository navigation
Revisit user object lifetime testing #2044
Description
User objects attached to graphs require testing to ensure they are freed. The CUDA driver deallocates user objects asynchronously — cuUserObjectCreate requires CU_USER_OBJECT_NO_DESTRUCTOR_SYNC as the only legal flag — making it difficult to write such tests. The current approach, used in cuda_core/tests/graph/test_graph_definition_lifetime.py::test_kernel_args_buffer_lifetime (added in #2041 to address Rob's review comment on the original fix for #2039), relies on polling to observe cleanup. Leo flagged the polling pattern as unreliable for SWQA. Consider other approaches; the broader graph-ownership rework planned for #1330 is a likely place to revisit this.
Activity
[-]Replace polling-based release assertion in test_kernel_args_buffer_lifetime with a deterministic check[/-][+]Improve lifetime tests for graph resources[/+][-]Improve lifetime tests for graph resources[/-][+]Revisit user object lifetime testing[/+]Andy-Jost commentedon Aug 12, 2026
The user-object system has been comprehensively rewritten since this was filed, and the original concerns are effectively addressed.
Cleanup mechanism is now deterministic at the reference-count level. The
DeferredCleanupQueue+Py_AddPendingCalldesign ensures that once CUDA fires the destructor callback, the attachment drains from Python's main thread. The CUDA callback only enqueues — no Python calls there — with coalescing and queue-saturation retry built in.Polling is unavoidable with this mechanism. Since
Py_AddPendingCallonly fires between bytecode instructions, there is no synchronous signal to wait on._wait_untilactively drives cleanup viagc.collect(),time.sleep(0)(GIL yield), and a bounded timeout. Timeout is a hard failure, not a warning, so a real leak still fails the suite — this addresses Leo's SWQA concern about the polling pattern being unreliable.Coverage is now comprehensive.
test_graph_definition_lifetime.pyhas grown from one test to ~50 scenarios covering kernel-arg lifetime, memcpy/memset operand lifetime, event-node lifetime, child- and conditional-graph lifetime, clone propagation, executable-graph lifetime, queue saturation, thread affinity, and shutdown safety.The one intentionally unverified area is shutdown-time cleanup (attachments are intentionally leaked during finalization to avoid calling Python in an unsafe context;
test_pending_cleanup_is_safe_during_python_shutdownverifies there is no hang or crash). This is a documented, acceptable tradeoff.