Repository navigation
[BUG]: default_stream() singletons permanently cache the first-seen context and device #2485
Description
Type of Bug
Silent Failure
Component
cuda.core
Describe the bug
default_stream() returns one of two module-level singletons — LEGACY_DEFAULT_STREAM or PER_THREAD_DEFAULT_STREAM — created at import time. Both wrap a default-stream token, which means "whatever context is current." Neither carries a context of its own: get_stream_context returns empty for those handles, so _from_handle leaves _h_context unset.
The first caller that needs a context fills it in. Stream_ensure_ctx asks cuStreamGetCtx, gets whatever is current at that moment, and stores it on the singleton. Stream_ensure_ctx_device then caches _device_id the same way. Neither field is ever cleared.
So a process-wide object ends up permanently bound to one context, even though the token it wraps isn't.
That shows up in a few places:
-
Stale answers after a context switch.
Stream.context,.device,.resources, and.record()all read the cached fields. On a multi-GPU machine they keep reporting the first device;record()builds the event on the first context, not the current one. -
First-touch order decides the answer. Even
__repr__populates the cache, so printing the stream in a log or a debugger permanently changes later behavior. -
A context stuck for the process lifetime. The cached
ContextHandleholds a reference, so a primary context that would otherwise be releasable never is. -
First use races. The write happens under
nogilon an object shared by every thread. ForPER_THREAD_DEFAULT_STREAMthe caching is wrong even in principle — each thread has its own per-thread default stream and its own current context.
Normal streams are fine. Their context is fixed at creation and _h_context comes from the stream box, which is what the existing note on Stream.device is talking about.
Spotted while reviewing #2455, which needs a context for a default-stream token and can't take it off the Stream for this reason. Reproduced on a local 2-GPU system.
How to Reproduce
On a host with two GPUs:
from cuda.core import Device, LEGACY_DEFAULT_STREAM
Device(0).set_current()
print(LEGACY_DEFAULT_STREAM.device.device_id) # 0
Device(1).set_current()
print(Device().device_id) # 1
print(LEGACY_DEFAULT_STREAM.device.device_id) # 0, expected 1
print(LEGACY_DEFAULT_STREAM.context) # still device 0's contextSame idea with two contexts on one device (e.g. a green context made current after the first query).
Order dependence on its own:
from cuda.core import Device, LEGACY_DEFAULT_STREAM
Device(0).set_current()
repr(LEGACY_DEFAULT_STREAM) # binds the singleton to device 0
Device(1).set_current()
LEGACY_DEFAULT_STREAM.record() # event created on device 0's contextFirst-touch pinning (fresh process):
from cuda.core import Device, LEGACY_DEFAULT_STREAM
Device(1).set_current()
print(LEGACY_DEFAULT_STREAM.device.device_id) # 1
Device(0).set_current()
print(LEGACY_DEFAULT_STREAM.device.device_id) # still 1(Device(...).default_stream is the same singleton; default_stream() itself is not exported from cuda.core.)
Expected behavior
A default-stream token has no context of its own, so queries against the default stream should resolve against whatever is current at call time. .context, .device, .resources, and .record() should follow context switches, and a query shouldn't pin a context reference beyond the call.
Two plausible fixes: skip the cache when the handle is CU_STREAM_LEGACY / CU_STREAM_PER_THREAD and resolve per call, or have default_stream() return a fresh Stream so per-object caching is sound. The first keeps today's object identity (__eq__ / __hash__ already key off the handle, not identity).
System information
Reproduced on a local 2-GPU system against an installed cuda.core built from recent main.
Activity
Andy-Jost commentedon Aug 4, 2026
Related issues (post-file scan)
No duplicate turned up for the singleton caching itself. Nearby items:
Device.create_*builds resources on the current context while labeling them with theDevicethey were called on. Same multi-device footgun family, different mechanism (noset_currentbefore the driver call, rather than a permanently cached context on a default-stream token).Stream.device/Stream.contextcircular-import cleanup; not about default-stream tokens.CUDA_PYTHON_CUDA_PER_THREAD_DEFAULT_STREAM=1#971 — CI coverage forCUDA_PYTHON_CUDA_PER_THREAD_DEFAULT_STREAM; useful once a fix lands, not the same bug.aryanputta commentedon Aug 4, 2026
Opened #2490 for this.
@Andy-Jost since you filed this, could you take a look at the implementation when you have time? Thanks.
It takes the first of the two options you outlined, skipping the cache when the handle is
CU_STREAM_LEGACYorCU_STREAM_PER_THREADand resolving per call, so object identity and the handle-keyed__eq__/__hash__are unchanged.Stream_ensure_ctxandStream_ensure_ctx_devicebecomeStream_get_ctxandStream_get_ctx_device, returning the context and device through out-parameters and writing back to the object only for non-default streams.Two things worth flagging for review:
record()previously resolved the context only when_device_id < 0, so a stream with a device ID already set but no context handle would have passed an empty context toEvent._init. The new path always resolves both. That is a behaviour change beyond the default-stream case._stream.pyxalready notes green contexts rejectCU_STREAM_DEFAULTfor created streams, so I did not want to assert semantics there that the repo has not established. If there is a preferred single-GPU way to exercise this, I am happy to add it.I also opened this before being assigned. Happy to close it if you would rather take this internally.
fix(cuda.core): harden default-stream context resolution for NVIDIA#2485
Andy-Jost commentedon Aug 4, 2026
@aryanputta Thank you for the contribution. I run the tests locally on a 2-GPU machine and touched up a few details (mostly docstrings). Planning to merge once CI is green.
aryanputta commentedon Aug 4, 2026
np! thanks for the review!
cuda.core: resolve default-stream context per call instead of caching…
Prepare for 13.4.1 release (#2788)
Pull 13.4.1 changes into main (#2789)