Repository navigation
[FEA]: Expose NVML events as async functions #1458
Description
Is this a duplicate?
- I confirmed there appear to be no duplicate issues for this request and that I agree to the Code of Conduct
Area
cuda.core
Is your feature request related to a problem? Please describe.
As of #1448, cuda.core.system exposes NVML events using a wait function with a timeout. The natural way to expose this to Python would be an async function, however, I'm wary of adding this functionality without a concrete use case and good testing. Filing this bug as a placeholder in case that arises.
Describe the solution you'd like
NA
Describe alternatives you've considered
NA
Additional context
NA
Activity
leofang commentedon Jan 15, 2026
It'd be nice to know what the latency requirement is for NVML event queries. If it needs to be very low (by how much?), I've heard in the past from @pentschev based on his experience in UCX-py/UCXX that Python async event loops may not be performant enough.
pentschev commentedon Jan 15, 2026
That's correct, I don't mind having an async interface, but I think a non-async interface needs to be provided regardless. I have previously analyzed overhead, and
asynccan be prohibitively expensive (progressively slower from a plain coroutine -> coroutine with future -> task), even on the most basic case of a plain coroutine it's 3x the cost of a regular blocking Python function. So while async would be a nice convenience, providing only async versions is probably insufficient for performance-critical areas.0z5a commentedon Sep 19, 2026
@mdboom
I have a concrete use case for this in long-running multi-GPU workers: the process already has an asyncio control plane, and I’d like to consume NVML events without dedicating a Python thread or blocking the event loop on the existing wait API.
I’d be happy to take a scoped implementation if this use case is sufficient.
For the first version I would keep the semantics narrow:
async waiting without blocking the event loop;
timeout and cancellation behavior;
clean shutdown with multiple GPU/event sets;
no requirement to fabricate real hardware failures in normal CI.
For testing, I can cover the asyncio/cancellation/lifetime behavior with a controllable test seam around the blocking wait path, and run additional multi-GPU validation on L20s. I would not rely on being able to inject real Xid/ECC failures.
Would that be a reasonable scope to assign?