Repository navigation
Handle Windows TCC/WDDM mode more robustly #206
Description
This issue tracks an internal discussion with QA. This simple snippet shows why using cuda.core today on Windows might fail, depending on if it's TCC or WDDM mode:
>>> from cuda import cuda, cudart
>>> print(cudart.cudaGetDevice())
(<cudaError_t.cudaSuccess: 0>, 0)
>>> print(cuda.cuDeviceGetMemPool(0))
(<CUresult.CUDA_ERROR_NOT_SUPPORTED: 801>, <CUmemoryPool 0x0>)cuda.core currently assumes the stream-ordered memory allocator is available. However, CUDA on Windows is a bit more complicated than on Linux, since there are two operation modes:
- In the WDDM mode (which is the case during
cuda.coredevelopment), things should work just fine. - In the TCC mode (as reported by QA), this is unsupported
We need some treatments to make it usable on TCC.
Activity
[-]Handle WIndows TCC/WDDM mode more robustly[/-][+]Handle Windows TCC/WDDM mode more robustly[/+]leofang commentedon Nov 1, 2024
xref: https://forums.developer.nvidia.com/t/cudaerrornotsupported-during-cudamallocasync-on-windows-10-based-azure-vm-with-tesla-t4-gpu/251405/2
jrhemstad commentedon Nov 1, 2024
Can't we just say the pool is not available on Windows in TCC mode? I don't think we need to go above and beyond to support something the driver doesn't support.
leofang commentedon Nov 1, 2024
It is not appropriate because CUDA does support Windows TCC mode, just not the mempool. Right now
cuda.coreis not functional at all only because I forgot (😞) mempools are not there, but we can easily provide a fallback path to make it work (by wrappingcudaMalloc/cudaFreeas suggested in #208).@jrhemstad I suggest us to take this seriously if we want CUDA Mode to succeed, as we have many Windows TCC users in the LLM space, and they all hit this issue (it only took me 1 min to quickly google these):
leofang commentedon Nov 12, 2024
#209 is related (the 3rd step we should take in the future to address this issue).
leofang commentedon Nov 25, 2024
Another reason that TCC is important is because it's the default of GHA Windows GPU runner, e.g.:
https://github.com/cupy-ci-poc/cupy/actions/runs/12004661559/job/33459987560#step:5:15
leofang commentedon Dec 29, 2024
Closing this as it's working for TCC now. As mentioned above the potential improvement is tracked in #209.