[BUG]: MemcpyNode.update() rejects stream-captured memcpy nodes #2649

Activity

1. github-actions

   added

   triageNeeds the team's attentionNeeds the team's attention

   on Aug 16, 2026
2. mdboom commented on Aug 20, 2026

   

   mdboom

   on Aug 20, 2026

   Contributor

   More actions

   Cc: @Andy-Jost for viz.
3. atiaomar1978-hub commented on Aug 20, 2026

   

   atiaomar1978-hub

   on Aug 20, 2026

   Last edited by atiaomar1978-hub

   Contributor

   More actions

   Cc: @Andy-Jost @mdboom

   RunPod verification (RTX 4090, driver 560.35, cuda.core main + CUDA 13.3)

   I reproduced this on GPU and traced the failure to _is_supported_memcpy_descriptor() rejecting driver-recorded UNIFIED memory types — not to anything multidimensional/pitched about the copy itself.

   What the driver actually records

   After GraphBuilder stream capture of dst.copy_from(src, stream=builder):

   Field	Value
   srcMemoryType	CU_MEMORYTYPE_UNIFIED (4)
   dstMemoryType	CU_MEMORYTYPE_UNIFIED (4)
   WidthInBytes	64
   Height / Depth	1 / 1
   All pitch/offset fields	0

   By contrast, a manually built GraphDefinition().memcpy(dst, src, 64) node gets CU_MEMORYTYPE_DEVICE on both sides — and MemcpyNode.update(size=32) succeeds there.

   So the error message is misleading for the common capture path: none of the pitched/offset/array conditions apply; the descriptor is a plain 1D device copy, just tagged UNIFIED because Buffer.copy_from lowers to cuMemcpyAsync under unified addressing.

   Driver accepts UNIFIED updates

   Bypassing cuda.core and calling cuGraphMemcpyNodeSetParams on the captured node (keeping UNIFIED types, changing WidthInBytes 64→32) succeeds, and the instantiated graph copies exactly 32 bytes on launch. The limitation is purely in cuda.core's guard, not the driver API.

   Suggested fix

   1. _is_supported_memcpy_descriptor() (cuda_core/cuda/core/graph/_subclasses.pyx ~237): allow CU_MEMORYTYPE_UNIFIED alongside HOST and DEVICE when all the existing 1D/unpitched checks pass:

   (params.srcMemoryType == cydriver.CU_MEMORYTYPE_HOST or
    params.srcMemoryType == cydriver.CU_MEMORYTYPE_DEVICE or
    params.srcMemoryType == cydriver.CU_MEMORYTYPE_UNIFIED)
   # same for dstMemoryType

   2. MemcpyNode.update() pointer extraction (~878–895): treat UNIFIED like DEVICE (CUDA docs: use srcDevice/dstDevice for UNIFIED):

   elif c_dst_type == cydriver.CU_MEMORYTYPE_UNIFIED:
       c_dst = params.memcpy.copyParams.dstDevice
   # ...
   elif c_src_type == cydriver.CU_MEMORYTYPE_UNIFIED:
       c_src = params.memcpy.copyParams.srcDevice

   3. Tests: add a GPU test that captures Buffer.copy_from via GraphBuilder, then calls MemcpyNode.update(size=…) — mirroring the issue repro. Optionally assert _create_from_driver / __repr__ still show sensible (D) labels for UNIFIED device pointers.

   This is a small, behavior-safe change: the same checks that reject truly multidimensional/pitched/offset copies still apply; only the memory-type filter was too narrow for stream-captured nodes.

   ---

   Probe

   Run on any CUDA GPU host with cuda.core built from main (CUDA 12.2+ for graph node updates):

   # after installing cuda.core from main on a GPU machine
   python3 scripts/issue2649-memcpy-node-probe.py
   # optional JSON output path
   ISSUE2649_PROBE_JSON=/tmp/issue2649-memcpy-node-probe.json python3 scripts/issue2649-memcpy-node-probe.py

   RunPod wrapper (creates pod, builds cuda.core from main + mini CUDA 13.3 CTK, runs probe):

   ./scripts/runpod-issue2649-probe.sh

   scripts/issue2649-memcpy-node-probe.py (full probe script)

   #!/usr/bin/env python3
   """Issue #2649 — MemcpyNode.update() rejects stream-captured memcpy nodes.
   
   Stream capture records cuMemcpyAsync with CU_MEMORYTYPE_UNIFIED on both operands
   (unified addressing). cuda.core's _is_supported_memcpy_descriptor() only allows
   HOST and DEVICE, so update() raises NotImplementedError on the common path.
   
   This probe:
     1. Reproduces the bug via GraphBuilder + Buffer.copy_from
     2. Dumps the driver-recorded CUDA_MEMCPY3D descriptor
     3. Compares GraphDefinition.memcpy (manual builder — works)
     4. Bypasses cuda.core and calls cuGraphNodeSetParams with a smaller size
     5. Verifies the updated graph executes correctly on GPU
   """
   
   from __future__ import annotations
   
   import json
   import os
   import subprocess
   import sys
   import traceback
   from datetime import datetime, timezone
   
   from cuda.bindings import driver
   
   
   def get_memcpy_params(node_handle):
       from cuda.core._utils.cuda_utils import handle_return
   
       return handle_return(driver.cuGraphMemcpyNodeGetParams(node_handle))
   
   
   def set_memcpy_params(node_handle, params):
       from cuda.core._utils.cuda_utils import handle_return
   
       handle_return(driver.cuGraphMemcpyNodeSetParams(node_handle, params))
   
   
   MEMORY_TYPE_NAMES = {
       int(driver.CUmemorytype.CU_MEMORYTYPE_HOST): "CU_MEMORYTYPE_HOST",
       int(driver.CUmemorytype.CU_MEMORYTYPE_DEVICE): "CU_MEMORYTYPE_DEVICE",
       int(driver.CUmemorytype.CU_MEMORYTYPE_ARRAY): "CU_MEMORYTYPE_ARRAY",
       int(driver.CUmemorytype.CU_MEMORYTYPE_UNIFIED): "CU_MEMORYTYPE_UNIFIED",
   }
   
   
   def check(name: str, err) -> None:
       if isinstance(err, driver.CUresult) and err != driver.CUresult.CUDA_SUCCESS:
           raise RuntimeError(f"{name} failed: {err}")
   
   
   def memory_type_name(value) -> str:
       return MEMORY_TYPE_NAMES.get(int(value), f"unknown({int(value)})")
   
   
   def descriptor_dict(params: driver.CUDA_MEMCPY3D) -> dict[str, object]:
       return {
           "srcMemoryType": memory_type_name(params.srcMemoryType),
           "srcMemoryType_raw": int(params.srcMemoryType),
           "dstMemoryType": memory_type_name(params.dstMemoryType),
           "dstMemoryType_raw": int(params.dstMemoryType),
           "srcDevice": int(params.srcDevice),
           "dstDevice": int(params.dstDevice),
           "srcHost": int(params.srcHost) if params.srcHost else 0,
           "dstHost": int(params.dstHost) if params.dstHost else 0,
           "WidthInBytes": int(params.WidthInBytes),
           "Height": int(params.Height),
           "Depth": int(params.Depth),
           "srcPitch": int(params.srcPitch),
           "dstPitch": int(params.dstPitch),
           "srcHeight": int(params.srcHeight),
           "dstHeight": int(params.dstHeight),
           "srcXInBytes": int(params.srcXInBytes),
           "dstXInBytes": int(params.dstXInBytes),
           "srcY": int(params.srcY),
           "dstY": int(params.dstY),
           "srcZ": int(params.srcZ),
           "dstZ": int(params.dstZ),
           "srcLOD": int(params.srcLOD),
           "dstLOD": int(params.dstLOD),
       }
   
   
   def is_supported_current(params: driver.CUDA_MEMCPY3D) -> bool:
       """Mirror _is_supported_memcpy_descriptor() in _subclasses.pyx (main)."""
       host = int(driver.CUmemorytype.CU_MEMORYTYPE_HOST)
       device = int(driver.CUmemorytype.CU_MEMORYTYPE_DEVICE)
       return (
           int(params.srcMemoryType) in (host, device)
           and int(params.dstMemoryType) in (host, device)
           and params.srcXInBytes == 0
           and params.srcY == 0
           and params.srcZ == 0
           and params.srcLOD == 0
           and params.srcPitch == 0
           and params.srcHeight == 0
           and params.dstXInBytes == 0
           and params.dstY == 0
           and params.dstZ == 0
           and params.dstLOD == 0
           and params.dstPitch == 0
           and params.dstHeight == 0
           and params.Height == 1
           and params.Depth == 1
       )
   
   
   def is_supported_proposed(params: driver.CUDA_MEMCPY3D) -> bool:
       """Proposed fix: treat UNIFIED like HOST/DEVICE for 1D unpitched copies."""
       host = int(driver.CUmemorytype.CU_MEMORYTYPE_HOST)
       device = int(driver.CUmemorytype.CU_MEMORYTYPE_DEVICE)
       unified = int(driver.CUmemorytype.CU_MEMORYTYPE_UNIFIED)
       allowed = (host, device, unified)
       return (
           int(params.srcMemoryType) in allowed
           and int(params.dstMemoryType) in allowed
           and params.srcXInBytes == 0
           and params.srcY == 0
           and params.srcZ == 0
           and params.srcLOD == 0
           and params.srcPitch == 0
           and params.srcHeight == 0
           and params.dstXInBytes == 0
           and params.dstY == 0
           and params.dstZ == 0
           and params.dstLOD == 0
           and params.dstPitch == 0
           and params.dstHeight == 0
           and params.Height == 1
           and params.Depth == 1
       )
   
   
   def nvidia_smi() -> str:
       try:
           return subprocess.check_output(["nvidia-smi"], text=True, stderr=subprocess.STDOUT)
       except Exception as exc:
           return f"(nvidia-smi unavailable: {exc})"
   
   
   def fill_buffer(buf, pattern: int, stream) -> None:
       buf.fill(pattern, stream=stream)
   
   
   def read_buffer_bytes(buf, nbytes: int, stream) -> list[int]:
       import ctypes
   
       from cuda.core import LegacyPinnedMemoryResource
   
       host = LegacyPinnedMemoryResource().allocate(nbytes)
       buf.copy_to(host, stream=stream)
       stream.sync()
       return list((ctypes.c_uint8 * nbytes).from_address(int(host.handle)))
   
   
   def main() -> int:
       from cuda.core import Device
       from cuda.core.graph import GraphDefinition, MemcpyNode
   
       import cuda.bindings
       import cuda.core
   
       print("=" * 72)
       print("Issue #2649 — MemcpyNode.update() stream-capture probe")
       print(f"Timestamp (UTC): {datetime.now(timezone.utc).isoformat()}")
       print("=" * 72)
   
       print("\n--- Environment ---")
       print(f"CUDA_HOME={os.environ.get('CUDA_HOME', '(unset)')}")
       print(f"CUDA_CORE_BUILD_MAJOR={os.environ.get('CUDA_CORE_BUILD_MAJOR', '(unset)')}")
       print(f"cuda-bindings={cuda.bindings.__version__}")
       print(f"cuda.core={cuda.core.__version__}")
   
       print("\n--- GPU / driver ---")
       print(nvidia_smi())
   
       report: dict[str, object] = {
           "issue": "NVIDIA/cuda-python#2649",
           "timestamp_utc": datetime.now(timezone.utc).isoformat(),
           "bindings_version": cuda.bindings.__version__,
           "cuda_core_version": cuda.core.__version__,
           "tests": {},
       }
   
       dev = Device()
       dev.set_current()
       stream = dev.create_stream()
   
       alloc_size = 64
       src = dev.memory_resource.allocate(alloc_size, stream=stream)
       dst = dev.memory_resource.allocate(alloc_size, stream=stream)
       fill_buffer(src, 0xAB, stream)
       fill_buffer(dst, 0x00, stream)
       stream.sync()
   
       print("\n--- A. Stream capture (GraphBuilder) — issue repro ---")
       builder = dev.create_graph_builder().begin_building()
       dst.copy_from(src, stream=builder)
       builder.end_building()
       captured_node = next(
           n for n in builder.graph_definition.nodes() if isinstance(n, MemcpyNode)
       )
       print(f"Captured node: {captured_node!r}")
   
       params_raw = get_memcpy_params(captured_node.handle)
       desc = descriptor_dict(params_raw)
       print("Driver-recorded descriptor:")
       for key, value in desc.items():
           print(f"  {key}: {value}")
   
       current_ok = is_supported_current(params_raw)
       proposed_ok = is_supported_proposed(params_raw)
       print(f"_is_supported_memcpy_descriptor (current):  {current_ok}")
       print(f"_is_supported_memcpy_descriptor (proposed): {proposed_ok}")
   
       update_error = None
       if not hasattr(captured_node, "update"):
           update_result = "missing_update_method"
           update_error = (
               "MemcpyNode.update() not in this cuda.core build "
               f"(installed {cuda.core.__version__}); need main branch or >=1.2"
           )
           print(f"MemcpyNode.update(size=32): SKIPPED — {update_error}")
       else:
           try:
               captured_node.update(size=32)
               update_result = "unexpected_success"
           except NotImplementedError as exc:
               update_error = str(exc)
               update_result = "NotImplementedError"
               print(f"MemcpyNode.update(size=32): {update_result}")
               print(f"  message: {update_error}")
           except Exception as exc:
               update_error = f"{type(exc).__name__}: {exc}"
               update_result = type(exc).__name__
               print(f"MemcpyNode.update(size=32): {update_error}")
   
       report["tests"]["stream_capture"] = {
           "node_repr": repr(captured_node),
           "descriptor": desc,
           "is_supported_current": current_ok,
           "is_supported_proposed": proposed_ok,
           "update_result": update_result,
           "update_error": update_error,
       }
   
       print("\n--- B. Manual GraphDefinition.memcpy (control — should work) ---")
       src2 = dev.memory_resource.allocate(alloc_size, stream=stream)
       dst2 = dev.memory_resource.allocate(alloc_size, stream=stream)
       fill_buffer(src2, 0xCD, stream)
       fill_buffer(dst2, 0x00, stream)
       stream.sync()
   
       graph_def = GraphDefinition()
       manual_node = graph_def.memcpy(dst2, src2, 64)
       print(f"Manual node: {manual_node!r}")
   
       params_manual_raw = get_memcpy_params(manual_node.handle)
       desc_manual = descriptor_dict(params_manual_raw)
       print("Manual descriptor:")
       for key, value in desc_manual.items():
           print(f"  {key}: {value}")
   
       manual_update_ok = False
       if not hasattr(manual_node, "update"):
           print("Manual MemcpyNode.update(size=32): SKIPPED — update() not in this build")
       else:
           try:
               manual_node.update(size=32)
               manual_update_ok = True
               print("Manual MemcpyNode.update(size=32): SUCCESS")
           except Exception as exc:
               print(f"Manual MemcpyNode.update(size=32): FAILED — {exc}")
   
       report["tests"]["manual_graph_definition"] = {
           "node_repr": repr(manual_node),
           "descriptor": desc_manual,
           "is_supported_current": is_supported_current(params_manual_raw),
           "update_result": "success" if manual_update_ok else "failed",
       }
   
       print("\n--- C. Driver bypass — cuGraphNodeSetParams on captured node ---")
       driver_update_ok = False
       exec_ok = False
       exec_bytes: list[int] | None = None
       driver_update_error = None
   
       try:
           # Fresh capture so prior failed update did not mutate state.
           src3 = dev.memory_resource.allocate(alloc_size, stream=stream)
           dst3 = dev.memory_resource.allocate(alloc_size, stream=stream)
           fill_buffer(src3, 0x5A, stream)
           fill_buffer(dst3, 0x00, stream)
           stream.sync()
   
           gb = dev.create_graph_builder().begin_building()
           dst3.copy_from(src3, stream=gb)
           gb.end_building()
           node3 = next(n for n in gb.graph_definition.nodes() if isinstance(n, MemcpyNode))
   
           p = get_memcpy_params(node3.handle)
           original_width = int(p.WidthInBytes)
           p.WidthInBytes = 32
           set_memcpy_params(node3.handle, p)
           driver_update_ok = True
           print(
               f"cuGraphMemcpyNodeSetParams WidthInBytes {original_width} -> 32: SUCCESS "
               f"(memory types unchanged: {memory_type_name(p.srcMemoryType)}, "
               f"{memory_type_name(p.dstMemoryType)})"
           )
   
           graph = gb.graph_definition.instantiate()
           graph.launch(stream)
           stream.sync()
           exec_ok = True
           exec_bytes = read_buffer_bytes(dst3, 64, stream)
           print(f"Post-launch dst first 32 bytes: {exec_bytes[:32]}")
           print(f"Post-launch dst bytes 32-63:    {exec_bytes[32:]}")
       except Exception as exc:
           driver_update_error = traceback.format_exc()
           print(f"Driver bypass FAILED:\n{driver_update_error}")
   
       report["tests"]["driver_bypass"] = {
           "cuGraphMemcpyNodeSetParams_ok": driver_update_ok,
           "graph_launch_ok": exec_ok,
           "dst_first_32_bytes": exec_bytes[:32] if exec_bytes else None,
           "dst_tail_bytes": exec_bytes[32:] if exec_bytes else None,
           "error": driver_update_error,
       }
   
       print("\n" + "=" * 72)
       print("VERDICT")
       print("=" * 72)
       bug_confirmed = (
           not current_ok
           and proposed_ok
           and desc.get("srcMemoryType") == "CU_MEMORYTYPE_UNIFIED"
           and (
               update_result == "NotImplementedError"
               or update_result == "missing_update_method"
           )
       )
       print(f"  Stream-captured descriptor uses UNIFIED: "
             f"{desc.get('srcMemoryType') == 'CU_MEMORYTYPE_UNIFIED'}")
       print(f"  cuda.core update() rejects captured node: "
             f"{update_result in ('NotImplementedError', 'missing_update_method')}")
       print(f"  Manual GraphDefinition.update() works: {manual_update_ok}")
       print(f"  Driver accepts UNIFIED descriptor update: {driver_update_ok}")
       print(f"  Updated graph executes on GPU: {exec_ok}")
       if exec_bytes:
           copied = exec_bytes[:32] == [0x5A] * 32
           untouched = exec_bytes[32:] == [0] * 32
           print(f"  Copy size honored (32 bytes 0x5A, tail zero): {copied and untouched}")
       print()
       print("  Root cause: _is_supported_memcpy_descriptor() omits CU_MEMORYTYPE_UNIFIED.")
       print("  Suggested fix: allow UNIFIED for 1D unpitched descriptors; read/write")
       print("  srcDevice/dstDevice for UNIFIED in MemcpyNode.update().")
       print("=" * 72)
   
       report["verdict"] = {
           "bug_confirmed": bug_confirmed,
           "root_cause": "_is_supported_memcpy_descriptor rejects CU_MEMORYTYPE_UNIFIED",
           "driver_accepts_unified_update": driver_update_ok,
           "proposed_fix_viable": driver_update_ok and exec_ok,
       }
   
       out_json = os.environ.get(
           "ISSUE2649_PROBE_JSON", "/tmp/issue2649-memcpy-node-probe.json"
       )
       with open(out_json, "w", encoding="utf-8") as fh:
           json.dump(report, fh, indent=2)
       print(f"\nJSON report: {out_json}")
   
       return 0 if bug_confirmed else 1
   
   
   if __name__ == "__main__":
       sys.exit(main())

   RunPod probe log (2026-08-20T17:40:09Z, RTX 4090, driver 560.35.03)

   Issue #2649 — MemcpyNode.update() stream-capture probe
   Timestamp (UTC): 2026-08-20T17:40:09.278789+00:00
   ========================================================================
   
   --- Environment ---
   CUDA_HOME=/tmp/cuda_toolkit
   CUDA_CORE_BUILD_MAJOR=13
   cuda-bindings=13.3.1
   cuda.core=0.0.0
   
   --- GPU / driver ---
   Thu Aug 20 17:40:09 2026       
   +-----------------------------------------------------------------------------------------+
   | NVIDIA-SMI 560.35.03              Driver Version: 560.35.03      CUDA Version: 12.6     |
   |-----------------------------------------+------------------------+----------------------+
   | GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
   | Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
   |                                         |                        |               MIG M. |
   |=========================================+========================+======================|
   |   0  NVIDIA GeForce RTX 4090        On  |   00000000:C2:00.0 Off |                  Off |
   |  0%   34C    P8             13W /  450W |       2MiB /  24564MiB |      0%      Default |
   |                                         |                        |                  N/A |
   +-----------------------------------------+------------------------+----------------------+
                                                                                            
   +-----------------------------------------------------------------------------------------+
   | Processes:                                                                              |
   |  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
   |        ID   ID                                                               Usage      |
   |=========================================================================================|
   |  No running processes found                                                             |
   +-----------------------------------------------------------------------------------------+
   
   
   --- A. Stream capture (GraphBuilder) — issue repro ---
   Captured node: <MemcpyNode handle=0x379d570 dst=0x302000200(D) src=0x302000000(D) size=64>
   Driver-recorded descriptor:
     srcMemoryType: CU_MEMORYTYPE_UNIFIED
     srcMemoryType_raw: 4
     dstMemoryType: CU_MEMORYTYPE_UNIFIED
     dstMemoryType_raw: 4
     srcDevice: 12918456320
     dstDevice: 12918456832
     srcHost: 0
     dstHost: 0
     WidthInBytes: 64
     Height: 1
     Depth: 1
     srcPitch: 0
     dstPitch: 0
     srcHeight: 0
     dstHeight: 0
     srcXInBytes: 0
     dstXInBytes: 0
     srcY: 0
     dstY: 0
     srcZ: 0
     dstZ: 0
     srcLOD: 0
     dstLOD: 0
   _is_supported_memcpy_descriptor (current):  False
   _is_supported_memcpy_descriptor (proposed): True
   MemcpyNode.update(size=32): NotImplementedError
     message: updating multidimensional, pitched, offset, or array-backed memcpy nodes is not supported
   
   --- B. Manual GraphDefinition.memcpy (control — should work) ---
   Manual node: <MemcpyNode handle=0x379ea40 dst=0x302000600(D) src=0x302000400(D) size=64>
   Manual descriptor:
     srcMemoryType: CU_MEMORYTYPE_DEVICE
     srcMemoryType_raw: 2
     dstMemoryType: CU_MEMORYTYPE_DEVICE
     dstMemoryType_raw: 2
     srcDevice: 12918457344
     dstDevice: 12918457856
     srcHost: 0
     dstHost: 0
     WidthInBytes: 64
     Height: 1
     Depth: 1
     srcPitch: 0
     dstPitch: 0
     srcHeight: 0
     dstHeight: 0
     srcXInBytes: 0
     dstXInBytes: 0
     srcY: 0
     dstY: 0
     srcZ: 0
     dstZ: 0
     srcLOD: 0
     dstLOD: 0
   Manual MemcpyNode.update(size=32): SUCCESS
   
   --- C. Driver bypass — cuGraphNodeSetParams on captured node ---
   cuGraphMemcpyNodeSetParams WidthInBytes 64 -> 32: SUCCESS (memory types unchanged: CU_MEMORYTYPE_UNIFIED, CU_MEMORYTYPE_UNIFIED)
   Post-launch dst first 32 bytes: [90, 90, 90, 90, 90, 90, 90, 90, 90, 90, 90, 90, 90, 90, 90, 90, 90, 90, 90, 90, 90, 90, 90, 90, 90, 90, 90, 90, 90, 90, 90, 90]
   Post-launch dst bytes 32-63:    [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
   
   ========================================================================
   VERDICT
   ========================================================================
     Stream-captured descriptor uses UNIFIED: True
     cuda.core update() rejects captured node: True
     Manual GraphDefinition.update() works: True
     Driver accepts UNIFIED descriptor update: True
     Updated graph executes on GPU: True
     Copy size honored (32 bytes 0x5A, tail zero): True
   
     Root cause: _is_supported_memcpy_descriptor() omits CU_MEMORYTYPE_UNIFIED.
     Suggested fix: allow UNIFIED for 1D unpitched descriptors; read/write
     srcDevice/dstDevice for UNIFIED in MemcpyNode.update().
   ========================================================================
   

   issue2649-memcpy-node-probe.json (full probe report — attached)

   {
     "issue": "NVIDIA/cuda-python#2649",
     "timestamp_utc": "2026-08-20T17:40:09.426004+00:00",
     "bindings_version": "13.3.1",
     "cuda_core_version": "0.0.0",
     "tests": {
       "stream_capture": {
         "node_repr": "<MemcpyNode handle=0x379d570 dst=0x302000200(D) src=0x302000000(D) size=64>",
         "descriptor": {
           "srcMemoryType": "CU_MEMORYTYPE_UNIFIED",
           "srcMemoryType_raw": 4,
           "dstMemoryType": "CU_MEMORYTYPE_UNIFIED",
           "dstMemoryType_raw": 4,
           "srcDevice": 12918456320,
           "dstDevice": 12918456832,
           "srcHost": 0,
           "dstHost": 0,
           "WidthInBytes": 64,
           "Height": 1,
           "Depth": 1,
           "srcPitch": 0,
           "dstPitch": 0,
           "srcHeight": 0,
           "dstHeight": 0,
           "srcXInBytes": 0,
           "dstXInBytes": 0,
           "srcY": 0,
           "dstY": 0,
           "srcZ": 0,
           "dstZ": 0,
           "srcLOD": 0,
           "dstLOD": 0
         },
         "is_supported_current": false,
         "is_supported_proposed": true,
         "update_result": "NotImplementedError",
         "update_error": "updating multidimensional, pitched, offset, or array-backed memcpy nodes is not supported"
       },
       "manual_graph_definition": {
         "node_repr": "<MemcpyNode handle=0x379ea40 dst=0x302000600(D) src=0x302000400(D) size=32>",
         "descriptor": {
           "srcMemoryType": "CU_MEMORYTYPE_DEVICE",
           "srcMemoryType_raw": 2,
           "dstMemoryType": "CU_MEMORYTYPE_DEVICE",
           "dstMemoryType_raw": 2,
           "srcDevice": 12918457344,
           "dstDevice": 12918457856,
           "srcHost": 0,
           "dstHost": 0,
           "WidthInBytes": 64,
           "Height": 1,
           "Depth": 1,
           "srcPitch": 0,
           "dstPitch": 0,
           "srcHeight": 0,
           "dstHeight": 0,
           "srcXInBytes": 0,
           "dstXInBytes": 0,
           "srcY": 0,
           "dstY": 0,
           "srcZ": 0,
           "dstZ": 0,
           "srcLOD": 0,
           "dstLOD": 0
         },
         "is_supported_current": true,
         "update_result": "success"
       },
       "driver_bypass": {
         "cuGraphMemcpyNodeSetParams_ok": true,
         "graph_launch_ok": true,
         "dst_first_32_bytes": [
           90,
           90,
           90,
           90,
           90,
           90,
           90,
           90,
           90,
           90,
           90,
           90,
           90,
           90,
           90,
           90,
           90,
           90,
           90,
           90,
           90,
           90,
           90,
           90,
           90,
           90,
           90,
           90,
           90,
           90,
           90,
           90
         ],
         "dst_tail_bytes": [
           0,
           0,
           0,
           0,
           0,
           0,
           0,
           0,
           0,
           0,
           0,
           0,
           0,
           0,
           0,
           0,
           0,
           0,
           0,
           0,
           0,
           0,
           0,
           0,
           0,
           0,
           0,
           0,
           0,
           0,
           0,
           0
         ],
         "error": null
       }
     },
     "verdict": {
       "bug_confirmed": true,
       "root_cause": "_is_supported_memcpy_descriptor rejects CU_MEMORYTYPE_UNIFIED",
       "driver_accepts_unified_update": true,
       "proposed_fix_viable": true
     }
   }
4. rparolin

   added this to the cuda.core 1.3.0 milestone on Sep 24, 2026
5. rparolin

   added

   P1Medium priority - Should doMedium priority - Should do

   and removed

   triageNeeds the team's attentionNeeds the team's attention

   on Sep 24, 2026
6. rparolin

   assigned

   leofang

   and

   Andy-Jost

   and unassigned

   leofang

   on Sep 24, 2026
7. rparolin

   added the

   Bug

   issue type on Sep 24, 2026
8. Andy-Jost

   closed this as completedin #3029on Oct 6, 2026
9. mdboom

   added a commit that references this issue on Oct 6, 2026

   cuda.core: accept stream-captured memcpy nodes in MemcpyNode.update (N…

   ...

   This issue will close once commit 4c893de is merged into the 'main' branch.

   4c893de