memory_ipc/test_send_buffers.py::TestIpcReexport::test_main[DeviceMR] hanging in CI

I manually canceled this job when it reached 58m 42s:

Test linux-64 / py3.12, 12.9.1, local, l4

It was hanging here, for almost 54 minutes:

2026-05-01T05:08:24.6867513Z tests/memory_ipc/test_send_buffers.py::TestIpcSendBuffers::test_main[3] PASSED [ 13%]
2026-05-01T06:02:21.2264769Z tests/memory_ipc/test_send_buffers.py::TestIpcReexport::test_main[DeviceMR]
2026-05-01T06:02:21.2758059Z ##[error]The operation was canceled.

A rerun worked, i.e. the job appears to be flaky.

I'm attaching the log for easy future reference:

job-logs.txt

Here is another case (this one hung for almost 4 hours):

Test linux-64 / py3.12, 12.9.1, local, l4

2026-05-06T00:45:35.6404725Z tests/memory_ipc/test_send_buffers.py::TestIpcReexport::test_main[DeviceMR-3-5] PASSED [ 12%]
2026-05-06T03:39:05.1997826Z tests/memory_ipc/test_send_buffers.py::TestIpcReexport::test_main[DeviceMR-4-5] 
2026-05-06T03:39:05.2534642Z ##[error]The operation was canceled.

job-logs.txt

Here is another case (this one timed out after blocking the runner for 6 hours):

Test linux-aarch64 / Python 3.12, CUDA 12.9.1 (local), GPU a100

2026-05-16T06:54:28.5249320Z tests/memory_ipc/test_send_buffers.py::TestIpcSendBuffers::test_main[3] PASSED [ 11%]
2026-05-16T12:50:09.4792178Z tests/memory_ipc/test_send_buffers.py::TestIpcReexport::test_main[DeviceMR]
2026-05-16T12:50:09.5126427Z ##[error]The operation was canceled.

job-logs.txt

Two more cases, this time in the same CI invocation:

Test linux-64 / Python 3.12, CUDA 12.9.1 (local), GPU l4

Test linux-aarch64 / Python 3.12, CUDA 12.9.1 (local), GPU a100

2026-05-21T12:06:12.5290875Z tests/memory_ipc/test_send_buffers.py::TestIpcSendBuffers::test_main[3] PASSED [ 12%]
2026-05-21T15:24:39.2279823Z tests/memory_ipc/test_send_buffers.py::TestIpcReexport::test_main[DeviceMR]
2026-05-21T15:24:39.2340408Z ##[error]The operation was canceled.

2026-05-21T11:58:33.1451246Z tests/memory_ipc/test_send_buffers.py::TestIpcSendBuffers::test_main[3] PASSED [ 12%]
2026-05-21T15:24:51.6896792Z tests/memory_ipc/test_send_buffers.py::TestIpcReexport::test_main[DeviceMR]
2026-05-21T15:24:51.7026263Z ##[error]The operation was canceled.

The pattern is very clear now: It's always a combination of Python 3.12 and CUDA 12.9.1

issue2004_job-logs_2026-05-21+082459.txt

issue2004_job-logs_2026-05-21+082449.txt

tests: kill zombie IPC child processes after join timeout

The same jobs are hanging again:

Test linux-64 / Python 3.12, CUDA 12.9.1 (local), GPU l4

Test linux-aarch64 / Python 3.12, CUDA 12.9.1 (local), GPU a100

tests: kill zombie IPC child processes after join timeout

tests: extend kill-after-join to remaining memory_ipc tests

tests: layered defense against IPC child-process hangs (#2004) (#2124)

Navigation Menu

Repository navigation

memory_ipc/test_send_buffers.py::TestIpcReexport::test_main[DeviceMR] hanging in CI #2004

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

memory_ipc/test_send_buffers.py::TestIpcReexport::test_main[DeviceMR] hanging in CI #2004

Description

Activity

rwgk commented on May 6, 2026

rwgk commented on May 16, 2026

rwgk commented on May 21, 2026

rwgk commented on May 21, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions