fix(core/kv): order host-KV copy streams behind the caller's default stream - #313

Description

Order the host-KV worker view's copy streams behind the caller's stream by recording a CUDA event on the issuing thread at call time (RecordProducerEvent) and waiting on it inside the copy task (WaitForProducerEvent). Applied to all five worker-view copy paths (two h2d page loads, the prefill offload, the per-layer decode append, the batched decode append) and the three compressed_state_host_manager.h paths that shared the old helper. Adds a GPU regression test that fails on main and a static contract test.

Motivation

The copy streams come from at::cuda::getStreamFromPool (cudaStreamNonBlocking). The worker runs the model on PyTorch's default stream, whose cudaStream_t handle is 0, and WaitForProducerStream returned early for a null producer handle — so no event ever ordered a copy behind the model's stream:

• a d2h prefill offload issued right after the kernel producing its source copied the buffer's previous contents (Kimi-K3: the first sequence offloaded per MLA layer; finite-but-wrong KV with warm buffers, NaN garbage after a phase re-configure freed them);
• a d2h decode append could read the token's K before it was written;
• an h2d page load could land before the torch.zeros memset of a re-created K cache and be wiped.

Measured on the Kimi-K3 512-request decode contract (2x8 H200): host readback != source for the first sequence of every admission wave on all 8 ranks; a collapsed row's host page == GPU page word-for-word with garbage only in the tail of one page (a partially written buffer). MMLU 64% vs 87% for the same prompts on SGLang. With the fix: 0/512 collapsed on the 512x128 probe (was 21/512) and MMLU 90.23% (462/512) on the frozen contract corpus.

Type of Change

• model — add/extend model support (models/** + registration seam + model kernels only)
• kernel — add/optimize a compute kernel (batchgen_kernels/** + in-tree kernel dirs)
• core — change scheduling/serving/runtime scaffolding (the only type that may)
• fix — narrow bug fix (+ a regression test)
• infra — build / CI / packaging / scripts / Docker
• docs — documentation only

File changes

Checklist

• I have read the CONTRIBUTING guide and the PR Merge Policy Contract.
• I have updated the tests (if applicable).
• I have updated the documentation (if applicable).

PR Merge Policy Contract — pre-merge checklist

• git diff --stat origin/main reviewed; every file traces to the task — no unrelated files (§3.1).
• Exactly one Type of Change ticked above; all changed files are within that type's permitted set (§2.5) — no scaffolding edits in a model/kernel PR (§2.6).
• File changes table lists every changed file and matches the diff (audit-able at draft; kept current).
• No debug_* / check_* / scratch_* / tmp_* scripts added to a production package or repo root (§1.1). (bench_* is allowed.)
• No test_*.py added inside the runtime package; tests are under tests/ (§1.2, §2.1).
• No new BATCHGEN_* env-var debug guard; debug behavior is a batchgen_debug batch flag (§1.3). A genuinely-needed new env var goes in its own PR.
• No leftover print() or commented-out code in the changed files (§1.4–§1.5). (logging.debug() is fine.)
• No logs, profiler traces, CSVs, checkpoints, wheels, or compiled artifacts staged; nothing added with -f (§1.6).
• Touched modules' MODULE.md updated if the public API changed (§2.2). (No public API change.)
• One concern only; diff is surgical (§3).
• Commits are clean: Angular format, body present, no co-author trailers (§4).
• CI is green (format, lint, hygiene, tests).

bash .github/workflows/scripts/check-pr-hygiene.sh origin/main (PR_TYPE=fix): 0 blocking, 0 advisory.

🤖 Generated with Claude Code