Repository navigation
ggml-cuda: generic DiT and VAE fusions for every family, plus a rope PDL race fix - #27
Conversation
…parallel group norm, channel rms norm, mul+add) Adds ggml patch 0007 on top of 0001-0006: - rope_pe_permute gains an interleaved pairing mode (Z-Image, Flux, Chroma rope) - rope_apply_f32 waits on the PDL predecessor (fixes nondeterministic output on sm90+) - parallel GROUP_NORM for groups >= 1024 elements, fused with the affine MUL/ADD and SILU - channel RMS norm fusion for CONT(PERMUTE) -> RMS_NORM -> MUL -> PERMUTE -> CONT (Wan/Qwen-Image VAE) - MUL -> ADD [-> ADD] broadcast fusion for modulation All fusions reproduce the stock rounding and reduction order, so outputs are bit-identical. Each has a kill switch (GGML_CUDA_GROUP_NORM_PAR, GGML_CUDA_NORM_CHANNEL, GGML_CUDA_FUSE_MUL_ADD). cmake detects the interleaved setter and defines SD_GGML_FUSED_ROPE_ATTN.
…conts Rope::attention builds q/k/v through rope_pe_permute (interleaved mode) when flash attention is on and the shapes match (Z-Image, Flux, Flux2, Chroma, Qwen-Image), falling back to the existing path otherwise. ggml_ext_attention_prepared accepts an optional mask. SD_FUSED_ROPE_ATTN=0 disables it. Conv2d im2col output and Wan Conv2dBut3d skip the final cont when the permute is a no-op (N == 1 / T == 1).
…chain The fused kernel walks every channel in one thread per position, so it is slower than CONT + RMS_NORM + MUL + CONT on small layers (VAE tiles). Fuse from 16384 positions when C < 1024 and from 65536 when C >= 1024; GGML_CUDA_NORM_CHANNEL=2 forces it at every size. Adds VAE-shaped perf cases for RMS_NORM_CHANNEL.
|
@codex review |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f170601cf8
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| + const char * x0 = (const char *) x->data; | ||
| + const char * d0 = (const char *) cont2->data; | ||
| + const bool partial_overlap = d0 != x0 && d0 < x0 + ggml_nbytes(x) && x0 < d0 + ggml_nbytes(cont2); | ||
| + if (P > INT64_C(1) << 40 || partial_overlap) { |
There was a problem hiding this comment.
Reject output buffers that overlap the weight
When w is produced inside the graph and the MUL is its final consumer, the allocator may reuse its buffer for cont2, because the unfused final CONT runs only after the MUL has finished reading w. This fusion instead launches at the first CONT and reads w while writing cont2; the overlap check covers only x, so such a reuse corrupts the normalization output. Use ggml_cuda_check_fusion_memory_ranges or explicitly reject overlap between cont2 and w before launching.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in aeb58f0: the channel RMS norm fusion now also rejects any overlap between its output (the second CONT) and the weight w, alongside the existing partial-overlap check against x, so a reused w buffer falls back to the unfused chain.
|
@codex review |
1 similar comment
|
@codex review |
|
Codex Review: Didn't find any major issues. 👍 Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
…odes outside a selected graph branch - Rope::attention's fused RoPE + flash path now applies ctx->attn_scale like ggml_ext_attention_ext, so --attn-scale keeps large K/V finite in F16 on both paths and they stay bit-identical - ggml_ext_attention_prepared: ggml_prec_set_acc instead of the deprecated ggml_flash_attn_ext_set_prec - 0007: the MUL+ADD(+ADD) and channel RMS norm matchers require GGML_TENSOR_FLAG_COMPUTE on every node they consume (graphs from ggml_build_forward_select), as ggml_can_fuse_ext does for GROUP_NORM - 0007: CPU interleaved ROPE_PE_PERMUTE evaluates the same expression as ROPE_APPLY so FMA contraction cannot make them differ - 0007: tests/test-fusion-select.cpp covers the selected-branch graphs and the CPU RoPE parity
|
@codex review |
1 similar comment
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 0f53e72072
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| + set(TEST_TARGET test-fusion-select) | ||
| + add_executable(${TEST_TARGET} ${TEST_TARGET}.cpp) |
There was a problem hiding this comment.
Remove test code from the committed patch
This hunk adds test-fusion-select.cpp to the patched ggml test suite, and the same patch also extends test-backend-ops.cpp; however, the repository's CONTRIBUTING.md explicitly prohibits including test code or test scripts in commits or PRs and requires keeping them local. Remove these test-related patch hunks while retaining the verification results in the PR description.
AGENTS.md reference: AGENTS.md:L120-L122
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Removed in 3fb027e: 0007 no longer touches tests/ (the test-backend-ops cases and test-fusion-select are kept locally), and the results are in the PR description. 0001 to 0009 still apply in order on a fresh ggml checkout.
|
@codex review |
1 similar comment
|
@codex review |
|
Codex Review: Didn't find any major issues. Nice work! Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
- fused GroupNorm and channel RMS norm assert eps >= 0 like the unfused ops - channel RMS norm: both CONTs must keep their PERMUTE's shape (a CONT_4D may reshape) - GroupNorm affine: w and b may not overlap the fused output, x only element for element (the generic range check skips leaf sources) - GroupNorm affine (+ SILU): accept the in-place chain sd.cpp builds (MUL / ADD / SILU are whole views of the GROUP_NORM result), keeping the COMPUTE, OUTPUT and use-count checks
|
@codex review |
|
Codex Review: Didn't find any major issues. Another round soon, please! Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
Problem
The CUDA levers from #17 to #25 were tuned on H3 and mostly sit in model-specific paths. Profiling the other Studio families on an RTX PRO 6000 Blackwell (sm120) and an L4 (sm89) showed where time still goes:
x*s + ychains add more passes.rope_apply_f32did not wait on its programmatic-dependent-launch predecessor. On sm90+ this made Z-Image renders differ between identical runs (PSNR 17 to 21 dB between two runs of the same seed on sm120). WithGGML_CUDA_PDL=0the output is stable, which pins the race.Change
A new ggml patch
scripts/unsloth/ggml-patches/0007-ggml-cuda-generic-dit-vae.patch(applied after 0001 to 0006) plus small sd.cpp changes. Every fusion reproduces the stock rounding and reduction order, so outputs are bit-identical to the ops they replace. Each has a kill switch (=0).Rope::attentionbuilds q / k / v with onerope_pe_permuteop each (interleaved pairing mode added to the op, CPU and CUDA). Falls back to the old path for non-interleaved RoPE, no flash attention, sage, CPU backends, head dim below 64, or a backend that does not support the opRope::attentionDiTsSD_FUSED_ROPE_ATTNGGML_CUDA_GROUP_NORM_PARGGML_CUDA_NORM_CHANNEL(=2forces every size)GGML_CUDA_FUSE_MUL_ADDggml_cuda_pdl_sync()at the top ofrope_apply_f32ggml_ext_conv_2d(im2col path) and WanConv2dBut3dskip a finalcontthat moves no data (batch 1 / single frame)cmake/ggml.cmakedetectsggml_rope_pe_permute_set_interleavedand definesSD_GGML_FUSED_ROPE_ATTN, so an unpatched ggml keeps the old graph. The switches are documented indocs/performance.md.Results
Base = master (afac7a7) with patches 0001 to 0006, head = this branch, both built on the same VM in the same session. Studio defaults (
--diffusion-fa --diffusion-conv-direct), seed 42, family default steps. Seconds, base runs then head runs, interleaved ABBA. Accuracy column compares decoded pixels of base and head.RTX PRO 6000 Blackwell Server (sm120, Colab G4)
GGML_CUDA_PDL=0; base itself is nondeterministic (PDL race), head is deterministic--vae-tiling7x7--vae-tiling7x7L4 (sm89, Colab)
The L4 Qwen-Image-2.1 row was measured before the channel RMS norm fusion engaged on that VAE, so its decode gain is not included there.
Summary: DiT sampling 5 to 7% faster on Z-Image (both GPUs), 4 to 5% on FLUX.2-klein and FLUX.1 (FLUX.1 on sm120 is within noise), 1% on Qwen-Image-2.1, flat on the U-Net families. VAE decode 4 to 9% faster on KL VAEs, 15 to 20% on the Qwen-Image-2.1 VAE untiled and 5% tiled.
MiniMax-H3 and Wan 2.2, B200 (sm_100)
Base afac7a7 (patches 0001 to 0006) vs head (0001 to 0007), each applied to a clean ggml checkout, CI CUDA flags, exclusive card, interleaved ABBA, fresh process per run. H3: unsloth/MiniMax-H3-GGUF fl2va Q2_K with Studio's flags (
--mode vid_gen --cfg-scale 1.0 --rng cpu --fps 24 --diffusion-fa), seed 11. Wan: Wan2.2-TI2V-5B Q4_K_M, euler, CFG 6, flow shift 3. Medians.GGML_CUDA_PDL=0; base default differs run to run (8 of 8 distinct)Tests
test-backend-opson CUDA vs CPU (sm120), with local test cases for the new ops (not part of the patch): ROPE_PE_PERMUTE 120/120 (new interleaved cases), GROUP_NORM_AFFINE 16/16 (affine, SILU and reshape variants), RMS_NORM_CHANNEL 24/24 (C up to 2100, also withGGML_CUDA_NORM_CHANNEL=2on sm100), MUL_ADD_CHAIN 12/12, GROUP_NORM, GROUP_NORM_MUL_ADD 2/2, FLASH_ATTN_EXT hsk=128 577/577.GGML_CUDA_NORM_CHANNEL_DEBUG=1all 36 channel RMS norms of the Qwen-Image-2.1 decode match the pattern and the 19 above the size gate fuse (untiled 1024); the kill-switch runs show the timing gains disappear with the switches off.0the head output is identical to base.GGML_NATIVE=OFF, the prebuilt CPU flags) of this branch compiles with all seven patches applied, and SD 1.5 at 256x256, 4 steps, seed 42 on CPU gives the same pixels as master (max abs diff 0).Review fixes (0f53e72)
--attn-scaleoverride like the unfused path. Before, FLUX.2-klein with--attn-scale 0.0078125gave the same pixels as no override on the fused path; now fused and unfused both give the unfused path's image, and without the override the image is unchanged.GGML_TENSOR_FLAG_COMPUTEon every node they consume, as the GROUP_NORM matcher already did, so a graph built withggml_build_forward_selectnever has a selected output skipped. sd.cpp does not build such graphs today.ROPE_PE_PERMUTEevaluates the same expression asROPE_APPLY, so FMA contraction cannot make them differ (6 of 12 F32 cases differed bitwise before on an-mfmabuild, 12 of 12 match now).ggml_prec_set_accreplaces the deprecated precision setter.test-fusion-selectprogram (kept out of the patch, per CONTRIBUTING.md): selected-branch graphs on CPU and CUDA must match an expand-only graph bitwise and leave inactive outputs untouched, plus CPU RoPE parity. The previous head fails 16 of 40 on a B200, this head passes 40 of 40 (26 of 26 on a CPU-only build).test-backend-opson the 0007 ops: 158 of 158 on both heads. FLUX.2-klein and SD1.5 renders are pixel-identical between the two heads, and 0001 to 0009 apply in order on a fresh ggml checkout.Review fixes (f017bf3)
eps >= 0like the unfused ops (a negative eps used to produce NaNs on the fused channel RMS path).GGML_CUDA_GROUP_NORM_PAR=0matches too. The localtest-fusion-selectprogram goes from 46 of 48 to 48 of 48, andtest-backend-opson the 0007 ops plus GROUP_NORM passes 162 of 162 on both heads. 0001 to 0009 apply in order on a fresh ggml checkout.Not covered
Rope::attentionfalls back when the op is unsupported.