CPU bundles: runtime-selected ggml CPU variants and repacked quantized weights (1.3x to 3x faster on CPU) - #28

Problem

Every CPU path in the prebuilt bundles ran a fraction of the CPU it was on.

• One instruction set level for every host. The Linux x64, Linux arm64, macOS x86_64 and Windows CPU bundles are static -DGGML_NATIVE=OFF builds. On x86 that is ggml's fixed AVX/AVX2/FMA/F16C/BMI2 baseline: no AVX512, AVX512-VNNI or AMX on the CPUs that have them, and SIGILL on any CPU without AVX2. On Linux arm64 no -march is passed at all, so the build is plain Armv8.0 (System Info: NEON = 1 | ARM_FMA = 1): no dotprod, no fp16 arithmetic, no i8mm, which are the instructions every quantized and f16 matmul uses on Arm.
• ggml's CPU weight repacking never ran. ggml's CPU backend converts Q4_0 / Q4_K / Q2_K / Q5_K / Q6_K / IQ4_NL weights into interleaved layouts (and, on AMX CPUs, Q4_0 / Q4_K / Q8_0 and others into AMX tiles) when they are uploaded into its "extra" buffer types, so each matmul reading them runs a wider GEMM kernel. sd.cpp put every weight in the plain CPU buffer, so none of these kernels were reachable. A diffusion transformer is almost entirely such matmuls.

Change

Four commits.

1. Ship every ggml CPU variant in the CPU bundles (unsloth-sd-prebuilt.yml). The Linux x64, Linux arm64, macOS x86_64 and win-cpu-x64 legs build with GGML_BACKEND_DL=ON GGML_CPU_ALL_VARIANTS=ON (shared ggml), so ggml scores each libggml-cpu-<variant> against the host at startup and loads the best: x86-64 baseline, SSE4.2, AVX, AVX2 (haswell), AVX-VNNI, AVX512 / VNNI / BF16 and AMX on x86 (MSVC builds the 9 variants it can compile), Armv8.0 up to Armv8.6 i8mm, SVE2 and SME on Linux arm64 (built with GCC 14, since GCC 13 rejects +sme). sd.cpp already creates its CPU backend through the ggml registry, so no source change was needed for this.
   • Linux libraries get DT_RPATH=$ORIGIN exactly like the CUDA bundles; macOS libraries get @loader_path only.
   • New scripts/unsloth/cpu_variants_gate.sh runs on every finished CPU zip (a PowerShell twin on Windows): the floor variant (x64 / armv8.0_1) must be present, no file may point back at the build tree, and sd-cli started from the unzipped copy in an empty cwd must load a variant from that copy.
   • The Metal (macOS arm64), Vulkan and CUDA legs are unchanged.
   • Bundle size, Linux x64: 31.6 MiB to 36.9 MiB.
2. Repack quantized matmul weights for the CPU backend. Linear tags its quantized weight as matmul-only (a slot in the leaf tensor's unused op_params, so the tag lives and dies with the tensor). When the compute backend is the CPU, the model manager places a tagged weight in the first CPU extra buffer type whose supports_op accepts that weight's MUL_MAT (AMX first, then the generic repack), and logs it once: cpu weight repack: quantized matmul weights use CPU_REPACK. Everything else, and every GPU path, is untouched.
   • A LoRA on such a weight is added to the output (x W^T + x dW^T, add_lora_to_output) instead of being folded into a copy of W, since a repacked weight can only feed MUL_MAT. Weights that are not repacked keep the existing path bit for bit.
   • --lora-apply-mode immediately (which rewrites weights in place) turns repacking off.
   • A 4D activation is flattened to 3D before such a matmul (the repacked kernel takes at most three activation dims).
   • Kill switch: SD_CPU_REPACK=0.
   • --mmap keeps weights in the file mapping, which the repacked layout cannot use, so mmapped weights are not repacked; the log says so once (cpu weight repack: skipped for mmapped weights). Studio does not pass --mmap.
   • A model whose LoRAs are merged in place (immediately) opts out of repacking for that context only; on CPU, AUTO picks at_runtime whenever --type or --tensor-type-rules quantizes while loading and the diffusion model, text encoder or VAE runs on the CPU.
3. On CPU, pick at_runtime LoRA when the weights end up quantized. auto looked only at the file's types, so an f16 file loaded with --type q4_0 chose immediately. That turned repacking off for every such run, LoRA or not. The conversion target now counts as quantized when the diffusion model runs on the CPU. (On the base, that same immediately path aborts with GGML_ASSERT(*cur_backend_id != -1) for an f16 SD1.5 file + --type q4_0 + a LoRA on CPU; with this change the run completes.)
4. AVX512 variants only on CPUs that also have AVX-VNNI (ggml-patches/0008). On GitHub's Azure EPYC 9V74 (Zen 4: AVX512 without AVX-VNNI) the AVX512 variants ran 15 to 50% slower than the AVX2 one, with MSVC and with clang. On Zen 5 (Strix Halo) and Emerald Rapids, both with AVX-VNNI, they are the fastest by a wide margin. Without AVX-VNNI an AVX512 variant now scores 0, so those hosts load the AVX2 variant, as the static build ran.

Results

Base = fork master afac7a7c built with the workflow's flags; head = this branch built with the new flags. Same host, same job, arms interleaved ABBA, one fresh sd-cli process per measurement, 3 repeats, median [min, max]. Seconds. Prompt "a photo of a red fox sitting in fresh snow, golden hour, detailed fur", seed 42, cfg 1. "Sampling" is sd-cli's sampling completed, "generate" its generate_image completed (text encoder + sampling + VAE).

AMD Ryzen AI MAX+ 395 (Strix Halo, Zen 5, 16C/32T, AVX512 + AVX-VNNI), Linux, 16 threads, exclusive box

Head loads libggml-cpu-zen4.so.

Where it comes from (same box, sampling):

Threads: head at 32 threads (SMT) instead of 16 gave the same sampling time (44.13 vs 43.83 s, Z-Image) and a slower VAE decode (9.12 vs 6.24 s), so Studio's physical-core default stays.

GitHub ubuntu-24.04-arm (Neoverse-N2, 4 cores, SVE2 + i8mm), 4 threads

Base System Info: NEON = 1 | ARM_FMA = 1; head loads libggml-cpu-armv8.6_2.so (FP16_VA | MATMUL_INT8 | SVE | DOTPROD).

Z-Image with SD_CPU_REPACK=0: 116.16 s sampling.

GitHub ubuntu-latest (AMD EPYC 7763, Zen 3, AVX2 only, 4 vCPU), 4 threads

Head loads libggml-cpu-haswell.so: the same instruction set as the static build, and with SD_CPU_REPACK=0 its images are bit-identical to base and its times equal (SD-Turbo q8_0 / q4_K / q4_0 / f16 sampling 23.89 / 23.79 / 25.35 / 24.33 s base, 23.86 / 23.69 / 25.27 / 24.31 s head). The gain is the repack:

Intel Xeon Platinum 8559C (Emerald Rapids, AVX512 + AVX-VNNI + AMX), Linux, 16 threads pinned to 16 physical cores

A shared host (load average 57 to 73 on 192 threads during the run), so ratios only. Head loads libggml-cpu-sapphirerapids.so and puts Q4_K / Q4_0 / Q8_0 weights in AMX tiles.

Z-Image with the AVX2 variant forced and repack on: 68.56 s; AMX variant with SD_CPU_REPACK=0: 65.88 s.

GitHub windows-latest (MSVC, as the release leg builds it), 4 threads

This branch, with the fork's full ggml patch set applied to both arms, on an AMD EPYC 7763 (Zen 3, AVX2 only). Head loads ggml-cpu-haswell.dll.

SD-Turbo f16 head images are bit-identical to base. Z-Image's repack gain on this MSVC build is inside the noise (the same arm under another name ran 227.09 s).

Why commit 4 exists, from two earlier Windows runs without it (SD-Turbo, 512, 2 steps, sampling, SD_CPU_REPACK=0):

With commit 4 both CPUs load haswell, which matches base.

GitHub macos-15-intel (Core i7-8700B, AVX2 only, 4 vCPU)

Head loads libggml-cpu-haswell.so; SD-Turbo images are bit-identical to base at all four types. This runner's timings swung up to 2.5x within one arm (SD-Turbo q8_0 base sampling 23.13 to 66.35 s), so it gives no speed verdict either way; every head run completed.

Accuracy

An instruction set change legitimately moves low bits; the check is that head is no further from a high-precision reference than base.

• Where head runs the same variant as base with SD_CPU_REPACK=0 (Zen 3 above; AVX2 variant forced on this host with the full ggml patch set), images are bit-identical to base.
• LoRA through repacked weights, SD1.5 Q8_0 loaded as q4_0 + LCM LoRA, 4 steps, 512: repack on vs SD_CPU_REPACK=0 39.9 dB (AVX2 repack) and 42.6 dB (AMX); the same seed without the LoRA is 13.3 dB away, so the LoRA is applied. With the AVX2 variant and Q8_0 (no repack for that type on x86) head is bit-identical to base with the LoRA.

Tests

• This workflow, run on a staging fork at this branch's tree (CUDA legs and publishing off): the Linux x64, Linux arm64, macOS x86_64 and win-cpu-x64 legs all pass the new CPU variants gate. Each lists its variants (14 on Linux / macOS x64, 9 with MSVC, 8 on arm64) and sd-cli from the unzipped bundle loads haswell / armv8.6_2 / haswell / ggml-cpu-haswell.dll from it (an earlier Linux x64 runner loaded sapphirerapids). The gate caught two real problems on the way, both fixed in commit 1: build-tree LC_RPATHs in the macOS libraries, and the Windows step ending on sd-cli's own exit code.
• On the staging fork, win-cpu-x64 (and the unchanged win-vulkan-x64) fail the existing "every PE is signed" check, since a fork has no signing secrets; the signing step itself signs every PE in build/bin, which now includes the ggml-cpu-*.dll files.
• Bench workflows on GitHub ubuntu-latest, ubuntu-24.04-arm, macos-15, macos-15-intel, windows-latest and on the gfx1151 CI box: every head run exits 0.
• Local: base vs head with the AVX2 variant and SD_CPU_REPACK=0 bit-identical (SD-Turbo q4_0 / q8_0 / q4_K / f16, SD1.5 + LoRA); repack engagement in every DiT run (cpu weight repack: ... use CPU_REPACK / AMX, 324 to 449 tensors).

Not covered

• macOS arm64 (Metal bundle) is unchanged. GGML_CPU_ALL_VARIANTS' apple_m1 variant drops the fp16 arithmetic Apple clang's default target already has, and was 40% slower than base on GitHub's M1 runner, so it is not used there.
• AVX512 on Intel CPUs without AVX-VNNI (Skylake-X, Cascade Lake, Ice Lake, Tiger Lake, Rocket Lake) is now gated off and was not measured either way; they run the AVX2 variant as before.
• No Zen 4 machine measured AVX512 with repack on; the gate sends Zen 4 to the AVX2 variant (+ repack).
• Seen but not changed here: on the same EPYC 7763 class, the MSVC Windows build runs Z-Image Q4_K_M about 3x slower than the GCC Linux build (sampling 235.9 s vs 75.5 s at 256, 2 steps), while SD-Turbo q4_0 is close (27.1 vs 25.4 s). That points at MSVC code generation for the k-quant kernels; a clang Windows CPU build is the follow-up to measure.
• MiniMax-H3 and SDXL were not benchmarked on CPU. Windows on ARM and Windows on gfx1151 were not run.
• The bench builds apply the fork's ggml patch set only in the Windows and local runs; the Linux/macOS/Strix bench builds used the bare submodule for both arms.