Skip to content

Ship current Vulkan prebuilts and restore Volta CUDA support - #22

Merged
danielhanchen merged 3 commits into
unslothai:masterfrom
oobabooga:fix/h3-native-prebuilts
Oct 7, 2026
Merged

danielhanchen merged 3 commits into
unslothai:masterfrom
oobabooga:fix/h3-native-prebuilts

Conversation

@oobabooga

@oobabooga oobabooga commented Oct 6, 2026 •

Copy link
Copy Markdown
Member

MiniMax-H3's Vulkan path can crash during denoising on Linux or abort while loading the text encoder on Windows. The fixes are already in this repository, but Studio gets an older upstream Vulkan build because this workflow does not publish Vulkan assets.

The Linux CUDA prebuilt also has no sm70 kernels, so it cannot run on a V100 even though the device probe can list the card.

Root cause

The older Vulkan build failed in two different places on the AMD runners:

  • On Linux, the scheduler rewrites graph pointers during a mixed CPU/Vulkan segment. The next segment reuses those pointers after the scheduler has freed their tensors, causing a segfault.
  • On Windows, the embedding type allowlist excludes BF16. The encoder's embedding tensor is expanded to a 3.11 GB F32 buffer, which exceeds the driver's buffer-size limit.

Current source restores the graph state between segments and keeps supported BF16 embedding weights in BF16. This PR packages those fixes for Vulkan.

Fix

  • Build sd-cli and sd-server for Linux x86_64 Vulkan and Windows x64 Vulkan. The Windows binaries go through the existing signing steps.
  • Build Linux on Ubuntu 22.04. Studio does not check glibc when selecting an asset, so a 24.04 build could be installed on a host that cannot load it.
  • Use Vulkan SDK 1.4.328.1, with both downloads checked against pinned SHA256 values before extraction or execution. The SDK supplies glslc, which Ubuntu 22.04 does not package.
  • Add 70 to the Linux CUDA architectures.

Both Vulkan assets are required for publication. A Vulkan build failure therefore blocks CPU and Apple releases too. Allowing a release without them would leave affected AMD users on the older fallback after the Studio pin update. CUDA keeps its existing best-effort policy.

After merging, publish the native release, then update Studio's prebuilt pin. unslothai/unsloth#12902 fixes Windows CPU selection when a release contains both CPU and Vulkan assets.

Testing

Live A/B on the AMD DevLab Linux and Windows runners, both with a Radeon 8060S. Old source bfbef5b against current source 7bbffa3, using Q4, 1344x768, 124 frames, two denoising steps and a 20 GiB VRAM budget.

Old Vulkan build Current source
Linux Segfault at a graph-segment transition Video and audio, about 40 minutes
Windows Abort allocating the encoder buffer Video and audio, about 67 minutes

All 124 frames and the audio decoded successfully on both. An isolated backport of the graph-state fix also completed the Linux run.

Linux run / Windows run

  • The hardware runs used binaries from a separate test workflow, with the Linux one built on Ubuntu 24.04. After moving to 22.04, the Linux Vulkan sd-cli and sd-server built, were packaged and extracted through Studio's installer helpers, and loaded in an Ubuntu 22.04 container without the SDK mounted. Their highest required GLIBC symbol version is 2.34. The GPU run was not repeated with this package.
  • This workflow ran end to end on a fork with signing and publication removed (run). All eight build legs and assembly passed. Both Vulkan legs passed the pinned SHA256 checks and found Vulkan 1.4.328 and glslc from the SDK. The fingerprint check covered all eight bundles, and the coverage gate found all seven required assets.
  • The CUDA bundle from that run contains sm_70 kernels in both sd-cli and sd-server, alongside sm_75 through sm_120a.
  • Both SDK downloads matched LunarG's checksum index. Studio selects the new asset names. Actionlint passed.

Not tested:

  • V100 hardware, Windows signing, or publication.
  • GPU runs of the binaries this workflow produces.
  • Visual quality at normal step counts. These were two-step execution checks.
  • The original RX 7900 DeviceLost failure was not reproduced identically on the 8060S.

@oobabooga

Copy link
Copy Markdown
Member Author

Reached Codex review convergence at oobabooga#1.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants