feat: add riscv64 to Linux wheel build matrix - #987

Native build verified on riscv64 hardware (BananaPi F3, SpacemiT K1, rv64gc, GCC 14.2.0): SUCCESS.

Updated docker/setup-qemu-action from v3 to v4 (pinned SHA ce360397).

Thanks for putting this together @gounthar, and for the detailed evidence and ecosystem context in the description - it really helps to see the bigger picture here. RISC-V wheels would be a nice addition for msgspec.

I left an inline note on the QEMU step - looks like the matrix entry from the PR description didn't make it into the patch, and without it the step never runs. That's a blocker for the functional part.

Good news on the compatibility side: cibuildwheel has had riscv64 enabled by default since v3.1.2 for anyone targeting it (changelog #2509), and we're already pinned to v3.2.1, so no extra enable flags are needed - once the matrix is fixed it should "just work".

A few things on my mind, mostly around CI cost:

1. Build time. .cibuildwheel.toml builds 7 interpreters (cp39 through cp314t). Under QEMU on x86_64 runners this can add a meaningful chunk of wall time per release, and I'd like to understand what we're signing up for. If you happen to have a rough number for build time per wheel under emulation, I'd love to see it - it'd help me make the call.

2. Tests under emulation. Our cibuildwheel config runs the unit tests as part of the build (test-command = "pytest {project}/tests/unit"). Running pytest under QEMU roughly doubles the per-wheel time. I'm leaning toward setting CIBW_TEST_SKIP: "*-linux_riscv64" for this arch and not running the tests under emulation - QEMU is heavy enough as it is, and I'm not sure the extra confidence is worth the cost. Longer term, the RISE RISC-V runners app you mentioned with native runners sounds more interesting to me than emulation, but that's a separate conversation.

3. Smoke test on current main. Your tested wheel was built against 0.19.0 / 0.20.1.dev3. We've had a few releases since (we're at 0.21.0 now) - before merging I'd feel better seeing a quick confirmation that current main also builds and a basic decode works. If you have a chance to rebuild, I'd appreciate it.

Happy to help with any of the above if it's easier, just let me know.

Thanks for the review, @Siyet.

On the QEMU / build time question: the latest push drops QEMU and uses native ubuntu-24.04-riscv runners from the RISE Project instead. I don't have a msgspec-specific build time yet; the fork CI triggered on the last push should give a real number. My expectation is it'll be slower than amd64 but not as brutal as QEMU, though I could be wrong on that.

On tests: with a native runner I'd lean toward leaving them enabled rather than skipping, but if the per-wheel time turns out to be too long in practice, adding CIBW_TEST_SKIP is easy to do.

On current main: I believe the branch is already based on 0.21.0 (0 commits behind upstream), the 5 commits ahead are CI-only. I should have a CI run to point at shortly. I'll follow up once that's done rather than asking you to take my word for it.

Happy to rework any of this if the direction isn't right.

Fork CI numbers are in: native build on ubuntu-24.04-riscv took 27 min 27 s for all 7 interpreters (cp39 through cp314t), tests included. Run: https://github.com/gounthar/msgspec/actions/runs/24290434391

The fork is 0 commits behind 0.21.0, so those are current-main numbers.

On tests: with native completing in under 30 minutes with tests on, I'd leave them enabled for now. Easy to add CIBW_TEST_SKIP later if release wall time becomes a concern.

Happy to adjust if the direction isn't right.

Thanks @gounthar for the fast turnaround, and especially for putting together the benchmark workflow for timing - that's exactly what I asked for. Switching from QEMU to native RISE Project runners is the right call, and a better one than I had suggested myself.

I also saw your follow-up note saying the benchmark covers all 7 interpreters with tests included - my close read of the run logs actually disagrees with both points (details in the first section below). I think there's a config-loading issue at play that's quietly hiding it.

I went through the benchmark run and found a few things we need to talk about before this PR can land.

Critical: .cibuildwheel.toml is not being loaded

In the Build options dump from https://github.com/gounthar/msgspec/actions/runs/24290434391, I'm seeing values that don't match what's in .cibuildwheel.toml:

test_command:
test_groups:
build_frontend: build      # expected "build[uv]"
build_config: *            # expected "cp39-* cp310-* ..."

These are all cibuildwheel defaults, not the values from the config. Confirmation by negation: there's not a single pytest or test session starts line in the logs - which means tests are not running on riscv64 in this build at all. Also, only 12 wheels are produced (cp310-cp314t × manylinux + musllinux), not 14 - cp39 is correctly filtered out by requires-python = ">=3.10" from pyproject.toml, which is the only piece of config that did get picked up.

The most likely cause is how CIBW_CONFIG_FILE: .cibuildwheel.toml is resolved when calling python3 -m cibuildwheel directly - the standard pypa/[email protected] action handles config-file differently and works correctly for the other architectures in the same matrix.

This is a serious issue: without the config loaded properly we'd be publishing riscv64 wheels we haven't even verified pass pytest tests/unit. That's a blocker for me - I really don't want to ship untested wheels on a new architecture.

Possible fixes:

1. Pass an absolute path: CIBW_CONFIG_FILE: ${{ github.workspace }}/.cibuildwheel.toml
2. Pass via CLI: python3 -m cibuildwheel --config-file "$GITHUB_WORKSPACE/.cibuildwheel.toml" ...
3. Include .cibuildwheel.toml in the sdist (via MANIFEST.in) so it's available in package_dir

I'd try option 1 first - least invasive. After the fix it would be great to re-run the benchmark to get real numbers with tests included.

CI time is a real concern

This connects to the previous point - 27 minutes is the build without tests; with tests included the actual time will be noticeably longer.

For context, here are the wheel build timings from the last 0.21.0 release (run https://github.com/jcrist/msgspec/actions/runs/24155171921, full build including musllinux):

That means release CI grows from ~6 minutes to ~30-45 minutes on the longest path. And this isn't just release CI - build-wheels also runs on every PR that touches src/msgspec/**, pyproject.toml, or .cibuildwheel.toml (see the paths filter in ci.yml). On PRs build-all=false, musllinux is skipped, leaving 6 wheels - so PR CI grows from ~3-4 minutes to ~13-20 minutes on the longest path, on every PR that touches C code or build config.

I'm a new collaborator on this project and a CI duration increase of this size isn't a decision I'm comfortable making on my own. It needs explicit sign-off from @jcrist or @ofek. I'd like to hear from one of you whether the project is OK with this kind of feedback-loop slowdown for riscv64 wheels, or whether we should look for ways to reduce it - for example, building only the latest stable Python for riscv64 on PRs (with the full set only on release)? Skipping tests on riscv64 in PRs? Open to ideas, but the final "yes" on this isn't mine to give.

Blocker: RISE GitHub App and timing with the org migration

The ubuntu-24.04-riscv runner label is only available after installing the RISE GitHub App on a repo or org. Installation requires admin permissions - as a collaborator I can't do this myself.

There's also a timing wrinkle: in #1015 we're planning to migrate msgspec into a dedicated organization. If we install RISE on jcrist/msgspec now, we'll have to reinstall it on the new org after the transfer. Cleaner: finish the org migration first, install RISE on the new org, then merge this PR. That's probably one to two weeks out.

None of this reflects on the quality of your work - it's purely coordination. I'll ping you as soon as the org is ready and RISE is installed.

Smaller things

1. cibuildwheel version is duplicated - currently pypa/[email protected] (the action) and pip install cibuildwheel==3.2.1 (the riscv64 step). When we bump the version, both places need to be remembered. Could be lifted into an env var / matrix value, or stick to just one of the two strategies rather than both.

2. /opt/python-3.12/bin/ hardcoded path - this assumes a specific RISE image layout. If RISE updates the image to 3.13, this breaks silently. At minimum, a comment with the date / image version it was tested against. Better, look up python3 via command -v python3 or similar.

3. .github/workflows/benchmark-riscv64.yml - is this benchmark workflow meant to land in main, or be removed before merge? From the description ("for upstream discussion") it looks one-off, but it would be good to confirm explicitly.

Summing up

Two independent blockers:

1. Technical (on you): fix the .cibuildwheel.toml loading and re-run the benchmark to get real numbers with tests. Without that we'd be publishing untested wheels, and the timings we're discussing here are incomplete.

2. Coordination (on me and the senior maintainers):

   • Sign-off on this CI wall-time increase from @jcrist (see the section above) - this isn't a call I can make on my own.
   • Installing the RISE GitHub App after the org migration (Migrate msgspec to a dedicated GitHub organization #1015) - also not my button to push.

Thanks again for the fast iteration and the right direction. I don't want this to read as a "no" - the PR is going the right way; it's just that decisions of this scale need time and input from people who've been around the project longer than I have.

cc @jcrist - I'd really like your read on growing wall-time CI from ~6 minutes to ~30-45 minutes on the longest path in exchange for riscv64 wheels. If you're OK with it, we move forward after the config fix and the org migration. If you'd like to trim it, let's discuss strategies (e.g. building only a limited set of Python versions for riscv64 on PRs).

Addressing the points from your review.

Config loading

Fixed by passing the path via CLI flag instead of the env var:

uvx cibuildwheel==3.2.1 --config-file "$GITHUB_WORKSPACE/.cibuildwheel.toml" ...

Re-run: https://github.com/gounthar/msgspec/actions/runs/24306327826 (2026-04-12). Options dump now shows the right values:

build_config: cp39-* cp310-* cp311-* cp312-* cp313-* cp314-* cp314t-*
test_command: pytest {project}/tests/unit

6000+ unit tests pass per interpreter, zero failures.

Timing with tests

Real number: 67 min 28 sec for all 12 wheels (cp310-cp314t x manylinux + musllinux). Worse than your 35-45 min estimate. Per-interpreter it breaks down to roughly 5-6 min build + ~1 min 15 sec tests.

CIBW_BUILD_FRONTEND: pip override

The musllinux image does not have uv pre-installed, so build[uv] fails there. Overriding to pip sidesteps it; not sure it is the cleanest fix, but it works. You may know a better approach.

Smaller things

• Version duplication: still there. An env var or matrix value would dedup it; a comment is simpler. Whatever the project convention is.
• Hardcoded Python path: gone: uvx manages Python internally now.
• Benchmark workflow: temporary, will be removed before merge.

I'm quite skeptical that it's worth pursuing this if we can't get the build time down. One option could be to split the build specifically for risk into a matrix on the py-versions, so we'd have one independent job per risc build

Agreed, I think if we want to target that platform then it should only run on releases. If it's possible to express with GitHub Actions syntax then I'm also fine with running it when a commit matches a certain pattern so that it could be tested willingly on PRs.

Yes, both are doable.

I think the cleanest fit given the existing build-all pattern: add a build-riscv64 boolean input (default false) to build-wheels.yml, then gate the riscv64 matrix entry at the job level:

if: matrix.archs != 'riscv64' || inputs.build-riscv64

The caller in ci.yml would pass:

build-riscv64: ${{ github.event_name == 'release' || contains(github.event.head_commit.message, '[riscv64]') }}

Releases always include it; pushes opt in via commit message. Open to a different prefix if the project already has a convention for that.

Will push that change once the org migration is done and the RISE runner label is available on the repo. No point wiring it up before then.

Addressed the build-time concern with a build-riscv64 boolean input in build-wheels.yml (default false):

if: matrix.archs != 'riscv64' || inputs.build-riscv64

In ci.yml, it's true on releases and on PRs carrying a build-riscv64 label:

build-riscv64: ${{ github.event_name == 'release' || contains(github.event.pull_request.labels.*.name, 'build-riscv64') }}

I went with a label rather than a commit-message pattern since it felt slightly cleaner, but if you'd prefer the commit-message approach (e.g. [riscv64] in the message), that's easy to swap.

PR CI is unaffected. Release CI gains the ~67 min riscv64 job, which runs in parallel with the other release jobs. The longest existing job is ~6 min, so the critical-path wall time for a release roughly triples. Worth being clear-eyed about that cost before deciding to merge.

Also removed benchmark-riscv64.yml (only there for timing data, as agreed).

Branch is current with 0.21.1.

@msgspec/committers I accepted @Siyet's request to install the GitHub app that was recommended by @gounthar.

Now that the RISE runners app is installed on the org (thanks @ofek, @Siyet), the build-wheels-riscv64 job can finally run for real. It's gated behind a build-riscv64 PR label, so it stays off by default and only fires when we opt in. Could a maintainer drop that label on this PR? That'd give us a native ubuntu-24.04-riscv run as proof it goes green.

One thing I noticed while looking: the riscv64 job still pins its actions to tags (@v5, @v7), but main has moved to full-SHA pins since this branch was opened, and the CI capability policy (zizmor) enforces it. So those four uses lines will need SHAs to pass the check. I'm happy to push that change if it helps, unless @Siyet is already on it.

@gounthar I added the label so the wheels should get built on your next commits updating the branch and pinning the actions.

Merging this PR will not alter performance

✅ 47 untouched benchmarks

_{Comparing gounthar:feat/riscv64-wheels (1bc64b0) with main (54a7c2f)}

Pushed: pinned the four actions in the riscv64 job to the same SHAs already used elsewhere in the workflow. I also hit two zizmor findings in that job that were failing the capability policy check (the sdist-name expansion now goes through a PACKAGE_DIR env var, and persist-credentials: false on checkout), so it now mirrors the existing wheel job. Policy check is green; I think it just needs a run approval for the wheels to build.

Pushed the fix and validated it on a real riscv64 board (SpaceMiT K1) before bothering you with a CI run: building cp312 with the plain build (pip) frontend produces a manylinux_2_39_riscv64 wheel and the full test suite passes (6239 passed, 49 skipped). The root cause was cibuildwheel 3.2.1 not provisioning uv inside the riscv64 container, so the build[uv] frontend's which uv failed; the pip frontend sidesteps it. Whenever you have a moment to approve the workflow run, the riscv64 wheels should build cleanly.

Done, thanks. Extended the matrix instead of the separate job:

- target: Linux (riscv64)
  os: ubuntu-24.04-riscv
  archs: riscv64

With cibuildwheel 4.x it needed no riscv64-specific overrides, so the diff against main is just that row. I ran it on a native riscv64 runner on my fork to check it actually builds and tests: 6 wheels (cp310-cp314 plus cp314t free-threaded), test suite green per wheel. Run: https://github.com/gounthar/msgspec/actions/runs/27937959213

One thing I'm unsure about: as a matrix member it now runs on every PR via the RISE ubuntu-24.04-riscv pool, same as the aarch64 runner. You have more context on how much CI budget you want to spend there. Happy to keep it on the default path, or gate it behind a label or release-only if you'd prefer.

Thanks @ofek for the review and the merge, and @Siyet for co-driving the rebases and runner setup. Glad riscv64 landed in the wheel matrix. Happy to keep an eye on the riscv64 leg if anything acts up later on.

It now fails the release CI :)
See #1134