Repository navigation
CI: Add wheel building and testing on Linux-riscv64 - #31488
Conversation
Co-Authored-By: Bruno Verachten <[email protected]>
luhenry
changed the title
|
Ran your branch (9f29dbe) on a Banana Pi F3 (SpaceMiT K1, rv64gc), since the CI job here is gated on approval. Built cp312-manylinux_riscv64 with cibuildwheel straight off the branch, about 38 min native. Test suite (label='fast') came out at 45984 passed, 8 failed. All 8 failures are the same test: The wheel bundles scipy-openblas 0.3.33, so this looks like OpenBLAS#5811 (the riscv64 Cholesky regression) rather than anything in the workflow. The build and test plumbing themselves work end to end. One practical thing I'm unsure about: since the riscv64 path runs label='fast' and that set includes these Cholesky cases, I'd expect the test step to hit the same 8 failures on the RISE runners until the OpenBLAS fix lands. Might be worth xfailing those four parametrized cases on riscv64 so the job can go green in the meantime, though you'd know better than me whether the runner hardware actually reproduces it. Happy to rerun once there's a fix, to confirm it clears. |
|
They don't fail on the RISE runners because the machines don't have RVV, so it doesn't hit the RVV-enabled path in OpenBLAS. You're hitting it on your BananaPi because it does have RVV. |
|
Ah, that explains it. I'd assumed the runners were RVV-capable, so the job goes green as-is and the xfail idea is moot. The flip side, if I'm reading it right: OpenBLAS's RVV kernels then stay untested in this CI, so a regression like #5811 wouldn't get caught here. Not a blocker for this PR, more something to revisit if RVV-capable runners ever join the pool. You'd know that end better than I do. Happy to rerun on the F3 once the OpenBLAS fix lands. |
rgommers
left a comment
rgommers
left a comment
There was a problem hiding this comment.
Thanks @luhenry, this looks great. Two minor comments to ensure we save the cache on main, so it's then usable in all PRs. With those, I think this can be merged.
I also considered adding cache hit stats, we can skip that for now since it's a little annoying - has to be done outside the container cibuildwheel runs in. If we want it in the future for debugging, it should be something like:
- name: Install ccache (host)
run: |
CCACHE_VERSION=4.13.6
curl -fsSL https://github.com/ccache/ccache/releases/download/v${CCACHE_VERSION}/ccache-${CCACHE_VERSION}-linux-$(uname -m)-musl-static.tar.gz | \
sudo tar -xzf - --strip-components 1 -C /usr/local/bin ccache-${CCACHE_VERSION}-linux-$(uname -m)-musl-static/ccache
- name: Ccache performance
run: |
ccache --evict-older-than 1d
ccache -s
Yes I considered it. The easiest way would have been to do a |
[skip ci]
|
There's a couple of warnings visible in the logs, for |
|
Bad luck, the job on the merge commit segfaulted (see this CI log): Not informative. This job passes multiple times before merging, so there is a real bug but probably infrequent. Needs to be figured out, might help to add |
|
That is a known but very very infrequent hardware bug. In the 400-500 jobs/day I see go by, I probably see it 2-4 times/week. Bad luck indeed. The issue is the hardware has a bug in its atomic operation implementation where a CAS can sometimes return "success" when it wasn't actually a success. In this case, a |
|
Ah, good to know - "hardware bug" was not one of the things that came to mind as a candidate root cause:) |
PR summary
Follow up to #30995 with ccache and in a separate
linux_riscv64.yml.AI Disclosure
AI tools used for research and understanding of the problem. Artisanal coding by (human) hands
cc @rgommers @andyfaff @gounthar