Skip to content

cuda.bindings build time on Linux becomes ~2x slower #642

Closed as not planned
Closed as not planned
cuda.bindings build time on Linux becomes ~2x slower#642
Assignees
cryos
Labels
CI/CDCI/CD infrastructureP0High priority - Must do!cuda.bindingsEverything related to the cuda.bindings modulepackagingAnything related to wheels or Conda packages
Milestone
cuda-python 13.0.2 & 12.9.3

Description

@leofang
leofang
opened on May 18, 2025

This is a new situation and is different from #602 (and the timings there can be used as a reference). I suspect this has to do with Cython 3.1.0, based on my eyeball-bisecting the recent build times taken in the main branch:
https://github.com/NVIDIA/cuda-python/actions/workflows/ci.yml?query=branch%3Amain++

I suspect this starts after Cython 3.1.0 was released, the first slowness observation:

in contrast with the previous run:

What I don't understand is Windows seems to be unaffected (and continues to be on the order of 4-5 mins)...

Activity

  1. leofang
    added this to the cuda-python 12-next, 11-next milestone on May 18, 2025
  2. leofang
    assigned
    cryos
    on May 18, 2025
  3. leofang
    added
    P0High priority - Must do!
    CI/CDCI/CD infrastructure
    packagingAnything related to wheels or Conda packages
    cuda.bindingsEverything related to the cuda.bindings module
    on May 18, 2025

  4. leofang commented on May 19, 2025

    @leofang
    leofang
    on May 19, 2025
    MemberAuthor

    Confirmed in #640 that the build time can be restored by pinning Cython<3.1.0

  5. leofang commented on Jun 23, 2025

    @leofang
    leofang
    on Jun 23, 2025
    MemberAuthor

    I can reproduce the 2x slowdown locally. But when I tried to chase after the root cause, I probably picked the wrong module to look at. cuda.bindings.cydriver compilation is slower with newer Cython, but its compilation/linking times are insignificant compared to the overall build time. Anyway I attach the .cpp file generated by Cython 3.0.12 (old) and 3.1.2 (new): cydriver.zip

    Below is the commands generated by setuptools with pip install --no-build-isolation --no-deps -v .:

    • compile
    /local/home/leof/miniforge3/envs/py312_cu129/bin/x86_64-conda-linux-gnu-c++ -fvisibility-inlines-hidden -fmessage-length=0 -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /local/home/leof/miniforge3/envs/py312_cu129/include -I/local/home/leof/miniforge3/envs/py312_cu129/targets/x86_64-linux/include -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /local/home/leof/miniforge3/envs/py312_cu129/include -I/local/home/leof/miniforge3/envs/py312_cu129/targets/x86_64-linux/include -fPIC -I/local/home/leof/miniforge3/envs/py312_cu129/include -I/local/home/leof/miniforge3/envs/py312_cu129/targets/x86_64-linux/include -I/local/home/leof/miniforge3/envs/py312_cu129/include/python3.12 -c old_cydriver.cpp -o cydriver.o -std=c++14 -fpermissive -Wno-deprecated-declarations "-D _GLIBCXX_ASSERTIONS" -fno-var-tracking-assignments -O3
    • link
    /local/home/leof/miniforge3/envs/py312_cu129/bin/x86_64-conda-linux-gnu-c++ -fvisibility-inlines-hidden -fmessage-length=0 -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /local/home/leof/miniforge3/envs/py312_cu129/include -I/local/home/leof/miniforge3/envs/py312_cu129/targets/x86_64-linux/include -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /local/home/leof/miniforge3/envs/py312_cu129/include -I/local/home/leof/miniforge3/envs/py312_cu129/targets/x86_64-linux/include -pthread -B /local/home/leof/miniforge3/envs/py312_cu129/compiler_compat -shared -Wl,--allow-shlib-undefined -Wl,-rpath,/local/home/leof/miniforge3/envs/py312_cu129/lib -Wl,-rpath-link,/local/home/leof/miniforge3/envs/py312_cu129/lib -L/local/home/leof/miniforge3/envs/py312_cu129/lib -Wl,--allow-shlib-undefined -Wl,-rpath,/local/home/leof/miniforge3/envs/py312_cu129/lib -Wl,-rpath-link,/local/home/leof/miniforge3/envs/py312_cu129/lib -L/local/home/leof/miniforge3/envs/py312_cu129/lib -Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,--disable-new-dtags -Wl,--gc-sections -Wl,--allow-shlib-undefined -Wl,-rpath,/local/home/leof/miniforge3/envs/py312_cu129/lib -Wl,-rpath-link,/local/home/leof/miniforge3/envs/py312_cu129/lib -L/local/home/leof/miniforge3/envs/py312_cu129/lib -L/local/home/leof/miniforge3/envs/py312_cu129/targets/x86_64-linux/lib -L/local/home/leof/miniforge3/envs/py312_cu129/targets/x86_64-linux/lib/stubs -fvisibility-inlines-hidden -fmessage-length=0 -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /local/home/leof/miniforge3/envs/py312_cu129/include -I/local/home/leof/miniforge3/envs/py312_cu129/targets/x86_64-linux/include -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /local/home/leof/miniforge3/envs/py312_cu129/include -I/local/home/leof/miniforge3/envs/py312_cu129/targets/x86_64-linux/include build/temp.linux-x86_64-cpython-312/cuda/bindings/cydriver.o -L/local/home/leof/miniforge3/envs/py312_cu129/lib/python3.12/site-packages -L/local/home/leof/miniforge3/envs/py312_cu129/lib -o build/lib.linux-x86_64-cpython-312/cuda/bindings/cydriver.cpython-312-x86_64-linux-gnu.so -Wl,--strip-all
  6. leofang
    modified the milestones: cuda-python 12-next, 11-next, cuda-python parking lot on Jul 31, 2025
  7. github-project-automation
    added this to CCCLon Jul 31, 2025
  8. github-project-automation
    moved this to Todo in CCCLon Jul 31, 2025
  9. leofang
    modified the milestones: cuda-python 13-next, 12-next, cuda-python parking lot on Oct 2, 2025

  10. leofang commented on Oct 9, 2025

    @leofang
    leofang
    on Oct 9, 2025
    MemberAuthor

    I can't tell what's changed, nor am I interested in digging, but it seems the slowness has been fixed lately. linux-aarch64 is still the slowest to build, but not too bad, and it might be runner-hardware-dependent. Perhaps Cython 3.1.4 emits better code now? Anyway, let's close this for now.

    This is the most recent run (on the main branch): https://github.com/NVIDIA/cuda-python/actions/runs/18361252550

    cc @cpcloud @mdboom for vis

  11. leofang
    closed this as not plannedon Oct 9, 2025
  12. github-project-automation
    moved this from Todo to Done in CCCLon Oct 9, 2025
  13. leofang
    modified the milestones: cuda-python parking lot, cuda-python 13.0.2 & 12.9.3 on Oct 9, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

CI/CDCI/CD infrastructureP0High priority - Must do!cuda.bindingsEverything related to the cuda.bindings modulepackagingAnything related to wheels or Conda packages

Type

No type

Projects

  • CCCL
    Status
    Done

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions