Skip to content

Flaky WSL cuda.bindings benchmark smoke test: pyperf raises ValueError: benchmark function returned zero #1885

Closed
Bug
#1899
#1889
Closed
Flaky WSL cuda.bindings benchmark smoke test: pyperf raises ValueError: benchmark function returned zero#1885
Bug
#1899
#1889
Assignees
danielfrg
Labels
P1Medium priority - Should docuda.bindingsEverything related to the cuda.bindings moduletestImprovements or additions to teststriageNeeds the team's attention

Description

@rwgk
rwgk
opened on Apr 9, 2026

Observed shortly after #1880 was merged.

I expect this will haunt us until we tweak the test.

Cursor-generated writeup:

Description

The WSL CI lane Test linux-64 / py3.12, 13.2.0, wheels, rtx4090, wsl can fail in the Run cuda.bindings benchmarks (smoke test) step with a zero-duration pyperf measurement:

python run_pyperf.py --fast --loops 1 --min-time 0
...
ValueError: benchmark function returned zero

This does not look like a functional CUDA regression. In the failing run, the job had already completed:

  • Run cuda.pathfinder tests with see_what_works
  • Run cuda.bindings tests

and then failed specifically in:

  • Run cuda.bindings benchmarks (smoke test)

The benchmark smoke step currently comes from .github/workflows/test-wheel-linux.yml:

pip install pyperf
pushd cuda_bindings/benchmarks
python run_pyperf.py --fast --loops 1 --min-time 0
popd

The benchmark implementation that was called out in the PR #1817 analysis is:

def bench_pointer_get_attribute(loops: int) -> float:
    _cuPointerGetAttribute = cuda.cuPointerGetAttribute
    _attr = ATTRIBUTE
    _ptr = PTR

    t0 = time.perf_counter()
    for _ in range(loops):
        _cuPointerGetAttribute(_attr, _ptr)
    return time.perf_counter() - t0

With --loops 1 --min-time 0, a zero-length timing sample on WSL and fast hardware seems plausible, so this looks like a benchmark flake rather than a deterministic product failure.

Failing example

Passing comparisons

Why this looks flaky

  • The failure occurs in the benchmark smoke step, not in the regular cuda.bindings test suite.
  • Nearby runs of the same WSL lane passed.
  • The exception comes from pyperf rejecting a zero-duration sample, not from a CUDA API error.

Possible follow-ups

  • Increase the amount of work in the smoke benchmark, for example by raising --loops or using a positive --min-time.
  • Retry the benchmark smoke step once before failing the whole job.
  • Consider using different benchmark-smoke settings on WSL than on non-WSL Linux runners.

Activity

  1. rwgk
    added
    triageNeeds the team's attention
    P1Medium priority - Should do
    testImprovements or additions to tests
    cuda.bindingsEverything related to the cuda.bindings module
    on Apr 9, 2026
  2. rwgk
    added the
    Bug
    issue type on Apr 9, 2026
  3. rwgk
    assigned
    danielfrg
    on Apr 9, 2026

  4. danielfrg commented on Apr 9, 2026

    @danielfrg
    danielfrg
    on Apr 9, 2026
    Contributor

    from all the things i expected to fail on windows that was not one of them :D

    I'll make a PR today.

  5. danielfrg
    added a commit that references this issue on Apr 9, 2026

    Fix #1885 add min-time to benchmark smoke test

    92a3d8f
  6. danielfrg
    closed this as completedin 87c305bon Apr 10, 2026
  7. pull
    added a commit that references this issue on Apr 10, 2026

    Fix NVIDIA#1885 add min-time to benchmark smoke test (NVIDIA#1889)

    87c305b

  8. rwgk commented on Apr 12, 2026

    @rwgk
    rwgk
    on Apr 12, 2026
    ContributorAuthor

    @danielfrg I'm still seeing a flake:

    First attempt (one job failed, see section from log below):

    Rerun (the previously failing job succeeded):

    Could you please take another look?

    + pushd cuda_bindings/benchmarks
+ python run_pyperf.py --fast --loops 1 --min-time 1
/__w/cuda-python/cuda-python/cuda_bindings/benchmarks /__w/cuda-python/cuda-python
Traceback (most recent call last):
  File "/__w/cuda-python/cuda-python/cuda_bindings/benchmarks/run_pyperf.py", line 8, in <module>
    main()
  File "/__w/cuda-python/cuda-python/cuda_bindings/benchmarks/runner/main.py", line 99, in main
    runner.bench_time_func(bench_id, registry[bench_id])
  File "/opt/hostedtoolcache/Python/3.12.13/x64/lib/python3.12/site-packages/pyperf/_runner.py", line 502, in bench_time_func
    result = self._main(task)
             ^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.12.13/x64/lib/python3.12/site-packages/pyperf/_runner.py", line 463, in _main
    bench = self._worker(task)
            ^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.12.13/x64/lib/python3.12/site-packages/pyperf/_runner.py", line 437, in _worker
    run = task.create_run()
          ^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.12.13/x64/lib/python3.12/site-packages/pyperf/_worker.py", line 300, in create_run
    self.compute()
  File "/opt/hostedtoolcache/Python/3.12.13/x64/lib/python3.12/site-packages/pyperf/_worker.py", line 364, in compute
    WorkerTask.compute(self)
  File "/opt/hostedtoolcache/Python/3.12.13/x64/lib/python3.12/site-packages/pyperf/_worker.py", line 289, in compute
    self.compute_warmups_values()
  File "/opt/hostedtoolcache/Python/3.12.13/x64/lib/python3.12/site-packages/pyperf/_worker.py", line 271, in compute_warmups_values
    self._compute_values(self.values, args.values)
  File "/opt/hostedtoolcache/Python/3.12.13/x64/lib/python3.12/site-packages/pyperf/_worker.py", line 85, in _compute_values
    raise ValueError("benchmark function returned zero")
ValueError: benchmark function returned zero
Traceback (most recent call last):
  File "/__w/cuda-python/cuda-python/cuda_bindings/benchmarks/run_pyperf.py", line 8, in <module>
    main()
  File "/__w/cuda-python/cuda-python/cuda_bindings/benchmarks/runner/main.py", line 99, in main
    runner.bench_time_func(bench_id, registry[bench_id])
  File "/opt/hostedtoolcache/Python/3.12.13/x64/lib/python3.12/site-packages/pyperf/_runner.py", line 502, in bench_time_func
    result = self._main(task)
             ^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.12.13/x64/lib/python3.12/site-packages/pyperf/_runner.py", line 468, in _main
    bench = self._manager()
            ^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.12.13/x64/lib/python3.12/site-packages/pyperf/_runner.py", line 681, in _manager
    bench = Manager(self).create_bench()
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.12.13/x64/lib/python3.12/site-packages/pyperf/_manager.py", line 243, in create_bench
    worker_bench, run = self.create_worker_bench()
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.12.13/x64/lib/python3.12/site-packages/pyperf/_manager.py", line 142, in create_worker_bench
    suite = self.create_suite()
            ^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.12.13/x64/lib/python3.12/site-packages/pyperf/_manager.py", line 136, in create_suite
    suite = self.spawn_worker(0, 0)
            ^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.12.13/x64/lib/python3.12/site-packages/pyperf/_manager.py", line 118, in spawn_worker
    raise RuntimeError("%s failed with exit code %s"
RuntimeError: /opt/hostedtoolcache/Python/3.12.13/x64/bin/python failed with exit code 1
....
Error: Process completed with exit code 1.
  9. rwgk
    reopened this on Apr 12, 2026
  10. rwgk
    closed this as completedin #1899on Apr 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

P1Medium priority - Should docuda.bindingsEverything related to the cuda.bindings moduletestImprovements or additions to teststriageNeeds the team's attention

Type

Bug

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions