Skip to content

[CI failure] CI scheduled runs #2745

Open
Open
[CI failure] CI scheduled runs#2745
Assignees
rwgkjpascucci-nv
Labels
P0High priority - Must do!ci-workflow-healthAssigned to rolling issues managed by the ci-workflow-health.yml workflow.
Milestone
cuda.core 1.3.0

Description

@github-actions
github-actions
bot
opened on Sep 1, 2026

This issue tracks an unhealthy unattended CI workflow.
Subsequent failures are recorded in comments; a successful run in the
same trigger scope closes the issue automatically.

First unhealthy run

CI run #5545, attempt 1 concluded with cancelled.

  • Trigger: schedule
  • Branch: main
  • Commit: 0244727
  • Completed: 2026-09-01T01:28:45Z

Created automatically by the CI workflow-health monitor.

Activity

  1. github-actions
    added
    bugSomething isn't working
    triageNeeds the team's attention
    CI/CDCI/CD infrastructure
    ci-workflow-healthAssigned to rolling issues managed by the ci-workflow-health.yml workflow.
    on Sep 1, 2026

  2. github-actions commented on Sep 2, 2026

    @github-actions
    github-actionsbot
    on Sep 2, 2026 – with GitHub Actions
    ContributorAuthor

    Another unhealthy run

    CI run #5569, attempt 1 concluded with failure.

    • Trigger: schedule
    • Branch: main
    • Commit: b81f9d6
    • Completed: 2026-09-02T01:52:57Z

  3. jpascucci-nv commented on Sep 2, 2026

    @jpascucci-nv
    jpascucci-nv
    on Sep 2, 2026
    Contributor

    There seems to be two problems here. One is #2725, which looks like it's in progress. The other appears to be a transient:
    unknown: failed to copy: httpReadSeeker: failed open: unexpected status from GET request to https://quay.io/v2/pypa/manylinux_2_28_aarch64/blobs/sha256:f417c91485c002aaf565f940ec46da5f0a23b04219433e3cdec0382a34456d2e: 502 Bad Gateway - I was just able to get that URL from my desktop.

  4. jpascucci-nv
    added
    duplicateThis issue or pull request already exists
    on Sep 2, 2026

  5. github-actions commented on Sep 3, 2026

    @github-actions
    github-actionsbot
    on Sep 3, 2026 – with GitHub Actions
    ContributorAuthor

    Another unhealthy run

    CI run #5586, attempt 1 concluded with cancelled.

    • Trigger: schedule
    • Branch: main
    • Commit: 53b4374
    • Completed: 2026-09-03T02:16:51Z

  6. jpascucci-nv commented on Sep 3, 2026

    @jpascucci-nv
    jpascucci-nv
    on Sep 3, 2026
    Contributor

    This appears to be failing in a different way, I'm going to remove the duplicate tag.

    Windows fatal exception: access violation

<Cannot show all threads while the GIL is disabled>
Stack (most recent call first):
  File "C:\actions-runner\_work\cuda-python\cuda-python\cuda_core\tests\test_module.py", line 289 in test_get_kernel
  File "C:\actions-runner\_work\cuda-python\cuda-python\cuda_core\tests\conftest.py", line 102 in wrapper
  File "C:\actions-runner\_work\_tool\Python\3.14.7\x64-freethreaded\Lib\site-packages\pytest_run_parallel\plugin.py", line 83 in closure
  File "C:\actions-runner\_work\_tool\Python\3.14.7\x64-freethreaded\Lib\threading.py", line 1024 in run
  File "C:\actions-runner\_work\_tool\Python\3.14.7\x64-freethreaded\Lib\threading.py", line 1082 in _bootstrap_inner
  File "C:\actions-runner\_work\_tool\Python\3.14.7\x64-freethreaded\Lib\threading.py", line 1044 in _bootstrap

Current thread's C stack trace (most recent call first):
  <cannot get C stack on this system>
/c/actions-runner/_work/cuda-python/cuda-python/ci/tools/run-tests: line 197:   825 Segmentation fault         ${SANITIZER_CMD} pytest -rxXs -v --randomly-dont-reorganize "${PYTEST_PARALLEL_ARGS[@]}" tests/
tests/test_module.py::test_get_kernel[1-5]
  7. jpascucci-nv
    removed
    duplicateThis issue or pull request already exists
    on Sep 3, 2026

  8. github-actions commented on Sep 4, 2026

    @github-actions
    github-actionsbot
    on Sep 4, 2026 – with GitHub Actions
    ContributorAuthor

    Another unhealthy run

    CI run #5599, attempt 1 concluded with cancelled.

    • Trigger: schedule
    • Branch: main
    • Commit: fb57dbf
    • Completed: 2026-09-04T01:28:56Z

  9. github-actions commented on Sep 5, 2026

    @github-actions
    github-actionsbot
    on Sep 5, 2026 – with GitHub Actions
    ContributorAuthor

    Another unhealthy run

    CI run #5621, attempt 1 concluded with cancelled.

    • Trigger: schedule
    • Branch: main
    • Commit: f84a7cf
    • Completed: 2026-09-05T02:02:43Z

  10. github-actions commented on Sep 6, 2026

    @github-actions
    github-actionsbot
    on Sep 6, 2026 – with GitHub Actions
    ContributorAuthor

    Another unhealthy run

    CI run #5625, attempt 1 concluded with cancelled.

    • Trigger: schedule
    • Branch: main
    • Commit: f84a7cf
    • Completed: 2026-09-06T01:18:30Z

  11. github-actions commented on Sep 7, 2026

    @github-actions
    github-actionsbot
    on Sep 7, 2026 – with GitHub Actions
    ContributorAuthor

    Another unhealthy run

    CI run #5627, attempt 1 concluded with cancelled.

    • Trigger: schedule
    • Branch: main
    • Commit: f84a7cf
    • Completed: 2026-09-07T01:17:32Z

  12. github-actions commented on Sep 8, 2026

    @github-actions
    github-actionsbot
    on Sep 8, 2026 – with GitHub Actions
    ContributorAuthor

    Another unhealthy run

    CI run #5629, attempt 1 concluded with cancelled.

    • Trigger: schedule
    • Branch: main
    • Commit: f84a7cf
    • Completed: 2026-09-08T03:32:16Z

  13. 35 remaining items

  14. github-actions commented on Oct 3, 2026

    @github-actions
    github-actionsbot
    on Oct 3, 2026 – with GitHub Actions
    ContributorAuthor

    Another unhealthy run

    CI run #6246, attempt 1 concluded with cancelled.

    • Trigger: schedule
    • Branch: main
    • Commit: 8e66f6b
    • Completed: 2026-10-03T01:47:35Z

  15. leofang commented on Oct 3, 2026

    @leofang
    leofang
    on Oct 3, 2026
    Member

    Looking at the above failure, all failing CIs timed out after 90 mins. @jpascucci-nv @rwgk As discussed on Slack I really, really think #2802 is a mistake that we should revert. Can we please do that and start analyzing if there is a pattern among timed-out runners?

  16. github-actions commented on Oct 4, 2026

    @github-actions
    github-actionsbot
    on Oct 4, 2026 – with GitHub Actions
    ContributorAuthor

    Another unhealthy run

    CI run #6257, attempt 1 concluded with cancelled.

    • Trigger: schedule
    • Branch: main
    • Commit: ab9e20f
    • Completed: 2026-10-04T02:32:35Z

  17. github-actions commented on Oct 5, 2026

    @github-actions
    github-actionsbot
    on Oct 5, 2026 – with GitHub Actions
    ContributorAuthor

    Another unhealthy run

    CI run #6262, attempt 1 concluded with cancelled.

    • Trigger: schedule
    • Branch: main
    • Commit: ab9e20f
    • Completed: 2026-10-05T02:15:57Z
  18. jpascucci-nv
    added a commit that references this issue on Oct 5, 2026

    Reset timeout to 60 minutes (#3019)

    8269b3b
  19. leofang
    assigned
    jpascucci-nv
    on Oct 5, 2026

  20. github-actions commented on Oct 6, 2026

    @github-actions
    github-actionsbot
    on Oct 6, 2026 – with GitHub Actions
    ContributorAuthor

    Another unhealthy run

    CI run #6281, attempt 1 concluded with cancelled.

    • Trigger: schedule
    • Branch: main
    • Commit: ea58e1b
    • Completed: 2026-10-06T05:10:02Z

  21. github-actions commented on Oct 7, 2026

    @github-actions
    github-actionsbot
    on Oct 7, 2026 – with GitHub Actions
    ContributorAuthor

    Another unhealthy run

    CI run #6316, attempt 1 concluded with cancelled.

    • Trigger: schedule
    • Branch: main
    • Commit: 4c893de
    • Completed: 2026-10-07T01:35:14Z
  22. Andy-Jost
    added a commit that references this issue on Oct 7, 2026

    test(cuda.core): fold the pairwise cross-type object-protocol tests i…

    93eb6d5

  23. github-actions commented on Oct 8, 2026

    @github-actions
    github-actionsbot
    on Oct 8, 2026 – with GitHub Actions
    ContributorAuthor

    Another unhealthy run

    CI run #6342, attempt 1 concluded with failure.

    • Trigger: schedule
    • Branch: main
    • Commit: 93eb6d5
    • Completed: 2026-10-08T02:42:49Z

  24. github-actions commented on Oct 9, 2026

    @github-actions
    github-actionsbot
    on Oct 9, 2026 – with GitHub Actions
    ContributorAuthor

    Another unhealthy run

    CI run #6359, attempt 1 concluded with cancelled.

    • Trigger: schedule
    • Branch: main
    • Commit: a2b1e31
    • Completed: 2026-10-09T01:18:54Z

  25. github-actions commented on Oct 10, 2026

    @github-actions
    github-actionsbot
    on Oct 10, 2026 – with GitHub Actions
    ContributorAuthor

    Another unhealthy run

    CI run #6381, attempt 1 concluded with cancelled.

    • Trigger: schedule
    • Branch: main
    • Commit: c7e6f09
    • Completed: 2026-10-10T01:16:47Z

  26. github-actions commented on Oct 11, 2026

    @github-actions
    github-actionsbot
    on Oct 11, 2026 – with GitHub Actions
    ContributorAuthor

    Another unhealthy run

    CI run #6383, attempt 1 concluded with cancelled.

    • Trigger: schedule
    • Branch: main
    • Commit: c7e6f09
    • Completed: 2026-10-11T01:14:52Z

  27. github-actions commented on Oct 12, 2026

    @github-actions
    github-actionsbot
    on Oct 12, 2026 – with GitHub Actions
    ContributorAuthor

    Another unhealthy run

    CI run #6384, attempt 1 concluded with cancelled.

    • Trigger: schedule
    • Branch: main
    • Commit: c7e6f09
    • Completed: 2026-10-12T01:22:23Z

  28. github-actions commented on Oct 13, 2026

    @github-actions
    github-actionsbot
    on Oct 13, 2026 – with GitHub Actions
    ContributorAuthor

    Another unhealthy run

    CI run #6406, attempt 1 concluded with cancelled.

    • Trigger: schedule
    • Branch: main
    • Commit: 5fe0749
    • Completed: 2026-10-13T01:17:41Z
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

P0High priority - Must do!ci-workflow-healthAssigned to rolling issues managed by the ci-workflow-health.yml workflow.

Type

No type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions