Skip to content

feat(cuda.core): bridge CUDA driver logs into Python logging - #3081

Open
lijinf2 wants to merge 12 commits into
NVIDIA:mainfrom
lijinf2:driver_log_2_logging
Open

lijinf2 wants to merge 12 commits into
NVIDIA:mainfrom
lijinf2:driver_log_2_logging

Conversation

@lijinf2

@lijinf2 lijinf2 commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

Description

close #671

Adds cuda.core.utils.register_cuda_error_log / unregister_cuda_error_log, a bridge that forwards CUDA driver cuLogs* messages (via cuLogsRegisterCallback) into Python's standard logging module.

  • register_cuda_error_log(logger=None, callback=None) registers a python callback with the driver that maps CUlogLevel to the corresponding Python logging level and emits to a logging.Logger (defaults to logging.getLogger("cuda.driver")). An optional callback can override the default forwarding behavior entirely.
  • unregister_cuda_error_log() tears down the registered callback.
  • Re-registering replaces any previously-registered callback/logger, and the ctypes callback is kept alive via a module-level reference (and guarded by a lock) to avoid the driver holding a dangling function pointer.
  • Exposed from cuda.core.utils.

Checklist

  • New or existing tests cover these changes.
  • The documentation is up to date with these changes.

Add register_cuda_log_bridge()/unregister_cuda_log_bridge() to
cuda.core.utils, forwarding cuLogs* driver logs to a Python
logging.Logger via cuLogsRegisterCallback/cuLogsUnregisterCallback.

Refs: NVIDIA#671
register_cuda_log_bridge() now accepts an optional
callback(logger, level, message) that overrides the default
"[CUDA Driver] %s" forwarding. Non-callable callbacks raise TypeError.
Add tests for callback override, level filtering, and validation.

Refs: NVIDIA#671
…_error_log

Matches CUDA's own Error Log Management terminology (cuLogs* API).

Refs: NVIDIA#671
Also fix custom-callback test: CUlogLevel enum was referenced via a
nonexistent driver.CULOG_LEVEL_* attribute instead of
driver.CUlogLevel.CU_LOG_LEVEL_*.

Refs NVIDIA#671
Also drop the stale userData TODO comment, move the CUlogLevel->logging
level map into _default_callback (its only user), and rename its
log_level param to cu_log_level for clarity.

Refs NVIDIA#671
bytes already carries its own length via len(), so length added no
information beyond what message already has. Updated the custom
callback test to match the new 3-arg signature.

Refs NVIDIA#671
Python closures already give callbacks access to enclosing state (e.g.
logger), and cuLogsRegisterCallback is always registered with userData
as None today, so it carried no information. Updated the custom
callback test to match the new 2-arg signature.

Refs NVIDIA#671
- Replace Chinese usage-comments with English equivalents.
- Assert on exact message count/content instead of loose any() checks.
- Add coverage for unregister() stopping further log forwarding.
- Simplify the custom-callback test: single unconditional DEBUG
  rewrite instead of dead WARNING/ERROR branching.

Refs NVIDIA#671
@copy-pr-bot

copy-pr-bot Bot commented Oct 12, 2026

Copy link
Copy Markdown
Contributor

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@github-actions github-actions Bot added the cuda.core Everything related to the cuda.core module label Oct 12, 2026
@lijinf2 lijinf2 self-assigned this Oct 12, 2026
@lijinf2 lijinf2 added the feature New feature or request label Oct 12, 2026
@coderabbitai

coderabbitai Bot commented Oct 12, 2026 •

Copy link
Copy Markdown

Review in Change Stack →

📝 Summary

Summary by CodeRabbit

  • New Features
    • Added forwarding for CUDA driver messages to Python logging, with error and warning levels mapped to corresponding Python levels.
    • Added options to use a custom logger or callback, and controls to register or stop message forwarding.
    • Driver messages can be filtered using standard logger settings, and callback handlers can customize how messages are processed.

Walkthrough

The change adds a bridge from CUDA driver log callbacks to Python logging. It exposes registration functions through cuda.core.utils, documents the API, and adds tests for forwarding, lifecycle, filtering, and custom callbacks.

Changes

CUDA error logging

Layer / File(s) Summary
Bridge API and registration
cuda_core/cuda/core/utils/_logging.py, cuda_core/cuda/core/utils/__init__.py, cuda_core/docs/source/api.rst
Adds and exports register_cuda_error_log and unregister_cuda_error_log. The bridge maps CUDA levels, decodes messages, supports raw callbacks, serializes registration, and retains the callback while registered. The API reference describes the bridge and its functions.
Bridge behavior validation
cuda_core/tests/test_cuda_log_bridge.py
Tests logger forwarding, unregistration, replacement, file output, level filtering, custom handlers, custom callbacks, cleanup, and invalid callback rejection.

Assessment against linked issues

Objective Addressed Explanation
Provide a hook to a custom Python logger and allow opt-in through an API in cuda.core.utils [#671] ✅
Support in-memory access to CUDA driver error logs [#671] ✅

Suggested reviewers: andy-jost

Priority: ➖ Normal

Change: Feature

Merge Risk: 🟡 Moderate · up to 1f7e5

The bridge tests can fail on supported driver and device configurations, and the public API is missing its required release-note entry. Address these before merging to keep the test workflow reliable and document the new API.

  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Comment @coderabbitai help to get the list of available commands.

✨ Finishing Touches 💡 1
🛠️ Fix failing CI checks 💡
  • Commit to this branch
  • Create a new PR

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 3

🧹 Nitpick comments (1)
cuda_core/cuda/core/utils/__init__.py (1)

20-23: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

suggestion: Add utils.register_cuda_error_log and utils.unregister_cuda_error_log to the cuda.core.utils API reference and to the 1.3.0 release notes. Document the default logger name "cuda.driver", replacement on re-registration, and that the underlying cuLogs* driver API requires CUDA 12.9 or newer. Do not confuse this driver requirement with the separate cuda-bindings 12.9.8 version floor.


ℹ️ Review info
⚙️ Run configuration
  • Configuration used: Repository: NVIDIA/cuda-python/.coderabbit.yaml
  • Review profile: CHILL
  • Plan: Enterprise
  • Run ID: ce5d1f11-47d8-4648-ae02-f15452c3bab9
📥 Commits

Reviewing files that changed from the base of the PR and between 0271e3a and 4e823c9.

📒 Files selected for processing (3)
  • cuda_core/cuda/core/utils/__init__.py
  • cuda_core/cuda/core/utils/_logging.py
  • cuda_core/tests/test_cuda_log_bridge.py

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.

Comment on lines +17 to +21
@pytest.fixture(autouse=True)
def teardown_bridge():
yield
# Unregister after every test so a leaked bridge can't leak into others.
unregister_cuda_error_log()

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🩺 Stability & Availability | 🟠 Major | ⚡ Quick win

important: Nothing in this module skips the tests when the driver is older than CUDA 12.9, where cuLogs* does not exist. On such drivers, register_cuda_error_log raises and every forwarding test fails instead of being skipped. The bindings test in cuda_bindings/tests/test_cuda.py:977-1007 already guards the same API with driver_version_less_than(12090). Add an equivalent module-level skipif here.

Comment on lines +32 to +34
# cuDeviceGet(9999) is out of range; driver writes two ERROR logs
with caplog.at_level(logging.ERROR, logger=logger_name):
driver.cuDeviceGet(9999)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🩺 Stability & Availability | 🟡 Minor | ⚡ Quick win

🔎 Supported by static analysis

🏁 Script executed:

printf '%s\n' '--- cuda_core/tests/AGENTS.md ---'
nl -ba cuda_core/tests/AGENTS.md
printf '%s\n' '--- test module ---'
nl -ba cuda_core/tests/test_cuda_log_bridge.py
printf '%s\n' '--- relevant conftest fixtures ---'
nl -ba cuda_core/tests/conftest.py | sed -n '210,245p'
printf '%s\n' '--- changed test diff ---'
git diff 0271e3ad33d56bd34530110a1decb8bebfdb0cb4 4e823c9fa263b0c817941be0e94887b71137abaf -- cuda_core/tests/test_cuda_log_bridge.py

Repository: NVIDIA/cuda-python

Length of output: 29840


important: Establish a current CUDA context for these driver tests.

driver.cuInit(0) initializes the driver but does not set a current context. The six tests that call driver.cuDeviceGet(9999) therefore violate cuda_core/tests/AGENTS.md and can inherit no or stale context. Add init_cuda independently of any skip for unsupported driver versions.

Suggested fix
-def test_cuda_driver_log_captured_via_caplog(caplog):
+def test_cuda_driver_log_captured_via_caplog(caplog, init_cuda):
 
-def test_cuda_driver_log_written_to_file(tmp_path):
+def test_cuda_driver_log_written_to_file(tmp_path, init_cuda):
 
-def test_cuda_driver_log_rebind_logger(caplog):
+def test_cuda_driver_log_rebind_logger(caplog, init_cuda):
 
-def test_cuda_driver_log_level_filtering(caplog):
+def test_cuda_driver_log_level_filtering(caplog, init_cuda):
 
-def test_cuda_driver_log_alert_counter():
+def test_cuda_driver_log_alert_counter(init_cuda):
 
-def test_cuda_driver_log_custom_callback_overrides_default(caplog):
+def test_cuda_driver_log_custom_callback_overrides_default(caplog, init_cuda):

Comment on lines +38 to +40
assert len(messages) == 2, f"Expected exactly 2 driver log messages, got:\n{messages}"
assert "Parameter ordinal must be between 0 and 1" in messages[0]
assert "CUDA_ERROR_INVALID_DEVICE" in messages[1] and "cuDeviceGet" in messages[1]

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

🔎 Supported by static analysis

🏁 Script executed:

sed -n '24,85p' cuda_core/tests/test_cuda_log_bridge.py

Repository: NVIDIA/cuda-python

Length of output: 2482


🏁 Script executed:

rg -n -F -- 'unregister_cuda_error_log' cuda_core/tests/test_cuda_log_bridge.py
sed -n '1,210p' cuda_core/tests/test_cuda_log_bridge.py

Repository: NVIDIA/cuda-python

Length of output: 6606


important: Remove driver-specific log wording and count assumptions.

The driver does not guarantee diagnostic text or callback count. These assertions can fail without a bridge failure.

The post-unregister block already checks forwarding independently of message text with assert not any(r.name == logger_name for r in caplog.records). Keep that assertion and remove the redundant literal-text check.

Suggested fix
-    # cuDeviceGet(9999) is out of range; driver writes two ERROR logs
+    # cuDeviceGet(9999) is out of range; the driver emits error logs
...
-    assert len(messages) == 2, f"Expected exactly 2 driver log messages, got:\n{messages}"
-    assert "Parameter ordinal must be between 0 and 1" in messages[0]
-    assert "CUDA_ERROR_INVALID_DEVICE" in messages[1] and "cuDeviceGet" in messages[1]
+    assert messages, f"Expected forwarded driver log messages, got:\n{messages}"
...
-    assert "Parameter ordinal must be between 0 and 1" not in caplog.text

Apply the same rule to the file, handler-count, and custom-callback checks: assert non-empty forwarded output or >= 1, not exact driver text or exactly two records.

📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
assert len(messages) == 2, f"Expected exactly 2 driver log messages, got:\n{messages}"
assert "Parameter ordinal must be between 0 and 1" in messages[0]
assert "CUDA_ERROR_INVALID_DEVICE" in messages[1] and "cuDeviceGet" in messages[1]
assert messages, f"Expected forwarded driver log messages, got:\n{messages}"

@lijinf2 lijinf2 added this to the cuda.core 1.3.0 milestone Oct 12, 2026
@lijinf2

lijinf2 commented Oct 12, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test 4e823c9

@github-actions

Copy link
Copy Markdown
Contributor

@lijinf2

lijinf2 commented Oct 12, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test 1f7e59a

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1


ℹ️ Review info
⚙️ Run configuration
  • Configuration used: Repository: NVIDIA/cuda-python/.coderabbit.yaml
  • Review profile: CHILL
  • Plan: Enterprise
  • Run ID: 0bd442cf-5439-4fb9-b769-bbe60e528a2a
📥 Commits

Reviewing files that changed from the base of the PR and between 4e823c9 and 1f7e59a.

📒 Files selected for processing (1)
  • cuda_core/docs/source/api.rst

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.

Comment on lines +132 to +133
:func:`~cuda.core.utils.register_cuda_error_log` bridges CUDA driver
``cuLogs*`` messages into Python's standard :mod:`logging` module.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

🔎 Supported by static analysis

🏁 Script executed:

#!/bin/bash
set -euo pipefail
rg -n -i 'register_cuda_error_log|unregister_cuda_error_log|driver logging|driver log' cuda_core/docs/source/release

Repository: NVIDIA/cuda-python

Length of output: 156


🏁 Script executed:

set -euo pipefail
printf '%s\n' '--- release files ---'
rg --files cuda_core/docs/source/release
printf '%s\n' '--- logging-related release text ---'
rg -n -i 'log|cuda\.core\.utils|public api|api' cuda_core/docs/source/release || test "$?" -eq 1

Repository: NVIDIA/cuda-python

Length of output: 7273


important: Add a release-note entry for register_cuda_error_log and unregister_cuda_error_log. The release notes contain no entry for this new public API or CUDA driver logging.

Source: Path instructions

@Andy-Jost Andy-Jost left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Putting a hold on this to prevent accidental merge. The callback runs Python inside a driver callback and _cpp/rt/DESIGN.md ("Which channel to use") rules that out for driver callbacks to avoid deadlocks. The capture needs to be a GIL-free C function in the _cpp/rt layer.

More generally I think this feature is complex, so we need a functional spec with sign-off.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cuda.core Everything related to the cuda.core module feature New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support the driver error logging system

2 participants