Skip to content

node-info: keep every GPU at boot when the DirectX registry is stale, and recover an empty Windows inventory - #98

Draft
dmmdea wants to merge 1 commit into
NVIDIA:developfrom
dmmdea:up/node-info-stale-luid-gate
Draft

dmmdea wants to merge 1 commit into
NVIDIA:developfrom
dmmdea:up/node-info-stale-luid-gate

Conversation

@dmmdea

@dmmdea dmmdea commented Sep 20, 2026 •

Copy link
Copy Markdown

Description

A Windows node can publish "GPUs": [] for hours after boot. detectGPUs() runs once at startup and gates DXGI adapters by the LUIDs under HKLM\SOFTWARE\Microsoft\DirectX\*\AdapterLuid. At boot that registry still carries the previous boot's LUIDs (measured: rewritten about 36 s after node-info started), so the RDP-clone gate (the fix in #68) drops every physical adapter, and nothing ever re-detects. Anyone starting the app at logon hits this on every Windows node.

What changed (services/nvpair-node-info):

  • gpu_windows.go: the candidate enumeration is split from a pure selectPhysicalAdapters. When the registry gate would drop every candidate it is treated as stale: all candidates are kept and one warning is logged. The RDP-clone case, where the gate keeps a subset, is unchanged.
  • stats_windows.go: an injectable detect; a goroutine re-detects every 10 s while the inventory is empty, then every 60 s, and publishes through the existing GPUInventory recovery seam (the Darwin build already uses it) only when the set of stats keys changed. The 1 s stats tick never waits on it.
  • stats.go: comment update for GPUInventory.
  • README section and tests.

Release intent

Changelog title

Windows nodes keep every GPU when the DirectX registry is stale at boot

Changelog body

  • The node GPU list is no longer empty for hours after a boot-time start on Windows.
  • The GPU inventory is re-detected in the background until adapters appear, then refreshed when the adapter set changes.

Bumps

  • services: patch
  • nvpair-cluster-manager: none
  • nvpair-engine-manager: none
  • nvpair-errors: none
  • nvpair-job-scheduler: none
  • nvpair-manual-nodes: none
  • nvpair-node-info: patch
  • nvpair-node-scanner: none
  • nvpair-node-settings: none
  • nvpair-proxy: none
  • nvpair-tui: none
  • nvpair-ui-broker: none
  • nvpair-workload-manager: none

Scope

Included: the Windows GPU inventory path in services/nvpair-node-info (stale-registry gate, background re-detect loop, README, tests). Excluded: Linux and Darwin collectors, the wire format, and services/versions.json / CHANGELOG.md (bot-owned).

Validation

Rebased on develop; one signed-off commit.

  • cd services/nvpair-node-info && go vet ./... && go test ./... -count=1 on Windows: pass. This includes the Windows-only tests that CI's ubuntu job never runs: TestSelectPhysicalAdapters (5 cases), TestWindowsCollectorRecoversEmptyGPUInventory, TestWindowsCollectorRepublishesChangedGPUInventory.
  • GOOS=linux go vet ./... in services/nvpair-node-info: pass.
  • node scripts/spdx-headers.mjs: 0 missing.
  • Each guard was broken once to see its test go red (stale branch reverted, one-shot loop).
  • Live on a 3-GPU Windows host with the stale-registry map: all three adapters kept; the real binary on a side port shows 3 rows, no duplicates, dynamic fields joined.

Risk

Low. Windows inventory path only; the gate's clone-removal behaviour is preserved whenever it keeps at least one adapter. The re-detect goroutine runs off the stats tick and never blocks it.

Checklist

  • I have read the Contributing Guidelines.
  • Every commit is signed off (git commit -s), certifying the Developer Certificate of Origin.
  • New or existing tests cover the change.
  • Relevant documentation is updated.
  • I checked the diff, changed filenames, and commit messages for credentials, private data, internal URLs, internal issue identifiers, and generated artifacts.
  • I recorded the validation commands and results above.
  • I declared version bumps in the release-intent block above. services/versions.json is written by automation — do not edit it by hand.

@Noah-Tervalon-Nvidia
Noah-Tervalon-Nvidia changed the base branch from main to develop September 21, 2026 21:57
… and recover an empty Windows inventory

detectGPUs ran once at start and gated DXGI adapters by the LUIDs under
HKLM\SOFTWARE\Microsoft\DirectX\*\AdapterLuid. At boot that registry still
holds the previous boot's LUIDs (it is rewritten tens of seconds after the
worker starts), so the gate dropped every physical adapter and the node
reported no GPUs until a restart.

Two changes: the gate is stale-safe (when it would drop every non-software,
non-virtual adapter it is skipped and logged), and the Windows inventory
re-detects off the stats tick — every 10 s while empty, every 60 s once
known, republishing only when the adapter set changed — through the
GPUInventory recovery seam the Darwin build already uses.

Tests: a fake registry map with stale LUIDs keeps every adapter; an empty
first detection recovers on the next loop pass and the recovered adapter
picks up its PDH stats by LUID key.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Signed-off-by: Daniel Martinez <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant