Skip to content

fix(scanner): re-enrich the local node on the peer refresh tick - #5

Open
canja006 wants to merge 1 commit into
NVIDIA:developfrom
canja006:fix-self-enrichment-staleness
Open

canja006 wants to merge 1 commit into
NVIDIA:developfrom
canja006:fix-self-enrichment-staleness

Conversation

@canja006

@canja006 canja006 commented Sep 4, 2026

Copy link
Copy Markdown

Fixes #4.

onBrowse deliberately skips our own entry, so the path that re-enriches every peer on every browse never runs for self. publishSelf — the only thing that enriches self — is called at startup, on service (un)register, and after an identity or address change. Nothing was left to converge the local node's GPU, CPU and memory figures, so they stayed at whatever they were when the daemon started.

This calls publishSelf() on the refreshPeersLoop tick that already refreshes advertised addresses, the mesh, cluster identity and models.

Verified on hardware, two paired nodes (Ubuntu + RTX 3060, macOS):

  • load a model with GPU offload → discovery:get-nodes follows within 20 s, where before it never moved at all
  • unload → follows back
  • node-info and the desktop UI were correct throughout; only the discovery snapshot was frozen

go vet and go test ./... clean in services/nvpair-node-scanner.

One tradeoff worth flagging: publishSelf emits node-updated unconditionally, so this adds one self update per 15 s tick. The tidier alternative is an applyInfo path with a changed check, mirroring applyModels — happy to switch to that shape if you prefer it.

@Noah-Tervalon-Nvidia

Copy link
Copy Markdown
Collaborator

Thank you for your PR and interest in PAIR! In general we prefer pulls to pushes, so making the update in applyInfo would be the right direction.

@canja006

Copy link
Copy Markdown
Author

Happy to move it — but I can't find applyInfo in the tree, and I'd rather ask than guess at which function you mean.

I checked main, origin/develop, the full history (git log --all -S), every file type, case-insensitively: no match. The apply* functions that do exist are all in other services — applyDiscovery (cluster-manager, engine-manager, errors), applyMembers/applyTombstones/applyRemovalProofs (roster), applyNodesChanged/applyUpsert/applyTelemetry (job-scheduler) — none in nvpair-node-scanner.

The closest thing to an "apply info" step in the scanner is the tail of enrichInfoCandidates:

node.GPUs = info.GPUs
node.CPU = info.CPU
node.Memory = info.Memory

reached through enrich() (all candidate IPs) or enrichAt() (one host). If that's what you meant, I'll work there.

One thing worth separating out in the push/pull framing, in case it changes the answer: publishSelf already pulls. publishSelfLocked builds the record, calls d.enrichAt(&node, loopbackHost) — the same node-info fetch a peer gets — and only then does d.dir.upsert(node) and d.emit(...). So the tick in this PR does fetch over loopback; what it also does is republish the whole record, and I assume that's the part you'd rather not have on a timer.

If so, the narrower change is to re-run only the apply step for our own entry: fetch node-info over loopback on the refresh tick and update the existing directory node's GPUs/CPU/Memory in place, no upsert, no emit. That keeps it a pull and touches three fields instead of republishing. Say the word and I'll rewrite it that way.

For context on why self is the odd one out: onBrowse returns early when the event's UUID is our own, so self never gets the per-browse re-enrichment every peer gets, and refreshPeersLoop refreshes advertised addresses, the mesh, models and cluster identity — nothing that touches self's hardware figures. Hence the stale numbers in #4.

If applyInfo lives on a branch that isn't published, just say so and I'll match its shape once it lands.

@Noah-Tervalon-Nvidia
Noah-Tervalon-Nvidia changed the base branch from main to develop September 21, 2026 21:54
@canja006

Copy link
Copy Markdown
Author

@Noah-Tervalon-Nvidia — while you are here, this one is still waiting on a question from 12 Sep.

I could not find applyInfo anywhere in the tree: main, develop, full history via git log --all -S, every file type, case-insensitively. No match. So I do not know which function you meant.

If it lives on an unpublished branch, just say so and I will match its shape once it lands. If you meant the apply step at the tail of enrichInfoCandidates, I will work there — and the narrower change I offered then still stands: fetch node-info over loopback on the refresh tick and update the existing directory node's GPUs/CPU/Memory in place, with no upsert and no emit. That keeps it a pull and touches three fields instead of republishing the record.

Happy to rewrite it either way. I would just rather ask than guess.

@Noah-Tervalon-Nvidia

Copy link
Copy Markdown
Collaborator

@canja006 Sorry for the slow reply and the confusion. applyInfo doesn't exist; it's the name from your PR description ("an applyInfo path with a changed check, mirroring applyModels"), and my comment was a yes to that.

The local node's hardware figures were captured once and then never refreshed.
onBrowse deliberately skips our own entry because self is registry-driven, and
publishSelf is otherwise only called when the registry moves: a service
(un)register, an identity change, an address change. refreshPeersLoop refreshes
advertised addresses, the mesh, models and cluster identity, none of which touch
the local figures. Nothing was left to converge them.

Measured on an RTX 3060: a model was loaded and the card went to 9.1 GB while
the directory kept serving the 0.12 GB it held at daemon start. node-info was
serving the true value every two seconds the whole time.

refreshSelfInfoOnce pulls the same loopback node-info a peer's browse would, and
moves three fields through a new directory.applyInfo that mirrors applyModels:
it guards on the node-info endpoint, compares before writing, and reports
whether anything actually changed. The record is therefore republished once per
real change rather than once per tick, and the whole self record is not rebuilt
on a timer.

Enrichment dials loopback, never the advertised ip=, keeping the property
publishSelfLocked documents: a firewall block on inbound to our own LAN address
must not blank the local card.

applyInfo treats a nil facet and a zeroed one as different. nil means node-info
has never answered for it, which is not the same statement as a reading of zero,
and collapsing the two would let a node that stopped reporting look like one
reporting idle hardware.

Signed-off-by: canja006 <[email protected]>
@canja006
canja006 force-pushed the fix-self-enrichment-staleness branch from 0b8d8aa to 42f776e Compare October 2, 2026 05:26
@canja006

canja006 commented Oct 2, 2026

Copy link
Copy Markdown
Author

Thanks — that settles it. Rewritten to the narrower shape.

What changed since 0b8d8aa:

  • directory.applyInfo mirrors applyModels: it guards on the node-info endpoint (service present, ip and port unmoved), compares before writing, and returns (node, changed, ok).
  • daemon.refreshSelfInfoOnce replaces the publishSelf() call on the tick. It pulls the same loopback node-info a peer's browse would, then moves GPUs, CPU and memory through applyInfo. The record is republished once per real change rather than once per tick, and the whole self record is no longer rebuilt on a timer.
  • Enrichment still dials loopback rather than the advertised ip=, keeping the property publishSelfLocked documents.
  • Rebased onto current develop.

One deliberate call worth flagging, in case you'd rather it went the other way: applyInfo treats a nil facet and a zeroed one as different. A nil means node-info has never answered for that facet, which is not the same statement as a reading of zero, and collapsing the two would let a node that stopped reporting look like one reporting idle hardware. It is a one-line change if you prefer them equal.

Tests in self_info_refresh_test.go:

  • TestApplyInfoGuard — mirrors TestApplyModelsGuard: removed node, re-addressed node, moved port, and a node with no node-info service.
  • TestApplyInfoChangedCheck — the compare itself, including the nil-vs-zeroed rule and a card appearing or vanishing.
  • TestRefreshSelfInfoOnceConvergesLocalNode — the regression this fixes, 0.12 GB to 9.1 GB. It uses newSelfTestDaemon with the stub on loopback while the node advertises a LAN ip, so it pins the loopback property as well, and it asserts the quiet case: an unchanged re-read reports no change.

gofmt, go vet and go test -race ./... are clean against develop.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Local node's GPU/CPU/memory enrichment is never refreshed after startup

2 participants