Skip to content

Rebuild the terminal interface around nodes and increase UX - #117

Open
Noah-Tervalon-Nvidia wants to merge 69 commits into
developfrom
feature/terminal-interface-uplift
Open

Noah-Tervalon-Nvidia wants to merge 69 commits into
developfrom
feature/terminal-interface-uplift

Conversation

@Noah-Tervalon-Nvidia

Copy link
Copy Markdown
Collaborator

Description

Rebuilds nvpair-tui around a node-first tab layout and closes the gaps against the desktop application, then moves the model catalogue into the backend so both front ends browse one implementation.

A node is the unit an operator reasons about, so the tabs become Nodes, Jobs, Service, Errors, and Logs, and everything specific to one machine hangs off its row. The ten tabs this replaces spread one machine's facts across four of them: its engines in one, its models in another, its ports in a third, its cluster standing in a fourth.

Much of the rest is about claiming no more than the backend guarantees. A port change reports the port actually bound rather than the one requested. Clearing an error is only offered where clearing sticks. The cluster name is labelled as this machine's own. Node presence follows the broker's snapshot instead of a timestamp that never advanced.

How to review this

Seventeen commits, in dependency order. Four are prerequisites, one is the rewrite, and the rest are separable features and fixes. The rewrite (5381e2f1) is 51 files and is best read in the order below — it follows the data rather than the alphabet.

The five views only exist together: the shell sizes them, they share its status line and frame budget, and the tab set is the change. Splitting them further would have meant authoring intermediate versions of nodedetail.go and jobs.go that never existed, so the guided read is offered instead.

Read the commits in this order

  1. 6b377e78 frame cap single-sourced across four hops — 5 files
  2. e99efb05 engine:catalog in the engine manager — the +109k is the committed Ollama list; review catalog.go and catalog_test.go
  3. 001b0e4a desktop served from that catalogue — the −109k is the same list leaving Electron
  4. c414dbf0 one inference-dispatcher, from cli-bin
  5. 9a61f3de broker stream errors reported as disconnects
  6. e70fbd21 shared frame primitives (table, status line, row budget)
  7. 5381e2f1 the rewrite — see the file order below
  8. ec7a6c84 release version stamped for the update notice
  9. 144e472f engine launch-arguments editor, ports onto the same write
  10. 32738dcd documentation and the architecture rule
  11. 953a9ab0 CI's staged-binary check learns about the demo client
  12. eaedf559 light-terminal readability
  13. 7f5610da the requestId a settings commit requires
  14. 67ca9630 background detected before Bubble Tea takes stdin
  15. fbb37e0a local settings snapshots kept current
  16. 94d041d2 leaving the Nodes tab returns to the list
  17. 33b5470c settings verdict taken from the backend

Commits 12–17 came from running the build on real hardware. Four of them fix defects in commit 9; each says what the backend actually does, which is worth reading even where the diff is small.

Read the rewrite's files in this order

Start with the shape — 400 lines, and the rest follows from it:

  1. ui/ui.go — the tab set. The whole change in 27 lines
  2. ui/model.go — the shell: frame budget, tab switching, the notice row
  3. ui/keys.go — why the bindings are what they are
  4. ui/view.go (unchanged) — the contract the five views implement

The data, before the screens that render it:

  1. ui/nodeswire.go — the discovery, cluster, and manual wire shapes
  2. ui/nodesmodel.go — merging those three feeds onto one key. The heart of the Nodes tab
  3. ui/nodenames.go — resolving a node id to a name
  4. ui/engineswire.go — engine status and model inventory shapes

The tabs, simplest first:

  1. ui/logs.go, ui/errors.go — smallest, and they establish the view idioms
  2. ui/service.go — worker liveness, log level, data reset
  3. ui/proxystatus.go — per-engine facade readiness and ports
  4. ui/jobs.go — the endpoints and live inference work
  5. ui/nodes.go — the merged node table, filtering, pairing keys
  6. ui/invite.go — pairing events scoped to their session

The drill-down and its two satellites — the largest file, read last:

  1. ui/nodedetail.go — engines and models for one machine
  2. ui/nodetelemetry.go — the direct /v1/node-info poll
  3. ui/catalog.go — the downloadable-model browser
  4. ui/demo.go, ui/demoschedule.go — the Inference Demo

Then the tests, which read as the specification:

  1. ui/framebudget_test.go — no view overflows its frame at any supported size
  2. ui/table_test.go — every table has columns at construction
  3. ui/nodesmodel_test.go, ui/nodedetail_test.go, ui/service_test.go — the per-area behaviour
  4. the remaining _test.go files

Deletions need no reading: cluster.go, engines.go, health.go, manualnodes.go, proxies.go, settings.go, workloads.go are the tabs the five replace.

Scope

Included: the terminal interface rewrite, its Inference Demo and update notice; engine:catalog replacing the Electron-only model hub; one inference-dispatcher in cli-bin; an editor for an engine's launch arguments with the two port fields moved onto the same revision-checked write; documentation.

Excluded: no change to routing, scheduling, or proxy behaviour; none to pairing or trust; the desktop renderer's settings UI is untouched, only its catalogue source moved.

Validation

Everything CI runs, run locally on macOS arm64, Go 1.26.3, Node 22:

  • node scripts/spdx-headers.mjs — 1039 checked, 0 missing
  • npm --prefix desktop run verify:build-scripts, service-contracts:check, typecheck, lint, dead-code:check, test:unit (230 passing)
  • npm --prefix desktop run build:modular-binaries -- --force — 13 binaries
  • Go component tests with -race across shared, nvpair-tui, nvpair-ui-broker, nvpair-engine-manager
  • cd services/tests && go test ./... -count=1 -timeout=20m — passes, no failures
  • services/build.sh, then the staged-binary comparison this branch updates

Two failures are pre-existing and reproduce identically on develop in a clean worktree: TestUninstallTerminatesRunningInstance in nvpair-engine-manager, and services/shared/splitlisten/splitlisten_test.go is not gofmt-clean.

Manual, on a Mac and a DGX Spark over SSH: pairing, engine lifecycle, model download, the demo, and the settings editor. The ui.ReleaseVersion stamp was read back out of both binary sets, since a -X against a wrong symbol path fails silently. Terminal background detection was checked in a pty against a terminal that answers the query and one that ignores it.

Risk

The engine settings path is the part to look at. The two port fields previously wrote through engine:set-port and the proxy's own set-port, which meant two writers with no shared revision — the last to finish won and neither could tell it had lost. All three fields are now one revision-checked write through engine:preview-settings and engine:apply-settings. That also makes them editable on a peer, which the old path refused because it had no remote form; whether a given engine accepts the write is the snapshot's Editable answer rather than a rule kept in the front end.

jsonrpc.WorkerFrameBytes raises the inbound frame cap on three hops that carry large worker replies, from the 1 MiB default to 8 MiB. The Ollama catalogue is roughly 1.9 MiB and could not previously cross them; an over-long frame is a terminal read error rather than a dropped message, so the hops have to agree.

No wire-format removals, no persisted-data migrations, no change to how nodes authenticate to each other.

Release intent

Changelog title

A rebuilt terminal interface, and one model catalogue for both front ends

Changelog body

  • The terminal interface is organised around machines: Nodes, Jobs, Service, Errors, and Logs, with a per-machine drill-down for engines, models, ports, and hardware.
  • Engines can be started with your own arguments and environment variables, edited from the terminal interface on this machine or on a paired one. Changes are validated before they are saved.
  • Downloadable models are served by the engine manager, so the terminal interface and the desktop application browse the same catalogue.
  • The Inference Demo runs from the terminal interface's Jobs tab, so a headless machine can demonstrate routing.
  • The terminal interface says when a newer release of PAIR is published, on every tab until dismissed. It installs nothing.
  • Pairing keys follow the words on screen: p pairs, a accepts, f finds a machine by address.

Bumps

  • services: major
  • nvpair-cluster-manager: none
  • nvpair-engine-manager: minor
  • nvpair-errors: none
  • nvpair-job-scheduler: none
  • nvpair-manual-nodes: none
  • nvpair-node-info: none
  • nvpair-node-scanner: none
  • nvpair-node-settings: none
  • nvpair-proxy: none
  • nvpair-tui: major
  • nvpair-ui-broker: patch
  • nvpair-workload-manager: none

Checklist

  • I have read the Contributing Guidelines.
  • Every commit is signed off (git commit -s), certifying the Developer Certificate of Origin.
  • New or existing tests cover the change.
  • Relevant documentation is updated.
  • I checked the diff, changed filenames, and commit messages for credentials, private data, internal URLs, internal issue identifiers, and generated artifacts.
  • I recorded the validation commands and results above.
  • I declared version bumps in the release-intent block above. services/versions.json is written by automation — do not edit it by hand.

A worker reply can be megabytes where a control message is bytes, and a frame
over a hop's cap is not a dropped message: bufio.Scanner cannot resync past an
over-long line, so the read loop ends and the peer dies silently while the child
keeps running.

The cap only works if every hop on a path agrees, so declare it once in
shared/jsonrpc and use it on the three hops that carry those replies -- the
broker's generic worker links, engine-manager's inbound codec, and the terminal
client's link to the broker. The purpose-built links keep their own smaller
caps, sized to what those workers actually send.

Engine-manager is unchanged in value, having already been at 8 MiB. The broker's
worker links and the terminal client move up from the 1 MiB default.

Signed-off-by: Terve <[email protected]>
Both front ends need to browse the models an engine can download, and only one
of them could: the desktop app carried its own Electron-side model hub, so the
terminal interface had no way to offer the same thing. Putting the catalog
behind engine:catalog gives both clients one implementation, next to the engine
manager that performs the pull.

One curated source per engine, because there is no generic GGUF registry worth
browsing. Ollama has no public catalog API, so the list is scraped by a
developer, reviewed as a diff, committed, and compiled in with go:embed --
serving it touches no network. LM Studio's catalog is the lmstudio-community
Hugging Face org, whose repo ids are exactly what lms get accepts, so it is
fetched live, cached for six hours, coalesced across concurrent callers, and
backed off after a failure so a dead upstream is not re-dialled per call.

The catalog is filtered for the platform the models will install on, not the
one serving it: MLX quantizations only install on Apple Silicon, so those rows
are marked appleOnly and dropped for a non-darwin target, and the reply echoes
the platform it filtered for.

Signed-off-by: Terve <[email protected]>
The Add Model browse was assembled in the main process: a committed Ollama list
bundled into the Electron bundle, plus a live Hugging Face fetch. The terminal
interface could not reach any of it, so the two front ends would have had to
maintain separate catalogs of the same models.

Replace the module with a thin relay over engine:catalog. What remains here is
the boundary work the renderer needs and the backend should not care about:
mapping EngineType to the manifest's engine name, translating the platform to
GOOS -- Node says win32 where Go says windows -- and normalizing rows, dropping
any without a pull-ready id rather than offering a model that cannot be
downloaded.

A catalog failure returns an empty hub rather than an error. The modal then
shows its empty state, which is the honest reading for a user who can do
nothing about a dead upstream.

The scraper stays in desktop/scripts: it is a TypeScript development tool and
only its output crosses into the services tree.

Signed-off-by: Terve <[email protected]>
The Inference Demo's HTTP client was built by its own script into its own
tools/ resource directory, which only the desktop app knew about. The terminal
interface runs the same demo and resolves everything beside its own executable,
so a second location meant it could not find the client at all.

Build it into cli-bin with the binaries it sits beside. It is still not a
service -- no entry in versions.json, no supervision -- so it carries the
services version rather than a component version, and it is named explicitly in
the expected-file set so cli-bin stays an exact inventory rather than merely
gaining an exemption.

Fingerprint what actually goes into a binary while here. Only .go files were
hashed, so regenerating an embedded asset -- the engine manager's Ollama
catalogue, now -- left the build believing cli-bin was current, and it shipped
the previous binary with nothing to indicate the list was stale. The
dispatcher's own sources sit outside the services tree and get walked for the
same reason.

Signed-off-by: Terve <[email protected]>
A bufio.Scanner is finished after a read error, including an over-long line,
which it cannot skip past. Treating that like a frame the client merely failed
to parse meant calling Scan again forever: the read loop spun at full speed
while the interface went on claiming the service was ready.

Name the unrecoverable case so the loop can tell the difference and report the
disconnect, and let the supervisor wait for the broker's own teardown before
force-killing it -- the workers it is shutting down are the ones holding the
files a reset is about to delete.

Signed-off-by: Terve <[email protected]>
Three things every view needs and none of them owned: a table that lays its
columns out against the width it was actually given, a status line that
expires on its own, and helpers that force a block of content to an exact
number of rows.

Each exists because the alternative kept going wrong. Columns summed to more
than the terminal and the last one was cut; a status line set during a
thirty-five second operation vanished six seconds in, leaving a screen
indistinguishable from a dropped keypress; and content that rendered one row
too many pushed the footer off the bottom.

Signed-off-by: Terve <[email protected]>
A node is the unit an operator reasons about, so the tabs become Nodes, Jobs,
Service, Errors, and Logs, and everything specific to one machine hangs off its
row rather than living in a tab of its own. The ten tabs this replaces spread
one machine's facts across four of them: its engines in one, its models in
another, its ports in a third, its cluster standing in a fourth.

The five views only exist together -- the shell sizes them, they share its
status line and frame budget, and the tab set is the change -- so this lands as
one commit. Read in the order the pull request gives, which follows the data
rather than the alphabet.

Much of it is about claiming no more than the backend guarantees. A proxy port
change reports the port actually bound rather than the one requested, because a
running engine holding that port wins and the broker resolves it elsewhere.
Clearing an error is only offered where clearing sticks -- delete-by-id on the
reporting node -- so a peer's entry names the node to clear it from instead of
silently reverting on the next sync. The cluster name is labelled as this
machine's own. Node presence follows the broker's snapshot rather than a
timestamp that never advanced.

Every table gets its columns at construction. The broker replays a baseline
snapshot as soon as a view subscribes, which can arrive before the first
WindowSizeMsg, and bubbles indexes its column slice per row cell -- so rows
against a zero-column table panicked rather than rendering empty.

Against the unified proxy the two identities are kept apart deliberately. The
Service tab's worker row keys on the process, engines.ProxyComponent, because
one nvpair-proxy hosts every facade and the broker reports one crash for it;
the facade views key on each engine's ComponentName, which is what addressing
"ollama-proxy:nodes/list" means. Getting that backwards is silent in both
directions, so the invariant is asserted rather than described.

Signed-off-by: Terve <[email protected]>
PAIR carries three numbers and the update check can only use one of them. The
published tags are named for the release version in desktop/package.json; the
services suite version and this component's own version describe parts of the
build. The suite version is currently the larger number, so comparing it
against the feed would not merely be wrong but silently wrong -- every check
concluding this build is ahead and saying nothing, forever.

Rename the symbol to ui.ReleaseVersion to match the vocabulary the release
tooling now uses, and stamp it from desktop/package.json in both build paths:
services/build.sh and .bat for a tarball install, and the desktop's
build-modular-binaries for the copy inside cli-bin. A -X against a symbol path
that does not exist fails silently, so both were verified by reading the value
back out of the built binary rather than by reading the flag.

Signed-off-by: Terve <[email protected]>
The engine manager gained editable launch settings: the arguments and
environment an engine starts with, alongside its server port and the
client-facing proxy port. The desktop app can edit all three; the terminal
interface could edit neither the arguments nor, on a peer, the ports.

"a" on an engine opens its arguments in the notation LAUNCH_TEXT.md defines.
Nothing is saved directly. The draft goes to engine:preview-settings, which
normalizes the text and reports per-field errors, a port conflict, and whether
applying restarts the engine; the commit then carries the settings the preview
returned rather than the text that was typed, so what lands is what was checked.
A restart is armed and confirmed with y, like the other interruptions.

The two port fields move onto the same path. They were writing through
engine:set-port and the proxy's own set-port, which meant two writers with no
shared revision: the last to finish won and neither could tell it had lost.
All three fields are now one revision-checked write, which is also what makes
them editable on a peer -- the backend relays it, and whether a particular
engine will accept the write is the snapshot's Editable answer rather than a
rule kept here that would drift from it.

Ports stay honest across the move. A port is a request, not a promise: a
running engine outranks the proxy, so the backend binds elsewhere and reports
the difference as the effective port. The outcome is read from the snapshot
that follows the write rather than from the write's own reply, which only says
the request was accepted -- and it is reported once, for a change made here,
because those snapshots also arrive when anyone else saves.

Signed-off-by: Terve <[email protected]>
The terminal interface guide still described the screen it replaced, and three
claims elsewhere had become false: that the model hub lives in Electron, that
the demo is the desktop app's alone, and that per-engine arguments are outside
the contract. All three are now the opposite of true.

The guide gains the startup-arguments editor and loses its own contradiction --
it documented the Jobs tab's test traffic in one section and denied having any
in another. Ports are no longer described as local-only, because they are not,
and the note about a port being a request rather than a promise moves into the
guide where an operator will meet it.

The architecture rule gains the rule worth keeping: the three settings fields
are one revision-checked write, and editability is the snapshot's answer rather
than something a front end decides for itself. Both halves are there because
both were got wrong here first.

Signed-off-by: Terve <[email protected]>
services/build.sh now stages inference-dispatcher beside the components, and
the check compares that directory against versions.json exactly -- so it fails
on a binary that is deliberately not a component.

Name the exception rather than loosening the comparison. The check exists to
fail when the build script and the manifest disagree about a component, and a
pattern that skipped anything unexpected would stop doing that. One extra
declared name keeps a second undeclared binary failing.

Signed-off-by: Terve <[email protected]>
Every colour adapts to the terminal's background, but the text drawn on top of
the accent did not: it was a fixed black, which suits the pale blue chosen for
a dark terminal and is unreadable on the dark blue chosen for a light one. It
applied to the active tab and the selected table row -- the row the operator is
looking at, on every tab -- so a light theme made the interface hard to use
rather than merely off-colour.

Pair the foreground with the background so both flip together, and assert the
pairing: a fixed foreground over an adaptive background is legible in exactly
one of the two terminals, and which one is decided by a detection this program
does not control.

Add --appearance for when that detection is wrong. It works by asking the
terminal for its background and waiting for an answer; a terminal that does not
reply -- common over SSH and inside tmux -- leaves lipgloss assuming dark. auto
still detects, light and dark state it outright, and anything else is refused
rather than quietly meaning auto, which would strand the one operator who
reached for the flag.

Signed-off-by: Terve <[email protected]>
Saving an engine's settings failed with "a request identifier is required"
after the check had already passed, which read as the interface calling its own
validated change invalid.

The identifier is the backend's idempotency key. It records a receipt against
it, replays return the original outcome, and a replay carrying different
settings is refused -- so a commit without one is rejected outright. Only the
commit needs it, which is why the preview succeeded first and the failure
arrived at the point of saving.

It is minted when the change is judged rather than when it is sent, so a change
held for a restart confirmation keeps the identifier it was judged with and
confirming is a replay rather than a second write.

The test asserts the marshalled request, not the Go field. `requestId` is
omitempty: unset, it does not travel as an empty string, it vanishes from the
object -- which is exactly how this shipped with a struct field that looked
correct and a payload that was missing it.

Signed-off-by: Terve <[email protected]>
Auto-detection did not work, so the colours had to be declared by hand on any
terminal that is not dark.

lipgloss resolves the background once, lazily, the first time an adaptive
colour is needed -- which is during the first render. By then Bubble Tea owns
the terminal and is reading stdin, so the terminal's reply to the query goes to
its reader, the query times out having learned nothing, and lipgloss falls back
to assuming dark.

Force it at startup instead, while stdin is still ours, and let the sync.Once
keep the answer for every later frame. Measured in a pty against a terminal
that answers with a light background: it is now detected as light, and the
first frame arrives in 0.19s.

The query runs alongside broker startup rather than in front of it, because a
terminal that never answers costs five seconds -- termenv's timeout is a
constant, and the query cannot be abandoned early since it owns the terminal
until it returns. That wait is not new: clean develop pauses the same five
seconds on the same terminal, and it is unrelated to --appearance.

Also correct what the documentation claims about when detection fails. SSH is
fine, because the query and its reply travel the connection like anything else.
It is screen and tmux that cannot answer, by their own design, along with
terminals that do not implement the query at all.

Signed-off-by: Terve <[email protected]>
The first settings change on a machine succeeded and every one after it failed
with "settings changed on this device; reload before applying", until the
screen was closed and reopened.

The broker stamps each snapshot with the node's UUID. This screen matched that
against the node argument the local RPCs take, which is empty precisely
because they are local -- so every push for this machine was discarded. The
cached revision stayed at whatever the first read returned while the real one
moved on with each save, and a write carries the revision it was based on.
Match on the row's key, which is the UUID discovery reports and the one the
broker sends.

Verified against a running broker: the snapshot's nodeId and discovery's
hostUuid are the same value, and the engine whose port had just been changed
was sitting at revision 2 against a cache still holding 1.

Make the failure recoverable while here. A revision the backend will not
accept cannot be repaired by repeating the write, so a rejected save now drops
the stale snapshot and re-reads it, and says what happened in terms of what to
do next rather than repeating the backend's instruction to reload -- which the
screen has by then already carried out.

Signed-off-by: Terve <[email protected]>
A node's detail screen replaces the whole tab, and it stayed open when the
operator switched away. Coming back to Nodes put them inside whichever machine
they had last opened rather than on the list they asked for, with the list a
keypress further away and nothing on screen saying why.

Leaving a tab now returns it to its own top-level screen. Expressed as an
optional interface, like the two a view already opts into, because this is not
something every tab should do: a selection or a filter is where the operator
left off and worth keeping. It is for a view that replaces itself with
something else entirely.

Unconditional on the way out, which is safe because a view holding a text
field or an armed confirmation captures the keyboard -- the tab cannot be
changed out from under one, so there is no half-finished state to discard.

Signed-off-by: Terve <[email protected]>
A successful engine port change was reported as a failure: "engine stayed on
:1237 - :1238 is taken", while the engine was in fact listening on 1238 and
nothing at all was on 1237.

The verdict was inferred by comparing the saved ports against the effective
ones. The effective engine port is observed at the moment the apply replies,
and an engine that restarts onto its new port finishes after that -- LM
Studio's server re-launches under launchd, detached, so the manager loses
sight of it and the reading lags a change behind. Reading failure into that
lag invented both the failure and a cause for it.

The snapshot already carries the backend's own verdict as a phase, with a
reason when it failed. Use those. The proxy port is still compared, because
that one is read live from the proxy process rather than observed in passing,
so a proxy that could not take the port it was given is a real difference
worth reporting -- stated as where it landed, without asserting why.

Signed-off-by: Terve <[email protected]>
@Noah-Tervalon-Nvidia

Copy link
Copy Markdown
Collaborator Author

I know this is a MAASSSSIIIVVEEE PR. I'm happy to scope it down into smaller components if that's desired, however, most of the large diffs are effectively copying pieces from the UI and putting them in go rather than typescript.

An inbound request was both a pinned status line and a row of the frame, in
two different wordings. Two prompts for one fact read as two requests, and the
pinned one outranked the status line, so nothing that happened next could be
reported there.

It also described only the first state. Pressing accept does not finish
anything -- it opens the PIN field -- so the prompt went on offering "a to
accept" while the PIN it actually wanted sat on the line below, telling the
operator to do the thing they had just done.

One prompt, in the frame, that follows the request: who is asking and which
keys answer, then which machine is being accepted and that its PIN is wanted.
The status line is left free to report the outcome.

Signed-off-by: Terve <[email protected]>
A machine PAIR is not paired with reported "spark-182c is not answering - no
engine list available", directly above a hardware readout for that same
machine updating every two seconds.

Both were true at once because they travel different paths. Telemetry comes
from an endpoint that needs no pairing; engines come over pin-based mTLS,
which only succeeds against a peer we hold a pin for. The call was made
anyway, and its failure is indistinguishable from silence once it has failed
-- so the one cause the screen could not observe was the one it reported.

Decide before asking instead. A node outside the cluster is not asked, and the
pane says which relationship it has and what would change it: pair from the
Nodes tab, leave the other cluster first, or wait for the handshake. The
models pane stops blaming a stopped engine for the same gap -- models come
from what the node advertises over discovery, and whether an engine is running
is exactly what cannot be seen from outside the cluster.

Signed-off-by: Terve <[email protected]>

@sylvesterkaczmarek sylvesterkaczmarek left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the manual-node/persistence intersection with #60 on current b843b95. The overhaul does not supersede the broker-persistence fix: nvpair-manual-nodes is still the authoritative in-memory worker, the broker README still documents that a worker restart loses entries, and the desktop still owns manual-nodes.json replay.

The new node-first TUI looks compatible with moving that durability into the broker. It continues to use node/add, node/remove and nodes/list unchanged, and the unified row deliberately retains manualID separately from the stable node key, so removing a discovered+manual node sends the worker alias rather than its HostUUID. I do not see a blocker in this part of #117, and #60 still looks like the right follow-up once this lands.

The engines pane said a node's engines were not visible from here, directly
above a model list whose ENGINE column read "Ollama" for every row.

Both came from the same place and only one was right. Discovery carries which
engine serves each model and needs no pairing to do it, so the attribution is
known for any machine on the network. What pairing buys is the engines' state
and their controls -- not their existence, which the screen was already
showing.

Say that instead: name the engines the node advertises, point at the list
below, and promise only what pairing would actually add.

Signed-off-by: Terve <[email protected]>
Comment thread services/nvpair-tui/ui/nodedetail.go
Comment thread services/nvpair-tui/ui/enginesettings.go
Comment thread services/nvpair-tui/ui/nodedetail.go
Comment thread services/nvpair-engine-manager/catalog.go
Comment thread services/nvpair-tui/ui/nodedetail_test.go
Comment thread services/nvpair-engine-manager/catalog_test.go Outdated
Comment thread services/nvpair-engine-manager/catalog.go Outdated
Comment thread services/nvpair-engine-manager/catalog.go
Comment thread services/nvpair-engine-manager/catalog.go Outdated
Comment thread services/nvpair-tui/ui/toast_test.go
Comment thread services/nvpair-tui/ui/nodes.go
Comment thread services/nvpair-tui/ui/jobs.go
Comment thread services/nvpair-tui/ui/proxystatus.go Outdated
Comment thread services/nvpair-tui/ui/jobs.go
Starting an engine behind a slow readiness probe reported failure while the
engine went on to come up.

The engine manager answers a start only once the engine passes its readiness
probe, and the manifests allow that six hundred seconds for Ollama and sixty
for LM Studio. The desktop app gives the same calls a fourteen-minute envelope
for that reason; the terminal interface held them to its thirty-five-second
reply deadline and read the deadline as a verdict.

Start and restart join the operations whose deadline means the reply was slow
rather than that the work failed, and the engine's state push carries the real
outcome. Stop stays out: it is bounded by the manifest's five-second grace, so
a deadline there is a genuine fault.

Signed-off-by: Terve <[email protected]>
The terminal chose each engine's engine:action names and params with
string comparisons against "ollama" inside a switch, and anything that
was not Ollama got LM Studio's spelling, so a third engine would have
been sent another engine's vocabulary without complaint.

Give the engine ids named constants in the shared engine table, and keep
each engine's spelling in one entry of a table keyed by them. An engine
without an entry is refused, and a test fails for any engine in the
shared table that lacks one. What is sent for Ollama and LM Studio is
unchanged.

Signed-off-by: Terve <[email protected]>
The initial read's reply and the errors:update pushes arrive by
different paths, so a reply landing after a push replaced the newer
snapshot with an older one. Keep the push once one has arrived.

Before this machine's identity arrived, every error was clearable, so a
peer's error could be cleared here and restored by the next sync. Hold
the clear until the identity is known and say why.

The column minimums summed past the forty-column floor, and what was
clipped off the right was the message. Let the node name and the message
share the width, weighted towards the message, with minimums that fit.

Also name the "re-render what is held" call instead of writing it as
setErrors(v.errs) in three places.

Signed-off-by: Terve <[email protected]>
Every rejected invite ended "remove the existing relationship first", including rejections that gave no reason or a reason unrelated to membership. Give the already-clustered reason its own remedy and report the rest plainly. The reason codes and their wording move to nodes.go, the only place that uses them.

Signed-off-by: Terve <[email protected]>
The key is t for tail; the label now says both words, so neither vocabulary has to be guessed.

Signed-off-by: Terve <[email protected]>
It lived in errors.go, which does not use it. It now sits beside clampWidth in table.go, and its test is table-driven, covering the cut, multi-byte boundaries, and degenerate widths.

Signed-off-by: Terve <[email protected]>
Drop the account of the old age-based presence rule from the comments; git records it. Point the manual-node shape at where ManualNodeStatus is defined, and put the any-service-answered check in one method so another probed service is one line there.

Signed-off-by: Terve <[email protected]>
Each tab's view type now starts with what it shows and the usual path through it, pointing at docs/terminal-interface.mdx for the user guide. The accounts of which older tabs each one replaced are gone; git records those.

Signed-off-by: Terve <[email protected]>

@kjlubick kjlubick left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, docs/architecture.mdx still says "terminal interface's Proxies tab" which I don't think is accurate anymore.

Comment thread services/nvpair-tui/ui/service.go Outdated
Comment thread services/nvpair-tui/ui/service.go Outdated
Comment thread services/nvpair-tui/ui/service.go Outdated
Comment thread services/nvpair-tui/ui/service.go
Comment thread services/nvpair-tui/ui/service.go Outdated
Comment thread services/nvpair-tui/ui/catalog.go
Comment thread services/nvpair-tui/ui/catalog.go Outdated
Comment thread services/nvpair-tui/ui/demo.go Outdated
Comment thread services/nvpair-tui/ui/demo.go
Comment thread services/nvpair-tui/ui/demoschedule.go
Title, View, and Help are called every frame, Help more than once because the footer is measured to size the content, so the View contract now says they must be cheap and leave the work to Update.

Signed-off-by: Terve <[email protected]>
Near 40 columns the tab bar numbers the tabs it cannot name, the footer keeps quit and help, and full help spreads across columns to fit.

Signed-off-by: Terve <[email protected]>
Every broker reply and push was decoded with its error discarded, so a
shape the broker and this client disagreed about showed up only as a
screen of zero values. Decode through a helper that logs the failure
with what was being read, and skip the two workload pushes that would
otherwise fold in a job with no identity.

The terminal's own log went to stderr, which the full-screen program
hides. While it runs, route it to the Logs tab beside the broker's,
without ever blocking, since the update loop that drains that channel
also logs. The channel is no longer closed when the broker's stderr
ends, because this program still writes to it afterwards.

Signed-off-by: Terve <[email protected]>
Every worker read "ok" from the first frame, before the service had
answered, before any errors snapshot had arrived, and before this
machine's UUID was known to tell its crashes from a peer's. Show "?"
until all three are in, and classify no crash before the UUID.

Also on this tab: the log levels and their names come from applog, the
level is held as a slog.Level starting from NVPAIR_LOG_LEVEL as the
broker's does, a level change no longer claims to have reached every
service (the broker forwards it best effort), and the settings rows
carry whole method names instead of a suffix glued to "settings/get-".

Signed-off-by: Terve <[email protected]>
The Nodes tab read the cluster name once at startup, and the settings worker sends no push when it changes, so a rename made on the Service tab looked unsaved there. The save now carries its method and value to every tab, and the Nodes tab takes the name from it.

Signed-off-by: Terve <[email protected]>
A failed status read was dropped, so the strip kept its last good answer while reads failed; it now marks the facade down. The broker's {ready:false, port:0} for a facade it is not running blanked the port that the error push deliberately kept; both now keep it. A failed subscription is logged instead of discarded.

Signed-off-by: Terve <[email protected]>
The table listed jobs in arrival order, so a backlog of older queued work
pushed new jobs below the fold; it now lists newest first. Jobs for the
same model from the same node, started together, read as duplicates, so
a short ID column shows the tail of each job's ID, which is where the
per-process counters differ.

A failed subscription or baseline read left the table empty for the
session, and a failed identity read left this machine's jobs named by
UUID, with nothing said. Both are retried on the existing five-second
poll until they succeed.

Signed-off-by: Terve <[email protected]>
Answering an inbound request cleared it before the answer went out. The
cluster manager refuses a malformed PIN with an error and leaves the
session open, so a typo lost a request that could still be answered.
The request now stays on screen until the answer settles, and a second
answer cannot be sent while one is in flight.

A second inbound request replaced the first, which stayed live with no
way back to it. Requests now queue, one on screen, with the count of
those waiting. While a node's detail screen is open, a line above it
keeps a waiting request or a live outbound PIN in view.

A cancel that failed left a live invite untracked, with its PIN gone and
nothing to press. It is now put back and can be cancelled again.

The invite helpers move into nodes.go, their only user, and the invite
states are named after the cluster manager's.

Signed-off-by: Terve <[email protected]>
A manual entry whose probe had read one machine's UUID still fell back
to matching by address, so it folded into a different machine that now
answered at the same IP, showing one machine's membership and models
under the other's name. Once an entry has a UUID, it no longer joins a
row with a different UUID of its own.

Roster reads and nodes:changed pushes each replaced the roster, but a
reply and a push reach the view by different paths, so a reply landing
after a newer push put the older roster back. A read now counts only if
no push has arrived since it was sent; pushes carry every change.

Signed-off-by: Terve <[email protected]>
Each kind of reply carried its own node field and its own check, so a
new command could skip the check and report one machine's outcome on
another's screen. Every broker call the screen makes now goes out
through one wrapper that tags the reply with the screen's node, and the
screen drops replies for another node before handling any of them.

Signed-off-by: Terve <[email protected]>
- A second settings change while the first is still applying is refused.
  The screen follows one outstanding write, so the second replaced it
  and the first's verdict was never reported.
- Engine and model keys are refused while the frame has no room for
  that list, rather than acting on a row the operator cannot see.
- engine:install-progress is ignored on a peer's screen: it carries no
  node, so it is always this machine's install.
- Progress frames show a percentage only when one is reported. The
  engine manager omits it while indeterminate and sends -1 on failure,
  which rendered as (0%) and (-1%); a failed install now reads as a
  failure with the engine manager's reason.

Signed-off-by: Terve <[email protected]>
…wered

- The address that answered the last poll is tried first. Walking the
  node's list from the top spent a full timeout on every unreachable
  address ahead of it, every two seconds.
- A reading whose hostUuid names another machine is refused, so an
  address that has passed to a different host does not show that
  host's hardware under this one's name. Rows keyed by a stand-in, and
  this machine over loopback, have nothing to check against.
- Drop the unused msSince field from the telemetry and manual-node
  shapes.

Signed-off-by: Terve <[email protected]>
The only way past a failed load was to close the browser and open it
again, and once the error expired from the status line the list read as
the engine having no models. The failure now stays in the body, and r
asks for the catalog again.

Signed-off-by: Terve <[email protected]>
The progress note counted every request the schedule reached, including
ones whose dispatcher failed to spawn. It now counts on a successful
start, for the run still going, as the desktop app does.

Signed-off-by: Terve <[email protected]>
The architecture page still described pinning a node from the terminal
interface's Proxies tab. The terminal interface no longer has that tab
or any way to pin, so automatic routing is the only case PAIR produces.

Signed-off-by: Terve <[email protected]>
cluster:invite-node answers failed, with no PIN, when the first
exchange does not complete, and with whatever a cancel or teardown
recorded when one got there first. Only rejected was checked, so either
reply read as "invite sent" and left the invite pending. Only pending
now counts as sent.

Signed-off-by: Terve <[email protected]>
A machine can only be in one cluster, and the cluster manager refuses an
accept on a machine that already is. Requests still waiting after a
successful accept came up one by one looking answerable, and each
sender went on showing its PIN until the invite expired. They are now
declined as soon as the pairing succeeds, which tells each sender, and
the outcome says how many were declined.

Signed-off-by: Terve <[email protected]>
@Noah-Tervalon-Nvidia

Copy link
Copy Markdown
Collaborator Author

the following I generated with AI just to capture the information. Not really for human consumption.

Deferred to follow-up PRs

These came up in review and are out of scope for this PR.

Broker

  • Report an optional worker whose binary is missing. startOptionalWorker skips an empty path without reporting it, so the Service tab shows that worker as ok.
  • Clear a proxy's ready state and port after a failed re-bind. The broker keeps the stale values, which is the only case where the status read and the ready push disagree.
  • Report a disabled proxy separately from one that is down, so the proxy strip can show them differently.

Cluster manager / node settings

  • When a machine joins a cluster, finish its other pending inbound pairing requests and notify their senders. Also notify the sender when an accept is refused because the machine is already in a cluster. The terminal now declines waiting requests itself after an accept, but the backend still leaves them pending.
  • A cluster name set while unclustered is lost when the machine joins a cluster.

node-info

  • Report platform and arch, so the terminal can filter engine:catalog for a peer and the desktop can show a peer's real OS.
  • Optionally, distinguish unreported memory and VRAM from 0. This needs a wire change, so only if it matters.

Shared

  • Named shared wire types in place of the terminal's anonymous structs, with constants for method names and for invite states shared with the cluster manager.
  • Engine-manager methods for local model operations, so neither front end copies the per-engine engine:action wire.

Terminal interface

  • Pass a context through the RPC client's Notify.
  • Remove the remaining comments in nvpair-tui/ui that describe past behavior.

Model catalog

  • Fix the Ollama scraper's field mapping (capability chips land in family, tags in parameter_size) and regenerate the catalog.

Desktop

  • Every peer's OS is a hard-coded "Windows" placeholder, and BackendRow uses it to decide install options. Depends on node-info reporting the platform.

Both front ends

  • A shared test fixture that both Inference Demo schedules are checked against, so they can't drift apart.


// dropInbound forgets an inbound request that is settled, bringing the next
// one waiting on screen if it was the one showing.
func (v *nodesView) dropInbound(id string) bool {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we drop the inbound, it would be a good idea to clear any pin typed in already

sequenceDiagram
    actor U as Operator
    participant T as Nodes tab
    participant B as Broker
    participant C as Cluster manager

    C-->>B: Incoming request from Alpha
    B-->>T: Show Alpha as current request
    U->>T: Open PIN field and type Alpha's PIN
    C-->>B: Incoming request from Beta
    B-->>T: Queue Beta behind Alpha

    C-->>B: Alpha's request expires
    B-->>T: Retire Alpha's request
    T->>T: Promote Beta to current request
    Note over T: PIN field stays open<br/>Alpha's typed PIN remains

    U->>T: Press Enter
    T->>T: Read retained PIN and current invite ID
    T->>B: Accept Beta using Alpha's PIN
    B->>C: cluster:respond-to-invite<br/>inviteId = Beta, pin = Alpha's PIN
    Note over C: If the PIN differs from Beta's,<br/>pairing fails and Beta's session is closed
    C-->>B: Beta failed, reason = incorrect-pin
    B-->>T: Report Beta's pairing failure
Loading

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's a good idea, will add.

A request that expired or was cancelled while its PIN field was open
left the field open with the digits in it. The next waiting request
came up under it, and enter sent the first request's PIN as the answer
to the second. The field now closes and clears whenever the request on
screen is dropped.

Signed-off-by: Terve <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants