Keep same-numbered jobs apart and apply each job's events in order - #142

Description

Four fixes to the job list on Overview.

• Jobs that share a number stay apart. Each engine's proxy numbers its jobs from 1, and the numbering restarts with the proxy. The desktop keyed jobs by origin node and number alone, so an Ollama job and an LM Studio job with the same number shared one card, and a job from before a restart was replaced by a new job that reused its number. The desktop now keys a job by (originatedFrom, engine, runId, id), the identity the broker store and the scheduler already use. Workload gains runId, which the proxy already sends.
• A delayed update can no longer undo a later one. A peer's events for one job can reach the broker out of order: the workload manager retries a send that timed out, and its re-sync frames skip dedup, so a late copy of an earlier event can land after a later one. The broker store merged by state rank, so a late queued on node A, arriving after the job had moved to node B, moved it back to A. When both the stored record and the incoming event carry the origin's seq, the store now applies only a higher one and never replaces a terminal record. The node-loss and staleness sweeps fail a record as an inferred guess that keeps its seq, and an origin event still overrides such a guess, unless its seq is lower: a retried copy of an earlier event that lands after a sweep is rejected rather than moving the job back. Events without seq keep the existing rules.
• The proxy writes each job's events in seq order. A job's events were numbered under a lock but written after it was released, so the disconnect watcher's terminal could reach the wire ahead of an earlier event. The new jobEvents type numbers and writes each event under the job's lock and replaces the request-local record, lock, once-guard and re-point closure. Every consumer already rejected the one inversion the old code allowed, so nothing visible changes today. The order is now structural, which the third pull request relies on.
• A refresh no longer drops updates that arrive while it loads. On a reconnect the job list re-fetched its snapshot and replaced itself with it, so a removal or a new job that arrived during the fetch was lost. The initial load and the refresh now share one loader that records every update during the fetch and replays them onto the snapshot in arrival order.

Removals come in two forms. The broker's workloads:remove still names only origin and id, and retires every engine and run that shares them. Electron's own evictions also name the engine and run, and drop only that job. workloadKeysRemovedBy resolves either form against the catalog. The exact form is a single lookup rather than a scan, because emptying the catalog sends one removal per entry and the catalog can hold the broker's whole 10,000-record history.

This is the second of three pull requests that make job states accurate. The first, #141, changes copy and color, and the two merge cleanly in either order. The third makes In flight and Running reflect engine slots.

Release intent

Changelog title

Job list fixes

Changelog body

• An Ollama job and an LM Studio job with the same job number no longer share one card.
• Jobs from before a restart are no longer replaced by new jobs that reuse their numbers.
• A delayed update can no longer move a job back to a node it had already left.
• Reconnecting no longer brings back a removed job or hides a new one.

Bumps

• services: patch
• nvpair-cluster-manager: none
• nvpair-engine-manager: none
• nvpair-errors: none
• nvpair-job-scheduler: none
• nvpair-manual-nodes: none
• nvpair-node-info: none
• nvpair-node-scanner: none
• nvpair-node-settings: none
• nvpair-proxy: patch
• nvpair-tui: none
• nvpair-ui-broker: patch
• nvpair-workload-manager: none

The nvpair-ui-broker and nvpair-proxy binaries change. The workload manager changes only in its spec.

Scope

Audited:

• nvpair-proxy, the producer. Every emit site now goes through jobEvents: admission, each dispatch and each gap between attempts (repoint), the commit (start), and the deferred reporter and disconnect watcher (finish).
• Every caller of the broker store. Proxy events, peer upserts and removals relayed by nvpair-workload-manager, the node-loss sweep, the stale-origin sweep, persisted history and the scheduler's replay. Both sweeps apply as inferred and copy the stored seq, so provenance decides them before the seq rule, and an origin event with a lower seq cannot override them. Persisted history is reloaded through ParseIncoming, so seq survives a broker restart. A heartbeat re-assertion still counts as a sighting even when the merge rejects it, because the sighting is recorded before the merge.
• In-order streams decide exactly as before. The proxy never emits a lower state, a second terminal, or a re-point to the node a job is already on, so every event in an in-order stream is either a higher rank or a placement change. The old rank rule applied both of those too.
• nvpair-workload-manager. Its dedup key already includes engine, run and seq, so there is no code change. Spec statements about the broker's key and ordering are updated.
• nvpair-job-scheduler. It already keys by the four-part identity and receives only updates the store accepted. Unchanged.
• The desktop. The Electron bridge (seed, upsert, broker removals, its own evictions), the renderer store (batched flush, and updates that arrive while the initial or refreshed snapshot loads), the job list, card and connection lines, and frontend-api.md.
• The wire. No JSON-RPC method or payload changes. runId and seq were already sent; the desktop now reads runId. The regenerated services-api.md adds jobevents.go as a dynamic notify site. It sends the same four workload:* notifications that proxy.go sent.

Excluded adjacent work:

• The terminal UI (nvpair-tui) still keys jobs by origin and number, so it still merges same-numbered jobs from different engines or runs. #117 replaces that view with one keyed by the same four-part identity, so this pull request leaves it alone. Because of it, the broker's stale-origin sweep still skips a record when another live record shares its origin and number. Once both this and Rebuild the terminal interface around nodes and increase UX #117 merge, nothing keys on that pair and a small follow-up removes the skip.
• The broker-to-client workloads:remove still names only origin and id. No producer emits a removal today.
• Engine-slot tracking, which is the third pull request.

Validation

Go:

• services/nvpair-proxy: go vet ./... and go test ./... pass. The new jobevents_test.go covers ordering, the lifecycle, terminal methods, a single terminal, no-op re-points and a nil job. The ordering test asserts that the job's lock is held while an event is written. With the old order restored temporarily (write after unlocking), it failed 3 runs out of 3.
• services/nvpair-ui-broker: go vet ./... and go test ./... pass, including 8 new tests in store_seq_test.go and markworkloadfailed_test.go, which pins that the sweeps' failure keeps seq. With the lower-seq check removed temporarily, the delayed-event test fails.
• services/tests (cross-process): go test ./... passes, with 86 passed and 1 skipped (TestBrokerShutsDownOnSignal, which cannot signal a process on Windows). The workload tests all ran, including cross-engine identity, out-of-order suppression, node loss and workload-manager rehydration. After the review fixes, go test ./... -run Workload passes again.
• -race was not run. This Windows machine has no C toolchain, and CI does not run it.

From desktop/, all passing:

• npm run typecheck
• npm run lint
• npm run dead-code:check
• npm run test:unit (247 passed, 2 skipped: Unix-only wipe-script tests). New: workload-identity.test.ts (11 tests) and workloads-store-identity.test.ts (7 tests). The two refresh tests fail against the old store.
• npm run service-contracts:check
• npm run build:modular-binaries -- --force (run before the review fixes; CI's platform builds cover the later commits)

node scripts/spdx-headers.mjs reports no missing headers.

Still to do by hand:

• With Ollama and LM Studio both running, run the Inference Demo, which sends to both engines, and confirm that jobs with the same number show as separate cards.
• Restart PAIR and confirm that earlier jobs stay in the list next to new jobs that reuse their numbers.

Risk

• Store rule. It applies only when both sides carry seq, and in-order streams decide exactly as before (see Scope). A producer that sends no seq keeps the rank rule. Against a sweep's guess it only rejects an origin event older than the record the guess replaced; the returning node's re-assertion carries an equal seq and still clears the guess.
• Desktop keys. Every catalog key is built by workloadKey, including the connection lines' anchor key from the card's data attributes. A producer without runId gets an empty run, consistent across its events.
• Lock order in the proxy. jobEvents holds the job's lock while writing to the codec. The order is job lock, then codec lock, and nothing that holds the codec lock takes a job's lock. A blocked stdout already stalled both the request goroutine and the watcher at the codec before this change.

Checklist

• I have read the Contributing Guidelines.
• Every commit is signed off (git commit -s), certifying the Developer Certificate of Origin.
• New or existing tests cover the change.
• Relevant documentation is updated.
• I checked the diff, changed filenames, and commit messages for credentials, private data, internal URLs, internal issue identifiers, and generated artifacts.
• I recorded the validation commands and results above.
• I declared version bumps in the release-intent block above. services/versions.json is written by automation — do not edit it by hand.