feat(proxy): order a stalled candidate behind its peers for one deadline - #144

Refs #6 — the cost half. The stale-workload sweep that mislabels a peer's completed work is a separate mechanism and is not touched here.

A node that accepts a connection and then produces nothing costs the caller a full first-byte deadline, and candidate selection had no memory of it: the next request ranked the same node first and spent the same deadline again. An absent node is cheap by comparison, because its dial fails in a tenth of the time.

The shape

• Triggered by measurement, not by error classification. An attempt that produced no content at all and reached the first-byte budget stamps that candidate. A dial failure is bounded by the much shorter dial timeout, and a status-based retry commits a response, so only the no-content case can reach the budget. This covers both shapes of silence — no headers at all, and headers with no body — without naming either.
• No liveness model. [Bug]: ollama-proxy: undocumented 120 s upstream response-header timeout turns a queued or cold-loading request into a 502 #57 is the case that rules one out: Ollama legitimately sends no headers for longer than this deadline while a model loads, so no single number separates a frozen node from a cold load. The cooldown does not try to. It says only that the last attempt against this candidate cost a full deadline, so prefer someone else for a while.
• One deadline, not a constant of its own. The trigger cost exactly that much, so waiting the same span adds nothing to tune and moves with whatever an operator has set the deadline to. On the current default that is 120s, inside the range you suggested.
• Demotion, never exclusion. A cooling node is still dispatched to once the others are exhausted, and with a single candidate the back is also the front, so a one-node cluster is unaffected.
• Expires on time, and a first byte clears it early. A node that went quiet and is never dispatched to again must not be held at the back forever, which is what would happen if only a success could clear it.

Two hooks, not the one you expected

You said the merged proxies would leave a single insertion point. That is true in the sense you meant — this no longer has to be written twice across separate binaries — but inside the merged proxy two mechanisms decide order, and both have to honour the cooldown or it does not hold:

resolveCandidates partitions the list, but the reserved head is picked by load rather than list position, so a scheduler-ranked node is put straight back at the front of the round it was just demoted from. reserveCandidate therefore skips cooling candidates too, falling back to the ordinary least-loaded pick when every candidate is cooling.

TestReserveCandidate_SkipsCooling fails with only the first hook, so this is a measured claim rather than a preference. The second hook is also what makes the demotion hold when an explicit selection is in play, since that path returns from reserveCandidate before the load pick.

Both read one predicate and both branch on the empty case, so a process that has never met a stalled node does exactly what it did before. TestReserveCandidate_UnchangedWithoutCooling pins that, and the balance invariants in reservation_test.go depend on it.

On the cross-node divergence you raised

Node A demoting B while C finds B healthy is the intended behaviour rather than a wrinkle to design out. The stamp records what that proxy measured on its own path to the peer, which is the only thing it can honestly claim — a direct-connect link, a congested hop or a busy queue genuinely differ per observer. Sharing the judgement would need consensus between proxies, which costs far more than the 2.1x it would be buying.

The divergence is also bounded rather than cumulative: the stamp lapses on time, so the two views reconverge without either proxy learning anything from the other, and nothing is persisted, so a restart starts from no opinion.

One decision worth your call

The demotion runs after all three ordering passes, so it also applies to an explicitly selected node. The selection is "this node first", not "this node only" — the rest of the list is still appended — so a demoted selection is still dispatched to once the healthy candidates are exhausted. TestResolveCandidates_ExplicitSelectionIsAlsoDemoted documents it so the behaviour is visible rather than incidental.

The argument for it: a pinned node that just cost a full deadline is exactly the case where the caller benefits from being routed around. The argument against: a pin is a user instruction, and overriding it silently is a bigger change than the issue asks for. Exempting it is a one-line move of the partition above the selection pass — say which you prefer.

Tests

cooldown_test.go, 10 cases:

• ordering: cooling node last; stable partition keeps relative order within both groups; sole candidate stays; expiry on time; a first byte clears it early; explicit selection also demoted
• reservation: unchanged with nothing cooling; cooling head skipped; all-cooling still reserves
• end to end: TestHandleHTTP_StalledNodeIsNotPaidForTwice asserts the dispatch count, not the candidate order. A reordering that still sent the next request into the same stall would pass an order-only test while costing the caller a second full deadline, so the staller's hit count is the thing checked.

gofmt, go vet and go test -race ./... clean against develop.