Repository navigation
feat(inputlayer-js): routed connection core with request ids, deadlines and reconnect - #237
Merged
Merged
Conversation
…n core Replace the single nextFrame resolver in connection.ts with one routed reader: every request carries an id and its replies (including streamed results) go back to the call that sent it, so calls pipeline on one connection. Pushes route to the notification dispatcher or to a subscription route by name and generation (stale generations are dropped and counted). - in-flight bound (15, one below the engine's) with a client-side queue - deadlines: timeout_ms on every call, then cancel and a transport ping probe; typed DeadlineExceededError, CancelledError (AbortSignal), ProtocolError, RateLimitedError (one resend after the rate window) - pending calls fail with ConnectionLostError(code, mayHaveCommitted) - idle delivery and a keepalive ping so a listening client is never idle-closed; notification iterators buffer instead of dropping - reconnect with jittered backoff re-opens the same graph (?kg=) with the notification cursor (last_seq + epoch); queued calls wait for it; reconnected / session_reset / notification_gap events - one connection per knowledge graph handle, bound with ?kg= and created on first use, instead of .kg use on a shared socket - adversarial frame fixtures replayed through a mock server, and live tests in make js-test-live
…d cursor handling
… queue handle calls directly
… probe and kg creation
…oned slots; re-arm streaming grace
This was referenced Oct 4, 2026
Merged
Closed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Intent
So fix the SDKs by improving them. (Context: the captain asked to rework the SDKs so agents and agentic loops use InputLayer without writing Datalog, and to fix the SDK release blockers by improving the SDKs rather than patching. The agreed design, signed off by an independent challenger after three rounds, is ~/firstmate/data/sdk-design/report.md; this item is its phase-1 PR P1-E P1-E (JS).)
Substance of the referenced item (design section 16, P1-E, applying section 10 "Connection core" to the JavaScript SDK): replace packages/inputlayer-js/src/connection.ts, whose single
nextFrameresolver is the structural reason the JS SDK cannot host subscriptions, with the connection core the design specifies, with adversarial frame fixtures replayed through a mock server in unit tests. The connection core is: one reader that reads every frame; frames with anidgo to the pending call registered under that id and streamed results (result_start/chunk/end) are assembled there and checked against the announced counts; pushes are routed by subscription and generation, a stale generation is dropped and counted; notifications go to the notification dispatcher; closing notices fail every pending call with ConnectionLost(code). One connection per knowledge graph handle, bound at connect time with?kg=<name>and opened lazily, never.kg use/.kg createon a connection that holds subscriptions, because the engine silently removes subscriptions on a switch and without the old execute lock two handles multiplexed on one socket could interleave their switches. Ids on every request; at most ws_max_in_flight_requests - 1 requests in flight (default 15) so a cancel can always be sent; queries are pipelined and replies matched by id. Deadlines and cancel: a timeout sets timeout_ms, a local timer at timeout + grace sends cancel and fails with DeadlineExceeded if no reply arrives; a too_late cancel_ack means the write committed and the reply is awaited. Idle delivery: the reader always runs, so notifications and deltas arrive while no call is in flight, and an application ping is sent every keepalive interval (default 20 s) when nothing else was sent so a listening client never hits the idle timeout. Pending calls fail on close with ConnectionLost, flagged may_have_committed for writes. Reconnect: exponential backoff 1 s to 30 s with jitter, re-authenticate on a connection re-opened with the handle's?kg=, reconnect the notification cursor with last_seq/epoch, emit a notification gap on replay_gap and a session_reset event so code holding session state can recreate it; exhausting max reconnect attempts ends with ConnectionLost. Errors per design 11.1: DeadlineExceeded, Cancelled, ConnectionLost(code, may_have_committed), ProtocolError for invalid_request (an SDK bug, said so), RateLimited after the SDK retries rate_limited once after the window; every exception carries the program sent. Each PR has unit tests on its behaviour and a live test against a real engine that runs in CI; millisecond latency must not regress. Out of scope: the other phase-1 items (the subscribe/watch/on API of P1-G, compile-path fixes of P1-B, Python P1-D, shared packages/conformance of P1-H) and phases 2-4.What Changed
packages/inputlayer-js/src/connection.tsso a single reader routes every frame. Replies are matched to pending calls by request id, and streamed results (result_start/chunk/end) are assembled and checked against their announced counts. Subscription pushes are routed by subscription and generation, and stale generations are dropped and counted. Notifications go to the dispatcher, and closing notices fail pending calls. Requests are pipelined, with in-flight requests capped one below the server limit so acancelcan always be sent. Each call can taketimeoutMsand anAbortSignal, and a local timer at timeout plus grace sendscancel. When the connection is idle, app pings keep it open, and transport pings detect half-open sockets. Reconnect uses exponential backoff with jitter, re-authenticates, replays notifications from the cursor, and emitssession_resetandnotification_gapevents.?kg=<name>, instead of switching graphs with.kg useon a shared socket.client.ts,knowledge-graph.tsandnotifications.tswere updated to match. AddedConnectionLostError(code,mayHaveCommitted),DeadlineExceededError,CancelledError,ProtocolErrorandRateLimitedError. Every call error now carries the program it sent asiql. The protocol types gaincancel,cancel_ack,timeout_msandcredential_expired.rate_limitedis retried once after the window before failing.tests/fixtures/connection/*.json), plus a liveconnection.integration.test.ts, which is now part ofmake js-test-live. Updated the JS SDK guide (and the regenerated GUI docs bundle) to cover the new options, deadlines and cancel, reconnect behaviour, per-handle connections and the new errors.Risk Assessment
Testing
I built the engine and ran isolated local servers, one with default settings, one with a 3 s idle timeout and one with the rate limit raised. Against them I ran the existing live connection-core tests and a throwaway script of seven scenarios that drove the head SDK. A TCP proxy simulated a peer that stops answering, a write whose reply never arrives, and refused reconnects. Each scenario passed. A long query in flight past several keepalive intervals was not dropped, which is the round-6 K1 fix. An idle listener stayed connected past the server's idle timeout and still got notifications; with keepalive off, the server closed it with idle_timeout. A half-open connection was detected in about 1 s, then reconnected with session_reset to the same graph. A write whose reply was lost failed with ConnectionLost and mayHaveCommitted=true, and it had in fact committed on the server. After the maximum reconnect attempts, the client gave up with ConnectionLost. Deadlines failed as DeadlineExceeded, the error carried the program, and the connection stayed usable. Rate limiting was retried once, then surfaced as RateLimitedError. Sequential query latency matched the base SDK at p50 about 0.48–0.52 ms. The mock-server fixture suite also passed. Not driven live: a ProtocolError for invalid_request, because the public API cannot send a malformed frame; the mock fixtures cover it. There is no UI surface, so there are no screenshots. Afterwards I stopped the servers and removed the throwaway test files, the temp data dirs and target/.
Evidence: Live connection-core integration tests against the real engine
Evidence: Live scenarios S1-S7 transcript (keepalive, half-open, mayHaveCommitted, reconnect exhaustion, deadline, rate limit)
Evidence: Live scenario driver source
Evidence: Latency base vs head
base p50=0.520/0.476ms p99=0.734/0.605ms; head p50=0.499/0.477ms p99=0.633/0.597msEvidence: Adversarial mock-server fixture tests
Pipeline
Updates from git push no-mistakes
... (7 earlier update rounds omitted to keep the PR body within GitHub's 65536-char limit; full history is in the run log.)
🔧 **Review** - 7 issues found → auto-fixed (7) ✅
🔧 Fix applied.
6 issues (5 warnings, 1 info) still open:
packages/inputlayer-js/src/connection.ts:599-probeServer()assumes that "a live server answers at once, even while computing". That is false when the engine's request pipeline is full. The engine reads client frames, including WebSocket pings, only whilerequests.has_capacity()(ws.rs:443, pipeline.rs:99), and an admittedcanceltakes a pipeline slot until itscancel_ackis released (ws.rs:606-616). With the default bounds (client maxInFlight 15, server 16), 15 calls in flight plus one deadlinecancelfill all 16 slots, so the probe's ping is not read and gets no pong. After timeoutGraceMs/2 (1 s) the client terminates a healthy connection. Example:Promise.allof a write that is slow to commit plus 14 queries pipelined behind it (replies come back in request order). At the write's deadline the client sends cancel (too_late) and the probe. The ping goes unread, and after 1 s every call fails with ConnectionLostError: the write with mayHaveCommitted, all the queries needlessly. The connection then reconnects and emits session_reset, so session state is lost. This probe code is from the original commit, and round 1 (the F2 fix) kept it. Fix: don't terminate on a missing pong while the server's window may be full (inFlight.size plus outstanding cancels >= maxInFlight), or count any frame received since the probe started as proof the server is alive. The overdue timer already bounds the call.packages/inputlayer-js/src/connection.ts:556-overdue()fails the call at deadline + grace even when its reply has already started arriving (call.streamis set by result_start). Example: a query withtimeoutMs: 30000finishes computing at 29 s and starts streaming a large result over a slow link. At 32 s the call rejects with DeadlineExceededError, which claims no reply came, and the rest of a successful result is thrown away. Each later result_chunk and the result_end is then counted separately instaleReplies, because the id is in neitherinFlightnorabandoned. The operator's F2 decision limits local failure to a call where "neither the call's reply nor its cancel_ack has arrived", and this path came in with the round-1 F2 fix. Fix: inoverdue, ifcall.streamis set, keep waiting for result_end (or re-arm the grace timer). When a call is failed locally, add its id toabandonedso its remaining frames are dropped as one stale reply.packages/inputlayer-js/src/connection.ts:731-finish()removes a call frominFlightas soon as the client fails it locally, and thenpump()sends more requests, but the engine still holds the request until it replies. Replies come back in request order, so this is reachable: queries pipelined behind a slow write go overdue together and free their slots at once. The client then sends another full window, and the engine's unreleased count goes pastws_max_in_flight_requests. The engine then stops reading, so a latercancel(from an AbortSignal) or probe ping is not read until the head of the line finishes. That breaks the intent's invariant: "at most ws_max_in_flight_requests - 1 requests in flight ... so a cancel can always be sent". The round-1 F2 fix introduced this throughoverdue()(line 561);abandon()(line 742) has the same early release for broken streams. Fix: keep a locally failed id counted againstmaxInFlight(for example inabandoned, checked bypump) until its final frame (result, error or result_end) arrives or the connection closes.packages/inputlayer-js/src/connection.ts:976- After an AuthenticationError the reconnect loop stops at once (line 955), yet the give-up error always saysReconnecting failed after ${maxReconnectAttempts} attempt(s). Example: a credential revoked during an outage gives "failed after 10 attempt(s)" after one attempt. Fix: report the actual attempt count.packages/inputlayer-js/src/connection.ts:577- Introduced by the round-4 N1/N3 fixes, which followed the operator's instructions to the letter but together leave no way to get a connection back. A locally failed call's slot now stays inabandoneduntil its final frame arrives or the socket closes (line 577; and line 770 for broken streams).pumpcounts those slots (line 534). The probe no longer terminates whileinFlight + abandoned + control >= maxInFlight(line 625). The keepalive ping's failure is ignored (lines 1026-1029: 'A failed ping surfaces as a close', which never happens on a half-open socket).Two paths leave the connection open but unable to send.
(a) The F2 case the operator required to stay usable: a live server that answers pings but never replies to a request (a lost reply). Each such call holds a slot for good. After
maxInFlight(15) of them,pumpnever sends again. Every later call fails with DeadlineExceededError 'nothing was applied', or hangs forever withtimeoutMs: 0. Pongs keep the socket alive, so it never reconnects. The silent_server tests only cover one slot.(b) A half-open connection (host crash or partition) while 15 calls are in flight: at each deadline, outstanding >= maxInFlight, so the probe never terminates. All 15 go to
abandoned, and the same wedge follows until the OS times out the TCP connection, which can take tens of minutes.Fixing this means choosing a new liveness rule, which needs a decision. For example: drop and reconnect the connection when
abandonedalone fills the window, or when an abandoned slot outlives a bound. Or treat a keepalive ping that times out as a dead connection. Without one of these, the N3 slot accounting turns a single lost reply per slot into a permanently dead handle.packages/inputlayer-js/src/connection.ts:574- Introduced by the round-4 N2 fix. When the grace period ends while a reply is streaming,overduereturns withcall.deadlinecleared and does not re-arm any timer, so the call has no bound left. If the connection goes half-open mid-stream, nothing ever notices:onFramecallsclearProbe);inFlight.size > 0(line 1026), so the client sends nothing and TCP never detects the dead peer.Example: a query with
timeoutMs: 30000starts streaming at 29 s, and the network partitions at 31 s. The call stays pending forever and holds its slot. Before round 4 it failed at deadline + grace. Fix, using the alternative the operator's N2 instruction allows ('or re-arm the grace timer'): inoverdue, ifcall.streamis set, re-armcall.deadline = setTimeout(() => this.overdue(call), timeoutGraceMs)and reset it on each result_chunk. A stream that keeps arriving completes; a stalled one fails locally like a silent call.🔧 Fix applied.
8 issues (1 error, 5 warnings, 2 infos) still open:
packages/inputlayer-js/src/connection.ts:599-probeServer()assumes that "a live server answers at once, even while computing". That is false when the engine's request pipeline is full. The engine reads client frames, including WebSocket pings, only whilerequests.has_capacity()(ws.rs:443, pipeline.rs:99), and an admittedcanceltakes a pipeline slot until itscancel_ackis released (ws.rs:606-616). With the default bounds (client maxInFlight 15, server 16), 15 calls in flight plus one deadlinecancelfill all 16 slots, so the probe's ping is not read and gets no pong. After timeoutGraceMs/2 (1 s) the client terminates a healthy connection. Example:Promise.allof a write that is slow to commit plus 14 queries pipelined behind it (replies come back in request order). At the write's deadline the client sends cancel (too_late) and the probe. The ping goes unread, and after 1 s every call fails with ConnectionLostError: the write with mayHaveCommitted, all the queries needlessly. The connection then reconnects and emits session_reset, so session state is lost. This probe code is from the original commit, and round 1 (the F2 fix) kept it. Fix: don't terminate on a missing pong while the server's window may be full (inFlight.size plus outstanding cancels >= maxInFlight), or count any frame received since the probe started as proof the server is alive. The overdue timer already bounds the call.packages/inputlayer-js/src/connection.ts:556-overdue()fails the call at deadline + grace even when its reply has already started arriving (call.streamis set by result_start). Example: a query withtimeoutMs: 30000finishes computing at 29 s and starts streaming a large result over a slow link. At 32 s the call rejects with DeadlineExceededError, which claims no reply came, and the rest of a successful result is thrown away. Each later result_chunk and the result_end is then counted separately instaleReplies, because the id is in neitherinFlightnorabandoned. The operator's F2 decision limits local failure to a call where "neither the call's reply nor its cancel_ack has arrived", and this path came in with the round-1 F2 fix. Fix: inoverdue, ifcall.streamis set, keep waiting for result_end (or re-arm the grace timer). When a call is failed locally, add its id toabandonedso its remaining frames are dropped as one stale reply.packages/inputlayer-js/src/connection.ts:731-finish()removes a call frominFlightas soon as the client fails it locally, and thenpump()sends more requests, but the engine still holds the request until it replies. Replies come back in request order, so this is reachable: queries pipelined behind a slow write go overdue together and free their slots at once. The client then sends another full window, and the engine's unreleased count goes pastws_max_in_flight_requests. The engine then stops reading, so a latercancel(from an AbortSignal) or probe ping is not read until the head of the line finishes. That breaks the intent's invariant: "at most ws_max_in_flight_requests - 1 requests in flight ... so a cancel can always be sent". The round-1 F2 fix introduced this throughoverdue()(line 561);abandon()(line 742) has the same early release for broken streams. Fix: keep a locally failed id counted againstmaxInFlight(for example inabandoned, checked bypump) until its final frame (result, error or result_end) arrives or the connection closes.packages/inputlayer-js/src/connection.ts:976- After an AuthenticationError the reconnect loop stops at once (line 955), yet the give-up error always saysReconnecting failed after ${maxReconnectAttempts} attempt(s). Example: a credential revoked during an outage gives "failed after 10 attempt(s)" after one attempt. Fix: report the actual attempt count.packages/inputlayer-js/src/connection.ts:577- Introduced by the round-4 N1/N3 fixes, which followed the operator's instructions to the letter but together leave no way to get a connection back. A locally failed call's slot now stays inabandoneduntil its final frame arrives or the socket closes (line 577; and line 770 for broken streams).pumpcounts those slots (line 534). The probe no longer terminates whileinFlight + abandoned + control >= maxInFlight(line 625). The keepalive ping's failure is ignored (lines 1026-1029: 'A failed ping surfaces as a close', which never happens on a half-open socket).Two paths leave the connection open but unable to send.
(a) The F2 case the operator required to stay usable: a live server that answers pings but never replies to a request (a lost reply). Each such call holds a slot for good. After
maxInFlight(15) of them,pumpnever sends again. Every later call fails with DeadlineExceededError 'nothing was applied', or hangs forever withtimeoutMs: 0. Pongs keep the socket alive, so it never reconnects. The silent_server tests only cover one slot.(b) A half-open connection (host crash or partition) while 15 calls are in flight: at each deadline, outstanding >= maxInFlight, so the probe never terminates. All 15 go to
abandoned, and the same wedge follows until the OS times out the TCP connection, which can take tens of minutes.Fixing this means choosing a new liveness rule, which needs a decision. For example: drop and reconnect the connection when
abandonedalone fills the window, or when an abandoned slot outlives a bound. Or treat a keepalive ping that times out as a dead connection. Without one of these, the N3 slot accounting turns a single lost reply per slot into a permanently dead handle.packages/inputlayer-js/src/connection.ts:574- Introduced by the round-4 N2 fix. When the grace period ends while a reply is streaming,overduereturns withcall.deadlinecleared and does not re-arm any timer, so the call has no bound left. If the connection goes half-open mid-stream, nothing ever notices:onFramecallsclearProbe);inFlight.size > 0(line 1026), so the client sends nothing and TCP never detects the dead peer.Example: a query with
timeoutMs: 30000starts streaming at 29 s, and the network partitions at 31 s. The call stays pending forever and holds its slot. Before round 4 it failed at deadline + grace. Fix, using the alternative the operator's N2 instruction allows ('or re-arm the grace timer'): inoverdue, ifcall.streamis set, re-armcall.deadline = setTimeout(() => this.overdue(call), timeoutGraceMs)and reset it on each result_chunk. A stream that keeps arriving completes; a stalled one fails locally like a silent call.packages/inputlayer-js/src/connection.ts:1043- The round-5 L1 fix brought this in. The keepalive now sends an application{type:'ping', id}while calls are in flight and drops the socket when nopongarrives withintimeoutGraceMs. The engine, however, answers an application ping as an ordinary pipelined request (request.rs:100Request::immediate, "released in order with the others"; ws.rs:137 "Replies come back in request order"). Its pong therefore waits behind every earlier request. Concrete sequence with the defaults (keepaliveMs 20 000, grace 2 000, defaultTimeoutMs 30 000):kg.execute(q)starts a query that takes 25 s on a healthy server. Nothing else is sent, so the keepalive tick at about 20 s sends a ping, and that pong cannot be released until the query's reply at 25 s. At about 22 srequest()times out andws.terminate()runs. The healthy query fails with ConnectionLostError, a write fails with mayHaveCommitted, and the reconnect emits session_reset. WithtimeoutMs: 0(no deadline), any call that runs longer than keepaliveMs + grace is killed the same way. The same false positive occurs with inFlight 0 when the engine still holds an abandoned call: the round-5 change no longer counts abandoned calls, but the engine still orders their replies first. Sibling: the ping also takes an engine pipeline slot, so 15 calls in flight plus the keepalive ping fill the engine's 16 slots. The engine then stops reading, and a deadline or signal cancel can no longer be read, which breaks the intent's 'ws_max_in_flight_requests - 1 so a cancel can always be sent'. The tests cannot catch this because the mock server (connection.test.ts:87) answers application pings at once, out of order. Fix: check liveness with a transportws.ping()and itspong(asprobeServerdoes; that pong does not wait behind the request order and takes no pipeline slot). Keep the application ping only for idle keep-alive, with no terminate-on-timeout, or count any frame received as proof of life. Add a mock fixture in which a ping's pong waits behind a slow reply.packages/inputlayer-js/src/client.ts:55- ThekeepaliveMsdocs in client.ts:55 and docs/content/docs/guides/js-sdk.mdx:66 still describe only idle keep-alive. They don't say that the round-5 keepalive now also drops and reconnects a connection whose ping goes unanswered withintimeoutGraceMs, though connection.ts:88-93 was updated to say so. Align the wording.🔧 Fix applied.
8 issues (1 error, 5 warnings, 2 infos) still open:
packages/inputlayer-js/src/connection.ts:599-probeServer()assumes that "a live server answers at once, even while computing". That is false when the engine's request pipeline is full. The engine reads client frames, including WebSocket pings, only whilerequests.has_capacity()(ws.rs:443, pipeline.rs:99), and an admittedcanceltakes a pipeline slot until itscancel_ackis released (ws.rs:606-616). With the default bounds (client maxInFlight 15, server 16), 15 calls in flight plus one deadlinecancelfill all 16 slots, so the probe's ping is not read and gets no pong. After timeoutGraceMs/2 (1 s) the client terminates a healthy connection. Example:Promise.allof a write that is slow to commit plus 14 queries pipelined behind it (replies come back in request order). At the write's deadline the client sends cancel (too_late) and the probe. The ping goes unread, and after 1 s every call fails with ConnectionLostError: the write with mayHaveCommitted, all the queries needlessly. The connection then reconnects and emits session_reset, so session state is lost. This probe code is from the original commit, and round 1 (the F2 fix) kept it. Fix: don't terminate on a missing pong while the server's window may be full (inFlight.size plus outstanding cancels >= maxInFlight), or count any frame received since the probe started as proof the server is alive. The overdue timer already bounds the call.packages/inputlayer-js/src/connection.ts:556-overdue()fails the call at deadline + grace even when its reply has already started arriving (call.streamis set by result_start). Example: a query withtimeoutMs: 30000finishes computing at 29 s and starts streaming a large result over a slow link. At 32 s the call rejects with DeadlineExceededError, which claims no reply came, and the rest of a successful result is thrown away. Each later result_chunk and the result_end is then counted separately instaleReplies, because the id is in neitherinFlightnorabandoned. The operator's F2 decision limits local failure to a call where "neither the call's reply nor its cancel_ack has arrived", and this path came in with the round-1 F2 fix. Fix: inoverdue, ifcall.streamis set, keep waiting for result_end (or re-arm the grace timer). When a call is failed locally, add its id toabandonedso its remaining frames are dropped as one stale reply.packages/inputlayer-js/src/connection.ts:731-finish()removes a call frominFlightas soon as the client fails it locally, and thenpump()sends more requests, but the engine still holds the request until it replies. Replies come back in request order, so this is reachable: queries pipelined behind a slow write go overdue together and free their slots at once. The client then sends another full window, and the engine's unreleased count goes pastws_max_in_flight_requests. The engine then stops reading, so a latercancel(from an AbortSignal) or probe ping is not read until the head of the line finishes. That breaks the intent's invariant: "at most ws_max_in_flight_requests - 1 requests in flight ... so a cancel can always be sent". The round-1 F2 fix introduced this throughoverdue()(line 561);abandon()(line 742) has the same early release for broken streams. Fix: keep a locally failed id counted againstmaxInFlight(for example inabandoned, checked bypump) until its final frame (result, error or result_end) arrives or the connection closes.packages/inputlayer-js/src/connection.ts:976- After an AuthenticationError the reconnect loop stops at once (line 955), yet the give-up error always saysReconnecting failed after ${maxReconnectAttempts} attempt(s). Example: a credential revoked during an outage gives "failed after 10 attempt(s)" after one attempt. Fix: report the actual attempt count.packages/inputlayer-js/src/connection.ts:577- Introduced by the round-4 N1/N3 fixes, which followed the operator's instructions to the letter but together leave no way to get a connection back. A locally failed call's slot now stays inabandoneduntil its final frame arrives or the socket closes (line 577; and line 770 for broken streams).pumpcounts those slots (line 534). The probe no longer terminates whileinFlight + abandoned + control >= maxInFlight(line 625). The keepalive ping's failure is ignored (lines 1026-1029: 'A failed ping surfaces as a close', which never happens on a half-open socket).Two paths leave the connection open but unable to send.
(a) The F2 case the operator required to stay usable: a live server that answers pings but never replies to a request (a lost reply). Each such call holds a slot for good. After
maxInFlight(15) of them,pumpnever sends again. Every later call fails with DeadlineExceededError 'nothing was applied', or hangs forever withtimeoutMs: 0. Pongs keep the socket alive, so it never reconnects. The silent_server tests only cover one slot.(b) A half-open connection (host crash or partition) while 15 calls are in flight: at each deadline, outstanding >= maxInFlight, so the probe never terminates. All 15 go to
abandoned, and the same wedge follows until the OS times out the TCP connection, which can take tens of minutes.Fixing this means choosing a new liveness rule, which needs a decision. For example: drop and reconnect the connection when
abandonedalone fills the window, or when an abandoned slot outlives a bound. Or treat a keepalive ping that times out as a dead connection. Without one of these, the N3 slot accounting turns a single lost reply per slot into a permanently dead handle.packages/inputlayer-js/src/connection.ts:574- Introduced by the round-4 N2 fix. When the grace period ends while a reply is streaming,overduereturns withcall.deadlinecleared and does not re-arm any timer, so the call has no bound left. If the connection goes half-open mid-stream, nothing ever notices:onFramecallsclearProbe);inFlight.size > 0(line 1026), so the client sends nothing and TCP never detects the dead peer.Example: a query with
timeoutMs: 30000starts streaming at 29 s, and the network partitions at 31 s. The call stays pending forever and holds its slot. Before round 4 it failed at deadline + grace. Fix, using the alternative the operator's N2 instruction allows ('or re-arm the grace timer'): inoverdue, ifcall.streamis set, re-armcall.deadline = setTimeout(() => this.overdue(call), timeoutGraceMs)and reset it on each result_chunk. A stream that keeps arriving completes; a stalled one fails locally like a silent call.packages/inputlayer-js/src/connection.ts:1043- The round-5 L1 fix brought this in. The keepalive now sends an application{type:'ping', id}while calls are in flight and drops the socket when nopongarrives withintimeoutGraceMs. The engine, however, answers an application ping as an ordinary pipelined request (request.rs:100Request::immediate, "released in order with the others"; ws.rs:137 "Replies come back in request order"). Its pong therefore waits behind every earlier request. Concrete sequence with the defaults (keepaliveMs 20 000, grace 2 000, defaultTimeoutMs 30 000):kg.execute(q)starts a query that takes 25 s on a healthy server. Nothing else is sent, so the keepalive tick at about 20 s sends a ping, and that pong cannot be released until the query's reply at 25 s. At about 22 srequest()times out andws.terminate()runs. The healthy query fails with ConnectionLostError, a write fails with mayHaveCommitted, and the reconnect emits session_reset. WithtimeoutMs: 0(no deadline), any call that runs longer than keepaliveMs + grace is killed the same way. The same false positive occurs with inFlight 0 when the engine still holds an abandoned call: the round-5 change no longer counts abandoned calls, but the engine still orders their replies first. Sibling: the ping also takes an engine pipeline slot, so 15 calls in flight plus the keepalive ping fill the engine's 16 slots. The engine then stops reading, and a deadline or signal cancel can no longer be read, which breaks the intent's 'ws_max_in_flight_requests - 1 so a cancel can always be sent'. The tests cannot catch this because the mock server (connection.test.ts:87) answers application pings at once, out of order. Fix: check liveness with a transportws.ping()and itspong(asprobeServerdoes; that pong does not wait behind the request order and takes no pipeline slot). Keep the application ping only for idle keep-alive, with no terminate-on-timeout, or count any frame received as proof of life. Add a mock fixture in which a ping's pong waits behind a slow reply.packages/inputlayer-js/src/connection.ts:639- The round-6 K1/K2 doc updates (client.ts:55, connection.ts:92-94 and the header at connection.ts:24-28) say the connection is dropped when there is "no WebSocket pong and no frame withintimeoutGraceMs".probeServeractually terminates afterMath.floor(timeoutGraceMs / 2)(line 639). It also never terminates whileinFlight.size + control.size >= maxInFlight, which the option docs leave out. The probe now runs on every keepalive tick rather than only after a deadline, so this window decides when an idle listener counts as dead. With the defaults it gives up after 1 s, where the docs say 2 s. Fix: make the docs and the code agree. Either state the half-grace window, or use the fulltimeoutGraceMsas the operator's K1 wording describes ("no pong and no frame within timeoutGraceMs").🔧 Fix applied.
1 info still open:
gui/lib/docs-bundle.ts:1482- The committed GUI docs bundle is out of date. js-sdk.mdx was edited in later rounds, including this round's keepaliveMs comment at js-sdk.mdx:66-67, but the bundledguides/js-sdkcontent still has the older textkeepaliveMs: 20_000, // ping a silent connection so it is never idle-closed (default: 20 000). Builds regenerate the bundle (gui/package.json prebuild), so the shipped GUI is not affected. The committed file still disagrees with its source, though. Fix: rerunnode gui/scripts/bundle-docs.mjsand commit the result.🔧 Fix applied.
✅ Re-checked - no issues remain.
✅ **Test** - passed
✅ No issues found.
npx vitest run tests/connection.test.ts, not against the real engine. A real inputlayer-s…npx vitest run tests/connection.test.ts tests/notifications.test.ts tests/statement-errors.test.ts(mock-server frame fixtures, 83 passed)make js-test-live(builds the release server and runs docs-queries plus connection live tests against it: 32 passed)tests/connection.integration.test.tsrepeated 10 times against a live server, checking the deadline/cancel race (R1) for flakiness: 10/10 passedCustom live driverzz-nm-live-driver.integration.test.ts, which spawns disposable servers: keepalive vs a 3 s idle timeout, server restart, SIGKILL mid-call, server gone for good, rate limit at 5 msg/s, R2 deadline during reconnect, 30 pipelined calls with one abortLive latency comparison of the base SDK (ea9d5d3c src) against the new SDK on the same server: 2000 sequential small queries x3 rounds✅ No issues found.
cargo build --release --bin inputlayer-server, then isolated engines on 127.0.0.1:8197 (defaults), :8198 (INPUTLAYER_HTTP__WS_IDLE_TIMEOUT_MS=3000) and :8199 (rate limit raised), each with its own temp data dirINPUTLAYER_TEST_SERVER=ws://127.0.0.1:8197/ws npx vitest run tests/connection.integration.test.ts(7 live connection-core tests)Throwaway live scenario script (S1-S7) run against the real engines, with a controllable TCP proxy to simulate half-open, blackholed-reply and refused connectionsThrowaway latency benchmark: base SDK (1ee8c8f8 src) vs head SDK, 3000 sequential queries x2 rounds against the real enginenpx vitest run tests/connection.test.ts(adversarial frame fixtures replayed through the mock server, 54 tests)✅ **Document** - passed
✅ No issues found.
✅ No issues found.
✅ **Lint** - passed
✅ No issues found.
✅ No issues found.
✅ **Push** - passed
✅ No issues found.
✅ No issues found.