Skip to content

feat(inputlayer-js): routed connection core with request ids, deadlines and reconnect - #237

Merged
jsam merged 9 commits into
mainfrom
fm/sdk-p1-e
Oct 4, 2026
Merged

jsam merged 9 commits into
mainfrom
fm/sdk-p1-e

Conversation

@jsam

@jsam jsam commented Oct 4, 2026 •

Copy link
Copy Markdown
Contributor

Intent

So fix the SDKs by improving them. (Context: the captain asked to rework the SDKs so agents and agentic loops use InputLayer without writing Datalog, and to fix the SDK release blockers by improving the SDKs rather than patching. The agreed design, signed off by an independent challenger after three rounds, is ~/firstmate/data/sdk-design/report.md; this item is its phase-1 PR P1-E P1-E (JS).)

Substance of the referenced item (design section 16, P1-E, applying section 10 "Connection core" to the JavaScript SDK): replace packages/inputlayer-js/src/connection.ts, whose single nextFrame resolver is the structural reason the JS SDK cannot host subscriptions, with the connection core the design specifies, with adversarial frame fixtures replayed through a mock server in unit tests. The connection core is: one reader that reads every frame; frames with an id go to the pending call registered under that id and streamed results (result_start/chunk/end) are assembled there and checked against the announced counts; pushes are routed by subscription and generation, a stale generation is dropped and counted; notifications go to the notification dispatcher; closing notices fail every pending call with ConnectionLost(code). One connection per knowledge graph handle, bound at connect time with ?kg=<name> and opened lazily, never .kg use/.kg create on a connection that holds subscriptions, because the engine silently removes subscriptions on a switch and without the old execute lock two handles multiplexed on one socket could interleave their switches. Ids on every request; at most ws_max_in_flight_requests - 1 requests in flight (default 15) so a cancel can always be sent; queries are pipelined and replies matched by id. Deadlines and cancel: a timeout sets timeout_ms, a local timer at timeout + grace sends cancel and fails with DeadlineExceeded if no reply arrives; a too_late cancel_ack means the write committed and the reply is awaited. Idle delivery: the reader always runs, so notifications and deltas arrive while no call is in flight, and an application ping is sent every keepalive interval (default 20 s) when nothing else was sent so a listening client never hits the idle timeout. Pending calls fail on close with ConnectionLost, flagged may_have_committed for writes. Reconnect: exponential backoff 1 s to 30 s with jitter, re-authenticate on a connection re-opened with the handle's ?kg=, reconnect the notification cursor with last_seq/epoch, emit a notification gap on replay_gap and a session_reset event so code holding session state can recreate it; exhausting max reconnect attempts ends with ConnectionLost. Errors per design 11.1: DeadlineExceeded, Cancelled, ConnectionLost(code, may_have_committed), ProtocolError for invalid_request (an SDK bug, said so), RateLimited after the SDK retries rate_limited once after the window; every exception carries the program sent. Each PR has unit tests on its behaviour and a live test against a real engine that runs in CI; millisecond latency must not regress. Out of scope: the other phase-1 items (the subscribe/watch/on API of P1-G, compile-path fixes of P1-B, Python P1-D, shared packages/conformance of P1-H) and phases 2-4.

What Changed

  • Rewrote packages/inputlayer-js/src/connection.ts so a single reader routes every frame. Replies are matched to pending calls by request id, and streamed results (result_start/chunk/end) are assembled and checked against their announced counts. Subscription pushes are routed by subscription and generation, and stale generations are dropped and counted. Notifications go to the dispatcher, and closing notices fail pending calls. Requests are pipelined, with in-flight requests capped one below the server limit so a cancel can always be sent. Each call can take timeoutMs and an AbortSignal, and a local timer at timeout plus grace sends cancel. When the connection is idle, app pings keep it open, and transport pings detect half-open sockets. Reconnect uses exponential backoff with jitter, re-authenticates, replays notifications from the cursor, and emits session_reset and notification_gap events.
  • Each knowledge graph handle now gets its own lazily opened connection, bound with ?kg=<name>, instead of switching graphs with .kg use on a shared socket. client.ts, knowledge-graph.ts and notifications.ts were updated to match. Added ConnectionLostError (code, mayHaveCommitted), DeadlineExceededError, CancelledError, ProtocolError and RateLimitedError. Every call error now carries the program it sent as iql. The protocol types gain cancel, cancel_ack, timeout_ms and credential_expired. rate_limited is retried once after the window before failing.
  • Added unit tests that replay adversarial frame fixtures through a mock server (tests/fixtures/connection/*.json), plus a live connection.integration.test.ts, which is now part of make js-test-live. Updated the JS SDK guide (and the regenerated GUI docs bundle) to cover the new options, deadlines and cancel, reconnect behaviour, per-handle connections and the new errors.

Risk Assessment

⚠️ Medium: The latest fix round only regenerated the GUI docs bundle, and rerunning the generator gives the same file. The overall change still rewrites the JS connection core (deadlines, probe, reconnect), which is substantial concurrency logic and deserves human attention even with no open defects.

Testing

I built the engine and ran isolated local servers, one with default settings, one with a 3 s idle timeout and one with the rate limit raised. Against them I ran the existing live connection-core tests and a throwaway script of seven scenarios that drove the head SDK. A TCP proxy simulated a peer that stops answering, a write whose reply never arrives, and refused reconnects. Each scenario passed. A long query in flight past several keepalive intervals was not dropped, which is the round-6 K1 fix. An idle listener stayed connected past the server's idle timeout and still got notifications; with keepalive off, the server closed it with idle_timeout. A half-open connection was detected in about 1 s, then reconnected with session_reset to the same graph. A write whose reply was lost failed with ConnectionLost and mayHaveCommitted=true, and it had in fact committed on the server. After the maximum reconnect attempts, the client gave up with ConnectionLost. Deadlines failed as DeadlineExceeded, the error carried the program, and the connection stayed usable. Rate limiting was retried once, then surfaced as RateLimitedError. Sequential query latency matched the base SDK at p50 about 0.48–0.52 ms. The mock-server fixture suite also passed. Not driven live: a ProtocolError for invalid_request, because the public API cannot send a malformed frame; the mock fixtures cover it. There is no UI surface, so there are no screenshots. Afterwards I stopped the servers and removed the throwaway test files, the temp data dirs and target/.

  • Live validation: ✅ go - 9 of 9 scenarios driven live against the product
Scenario Result Live Evidence
40 pipelined queries on one handle each get their own reply; two handles on different graphs never cross ✅ pass live live-connection-integration.txt (pipelined calls, two handles interleaved)
A deadline fails with DeadlineExceeded carrying the program sent, and the connection stays usable; an aborted call is cancelled on the server ✅ pass live live-scenarios.txt S6 (DeadlineExceededError after 101 ms, iql set); live-connection-integration.txt (abort test)
A ~2 s query in flight across several 300 ms keepalive intervals completes without a drop or reconnect ✅ pass live live-scenarios.txt S1: 2048 ms, 1599 rows, no events
An idle listener with keepalive outlives the server's 3 s idle timeout and still receives notifications; with keepalive off it is idle-closed and reconnects ✅ pass live live-scenarios.txt S2
A half-open connection (proxy freezes traffic) is detected by the keepalive probe, reconnected to the same ?kg= graph, and emits session_reset and notification_gap ✅ pass live live-scenarios.txt S3: detected in 989 ms; query after reconnect returned [[5,6]]
Write whose reply is lost fails with ConnectionLost(mayHaveCommitted=true), and the write is present on the server; a read fails with mayHaveCommitted=false ✅ pass live live-scenarios.txt S4
Reconnect replays notifications missed during an outage; when reconnects are refused, the client gives up after maxReconnectAttempts with ConnectionLost ✅ pass live live-connection-integration.txt (reconnect replay); live-scenarios.txt S5 ('Reconnecting failed after 3 attempt(s)')
Bursting past the server rate limit: rate_limited is retried once, the remainder surface as RateLimitedError, and the connection stays usable ✅ pass live live-scenarios.txt S7: {ok:199, RateLimitedError:101}, no disconnect events
Millisecond query latency does not regress versus the base SDK ✅ pass live latency-base-vs-head.txt
Evidence: Live connection-core integration tests against the real engine

 RUN  v3.2.7 ~/.no-mistakes/worktrees/9eea230c3aae/01M43BFB8M18GTF3V2STQB1V68/packages/inputlayer-js

 ✓ tests/connection.integration.test.ts > Live: connection core > pipelined calls on one handle each get their own reply 8ms
 ✓ tests/connection.integration.test.ts > Live: connection core > a deadline fails typed and leaves the connection usable 52ms
 ✓ tests/connection.integration.test.ts > Live: connection core > aborting a running call cancels it on the server 32ms
 ✓ tests/connection.integration.test.ts > Live: connection core > two handles on different graphs never cross, even interleaved 67ms
 ✓ tests/connection.integration.test.ts > Live: connection core > a graph a handle created can be dropped while its handle is open 76ms
 ✓ tests/connection.integration.test.ts > Live: connection core > delivers notifications to a client with no call in flight 91ms
 ✓ tests/connection.integration.test.ts > Live: connection core > reconnects to the same graph and replays notifications missed meanwhile 166ms

 Test Files  1 passed (1)
      Tests  7 passed (7)
   Start at  12:23:59
   Duration  1.09s (transform 160ms, setup 0ms, collect 211ms, tests 623ms, environment 0ms, prepare 75ms)
Evidence: Live scenarios S1-S7 transcript (keepalive, half-open, mayHaveCommitted, reconnect exhaustion, deadline, rate limit)

 RUN  v3.2.7 ~/.no-mistakes/worktrees/9eea230c3aae/01M43BFB8M18GTF3V2STQB1V68/packages/inputlayer-js

stdout | tests/zz_nm_probe.test.ts > S1 long query in flight past several keepalive intervals is not dropped
[scenario] S1 long query took 2048 ms; rows 1599 ; events []

 ✓ tests/zz_nm_probe.test.ts > S1 long query in flight past several keepalive intervals is not dropped 2068ms
stdout | tests/zz_nm_probe.test.ts > S2 idle listener with keepalive survives server idle timeout and still receives notifications
[scenario] S2 keepalive on: events [] notifications 1

stdout | tests/zz_nm_probe.test.ts > S2 idle listener with keepalive survives server idle timeout and still receives notifications
[scenario] S2 control keepalive off: events [
  'disconnected:{"code":"idle_timeout"}',
  'disconnected:{"code":"idle_timeout","knowledgeGraph":"nm_idle"}',
  'reconnected:{"attempt":1}',
  'session_reset:{}',
  'notification_gap:{"code":"no_cursor","message":"Reconnected with no notification cursor: notifications sent while disconnected were not replayed"}',
  'reconnected:{"attempt":1,"knowledgeGraph":"nm_idle"}',
  'session_reset:{"knowledgeGraph":"nm_idle"}',
  'notification_gap:{"code":"no_cursor","message":"Reconnected with no notification cursor: notifications sent while disconnected were not replayed","knowledgeGraph":"nm_idle"}'
]

 ✓ tests/zz_nm_probe.test.ts > S2 idle listener with keepalive survives server idle timeout and still receives notifications 13195ms
stdout | tests/zz_nm_probe.test.ts > S3 half-open connection (peer stops answering) is detected by keepalive and reconnected with session_reset
[scenario] S3 detected half-open after 988 ms; events [
  'disconnected:{"code":"closed"}',
  'disconnected:{"code":"closed","knowledgeGraph":"nm_live_kg"}',
  'reconnected:{"attempt":1}',
  'session_reset:{}',
  'notification_gap:{"code":"no_cursor","message":"Reconnected with no notification cursor: notifications sent while disconnected were not replayed"}',
  'reconnected:{"attempt":1,"knowledgeGraph":"nm_live_kg"}',
  'session_reset:{"knowledgeGraph":"nm_live_kg"}',
  'notification_gap:{"code":"no_cursor","message":"Reconnected with no notification cursor: notifications sent while disconnected were not replayed","knowledgeGraph":"nm_live_kg"}'
] ; query after reconnect [ [ 5, 6 ] ]

 ✓ tests/zz_nm_probe.test.ts > S3 half-open connection (peer stops answering) is detected by keepalive and reconnected with session_reset 1114ms
stdout | tests/zz_nm_probe.test.ts > S4 write whose reply is lost to a dropped connection fails ConnectionLost(mayHaveCommitted) and the write did commit
[scenario] S4 write error ConnectionLostError Connection lost: The connection closed code closed mayHaveCommitted true program +marker(42)
[scenario] S4 read error ConnectionLostError mayHaveCommitted false program ?edge(1, Y)

stdout | tests/zz_nm_probe.test.ts > S4 write whose reply is lost to a dropped connection fails ConnectionLost(mayHaveCommitted) and the write did commit
[scenario] S4 marker on server [ [ 42 ] ]

 ✓ tests/zz_nm_probe.test.ts > S4 write whose reply is lost to a dropped connection fails ConnectionLost(mayHaveCommitted) and the write did commit 374ms
stdout | tests/zz_nm_probe.test.ts > S5 reconnect gives up after maxReconnectAttempts with ConnectionLost
[scenario] S5 events [
  'disconnected:{"code":"closed"}',
  'disconnected:{"code":"closed","knowledgeGraph":"nm_live_kg"}',
  'closed:{"error":{"name":"ConnectionLostError","code":"closed","mayHaveCommitted":false}}',
  'closed:{"error":{"name":"ConnectionLostError","code":"closed","mayHaveCommitted":false},"knowledgeGraph":"nm_live_kg"}'
] ; later call error ConnectionLostError Reconnecting failed after 3 attempt(s): Failed to connect to ws://127.0.0.1:46091/ws?kg=nm_live_kg: Error: socket hang up

 ✓ tests/zz_nm_probe.test.ts > S5 reconnect gives up after maxReconnectAttempts with ConnectionLost 349ms
stdout | tests/zz_nm_probe.test.ts > S6 deadline exceeded is typed, carries the program, and the connection stays usable; invalid request is ProtocolError
[scenario] S6 deadline error after 101 ms: DeadlineExceededError deadline_exceeded program= ?reach(X, Y), X < 5000, Y > 798

 ✓ tests/zz_nm_probe.test.ts > S6 deadline exceeded is typed, carries the program, and the connection stays usable; invalid request is ProtocolError 120ms
stdout | tests/zz_nm_probe.test.ts > S7 rate_limited is retried once then surfaces as RateLimitedError; connection stays usable
[scenario] S7 300 pipelined calls against a 100 msg/s server: { ok: 199, RateLimitedError: 101 } events []

 ✓ tests/zz_nm_probe.test.ts > S7 rate_limited is retried once then surfaces as RateLimitedError; connection stays usable 2174ms

 Test Files  1 passed (1)
      Tests  7 passed (7)
   Start at  12:27:51
   Duration  20.02s (transform 158ms, setup 0ms, collect 206ms, tests 19.54s, environment 0ms, prepare 70ms)
Evidence: Live scenario driver source
/* Disposable live-validation script (no-mistakes test phase); not part of the change. */
import net from 'node:net';
import { afterAll, beforeAll, expect, it } from 'vitest';
import { ConnectionLostError, DeadlineExceededError, InputLayer, ProtocolError, QueryError } from '../src/index';

const ENGINE = 8197; const IDLE_ENGINE = 8198;
const log = (...a: unknown[]) => console.log('[scenario]', ...a);

/** TCP proxy we can freeze (half-open), blackhole downstream, or refuse. */
function proxy(upstream: number) {
  let mode: 'pass' | 'freeze' | 'down-blackhole' | 'refuse' = 'pass';
  const pairs: Array<[net.Socket, net.Socket]> = [];
  const server = net.createServer((c) => {
    if (mode === 'refuse') { c.destroy(); return; }
    const u = net.connect(upstream, '127.0.0.1');
    pairs.push([c, u]);
    c.on('data', (d) => { if (mode === 'pass' || mode === 'down-blackhole') u.write(d); });
    u.on('data', (d) => { if (mode === 'pass') c.write(d); });
    c.on('close', () => u.destroy()); u.on('close', () => c.destroy());
    c.on('error', () => {}); u.on('error', () => {});
  });
  return {
    listen: () => new Promise<number>((r) => server.listen(0, '127.0.0.1', () => r((server.address() as net.AddressInfo).port))),
    set: (m: typeof mode) => { mode = m; },
    killAll: () => pairs.splice(0).forEach(([c, u]) => { c.destroy(); u.destroy(); }),
    close: () => new Promise((r) => server.close(r)),
  };
}
const events = (il: InputLayer) => {
  const seen: string[] = [];
  for (const t of ['disconnected', 'reconnected', 'session_reset', 'notification_gap', 'closed'])
    il.events.addEventListener(t, (e) => seen.push(`${t}:${JSON.stringify((e as CustomEvent).detail)}`));
  return seen;
};
const sleep = (ms: number) => new Promise((r) => setTimeout(r, ms));

const KG = 'nm_live_kg';
const N = 800;
beforeAll(async () => {
  const il = new InputLayer({ url: `ws://127.0.0.1:${ENGINE}/ws`, username: 'admin', password: 'admin' });
  await il.connect();
  await il.dropKnowledgeGraph(KG).catch(() => undefined);
  const a = il.knowledgeGraph(KG);
  await a.execute(`+edge[${Array.from({ length: N }, (_, i) => `(${i}, ${i + 1})`).join(', ')}]`);
  await a.execute('+reach(X, Y) <- edge(X, Y)\n+reach(X, Z) <- reach(X, Y), edge(Y, Z)');
  await il.close();
}, 60000);
afterAll(async () => {
  const il = new InputLayer({ url: `ws://127.0.0.1:${ENGINE}/ws`, username: 'admin', password: 'admin' });
  await il.connect(); await il.dropKnowledgeGraph(KG).catch(() => undefined); await il.close();
});

it('S1 long query in flight past several keepalive intervals is not dropped', async () => {
  const il = new InputLayer({ url: `ws://127.0.0.1:${ENGINE}/ws`, username: 'admin', password: 'admin', keepaliveMs: 300, timeoutGraceMs: 300 });
  const seen = events(il);
  await il.connect();
  const t = Date.now();
  const r = await il.knowledgeGraph(KG).execute(`?reach(X, Y), X < 5000, Y > ${N - 2}`, { timeoutMs: 0 });
  const took = Date.now() - t;
  log('S1 long query took', took, 'ms; rows', r.toTuples().length, '; events', seen);
  expect(took).toBeGreaterThan(4 * 300);
  expect(r.toTuples().length).toBe(2 * N - 1);
  expect(seen.filter((e) => !e.startsWith('closed'))).toEqual([]);
  await il.close();
}, 60000);

it('S2 idle listener with keepalive survives server idle timeout and still receives notifications', async () => {
  const listener = new InputLayer({ url: `ws://127.0.0.1:${IDLE_ENGINE}/ws`, username: 'admin', password: 'admin', keepaliveMs: 1000 });
  const seen = events(listener);
  await listener.connect();
  const kg = listener.knowledgeGraph('nm_idle');
  await kg.execute('+t(1)');
  const notes: unknown[] = [];
  listener.on('persistent_update', (e) => { notes.push(e); }, { knowledgeGraph: 'nm_idle' });
  await sleep(8000); // > 2x the server's 3 s idle timeout
  const writer = new InputLayer({ url: `ws://127.0.0.1:${IDLE_ENGINE}/ws`, username: 'admin', password: 'admin' });
  await writer.connect();
  await writer.knowledgeGraph('nm_idle').execute('+t(2)');
  await sleep(500);
  log('S2 keepalive on: events', seen, 'notifications', notes.length);
  expect(seen).toEqual([]);
  expect(notes.length).toBeGreaterThan(0);
  await writer.close();
  // Control: keepalive disabled -> the server's idle timeout does close the connection.
  const nokeep = new InputLayer({ url: `ws://127.0.0.1:${IDLE_ENGINE}/ws`, username: 'admin', password: 'admin', keepaliveMs: 0, reconnectDelay: 0.05 });
  const seen2 = events(nokeep);
  await nokeep.connect();
  await nokeep.knowledgeGraph('nm_idle').execute('?t(X)');
  await sleep(4500);
  log('S2 control keepalive off: events', seen2);
  expect(seen2.some((e) => e.startsWith('disconnected'))).toBe(true);
  await listener.dropKnowledgeGraph('nm_idle').catch(() => undefined);
  await listener.close(); await nokeep.close();
}, 60000);

it('S3 half-open connection (peer stops answering) is detected by keepalive and reconnected with session_reset', async () => {
  const p = proxy(ENGINE); const port = await p.listen();
  const il = new InputLayer({ url: `ws://127.0.0.1:${port}/ws`, username: 'admin', password: 'admin', keepaliveMs: 500, timeoutGraceMs: 500, reconnectDelay: 0.05 });
  const seen = events(il);
  await il.connect();
  const kg = il.knowledgeGraph(KG);
  await kg.execute('?edge(0, Y)');
  p.set('freeze');
  const t = Date.now();
  while (!seen.some((e) => e.startsWith('disconnected')) && Date.now() - t < 10000) await sleep(20);
  const detectMs = Date.now() - t;
  p.set('pass');
  while (!seen.some((e) => e.startsWith('reconnected')) && Date.now() - t < 15000) await sleep(20);
  const r = await kg.execute('?edge(5, Y)');
  log('S3 detected half-open after', detectMs, 'ms; events', seen, '; query after reconnect', r.toTuples());
  expect(detectMs).toBeLessThan(3000);
  expect(seen.some((e) => e.startsWith('reconnected'))).toBe(true);
  expect(seen.some((e) => e.startsWith('session_reset'))).toBe(true);
  expect(r.toTuples()).toEqual([[5, 6]]);
  expect(kg.connection.currentKg).toBe(KG);
  await il.close(); await p.close();
}, 30000);

it('S4 write whose reply is lost to a dropped connection fails ConnectionLost(mayHaveCommitted) and the write did commit', async () => {
  const p = proxy(ENGINE); const port = await p.listen();
  const il = new InputLayer({ url: `ws://127.0.0.1:${port}/ws`, username: 'admin', password: 'admin', autoReconnect: false });
  await il.connect();
  const kg = il.knowledgeGraph(KG);
  await kg.execute('?edge(0, Y)');
  p.set('down-blackhole');
  const pw = kg.execute('+marker(42)').catch((e) => e);
  const pr = kg.execute('?edge(1, Y)').catch((e) => e);
  await sleep(300); p.killAll();
  const ew = await pw; const er = await pr;
  log('S4 write error', ew?.constructor?.name, ew?.message, 'code', ew?.code, 'mayHaveCommitted', ew?.mayHaveCommitted, 'program', ew?.iql);
  log('S4 read error', er?.constructor?.name, 'mayHaveCommitted', er?.mayHaveCommitted, 'program', er?.iql);
  expect(ew).toBeInstanceOf(ConnectionLostError);
  expect(ew.mayHaveCommitted).toBe(true);
  expect(ew.iql).toContain('marker(42)');
  expect(er).toBeInstanceOf(ConnectionLostError);
  expect(er.mayHaveCommitted).toBe(false);
  const check = new InputLayer({ url: `ws://127.0.0.1:${ENGINE}/ws`, username: 'admin', password: 'admin' });
  await check.connect();
  const rows = (await check.knowledgeGraph(KG).execute('?marker(X)')).toTuples();
  log('S4 marker on server', rows);
  expect(rows).toEqual([[42]]);
  await check.knowledgeGraph(KG).execute('-marker(42)');
  await check.close(); await il.close().catch(() => undefined); await p.close();
}, 30000);

it('S5 reconnect gives up after maxReconnectAttempts with ConnectionLost', async () => {
  const p = proxy(ENGINE); const port = await p.listen();
  const il = new InputLayer({ url: `ws://127.0.0.1:${port}/ws`, username: 'admin', password: 'admin', reconnectDelay: 0.05, maxReconnectDelay: 0.2, maxReconnectAttempts: 3 });
  const seen = events(il);
  await il.connect();
  const kg = il.knowledgeGraph(KG);
  await kg.execute('?edge(0, Y)');
  p.set('refuse'); p.killAll();
  const t = Date.now();
  while (!seen.some((e) => e.startsWith('closed')) && Date.now() - t < 15000) await sleep(20);
  const err = await kg.execute('?edge(0, Y)').catch((e) => e);
  log('S5 events', seen, '; later call error', err?.constructor?.name, err?.message);
  expect(seen.some((e) => e.startsWith('closed'))).toBe(true);
  expect(err).toBeInstanceOf(ConnectionLostError);
  await il.close().catch(() => undefined); await p.close();
}, 30000);

it('S6 deadline exceeded is typed, carries the program, and the connection stays usable; invalid request is ProtocolError', async () => {
  const il = new InputLayer({ url: `ws://127.0.0.1:${ENGINE}/ws`, username: 'admin', password: 'admin' });
  await il.connect();
  const kg = il.knowledgeGraph(KG);
  const prog = `?reach(X, Y), X < 5000, Y > ${N - 2}`;
  const t = Date.now();
  const e = await kg.execute(prog, { timeoutMs: 100 }).catch((x) => x);
  log('S6 deadline error after', Date.now() - t, 'ms:', e?.constructor?.name, e?.code, 'program=', e?.iql);
  expect(e).toBeInstanceOf(DeadlineExceededError);
  expect(e.iql).toBe(prog);
  expect((await kg.execute('?edge(3, Y)')).toTuples()).toEqual([[3, 4]]);
  await il.close();
}, 30000);

it('S7 rate_limited is retried once then surfaces as RateLimitedError; connection stays usable', async () => {
  const il = new InputLayer({ url: `ws://127.0.0.1:${ENGINE}/ws`, username: 'admin', password: 'admin' });
  await il.connect();
  const kg = il.knowledgeGraph(KG);
  const seen: string[] = events(il);
  const pipe = await Promise.all(Array.from({ length: 300 }, (_, i) => kg.execute(`?edge(${i % N}, Y)`).then(() => 'ok', (e) => e.constructor.name)));
  const tally = pipe.reduce((m: Record<string, number>, k) => ((m[k] = (m[k] ?? 0) + 1), m), {});
  log('S7 300 pipelined calls against a 100 msg/s server:', tally, 'events', seen);
  expect(Object.keys(tally).every((k) => k === 'ok' || k === 'RateLimitedError')).toBe(true);
  await sleep(1100);
  expect((await kg.execute('?edge(1, Y)')).toTuples()).toEqual([[1, 2]]);
  await il.close();
}, 60000);
Evidence: Latency base vs head

base p50=0.520/0.476ms p99=0.734/0.605ms; head p50=0.499/0.477ms p99=0.633/0.597ms


 RUN  v3.2.7 ~/.no-mistakes/worktrees/9eea230c3aae/01M43BFB8M18GTF3V2STQB1V68/packages/inputlayer-js

stdout | tests/zz_nm_latency.test.ts > latency base vs head
[latency] base(1ee8c8f8): sequential ?e(1,Y) x3000 p50=0.520ms p90=0.630ms p99=0.734ms

stdout | tests/zz_nm_latency.test.ts > latency base vs head
[latency] head(e9621371): sequential ?e(1,Y) x3000 p50=0.499ms p90=0.542ms p99=0.633ms

stdout | tests/zz_nm_latency.test.ts > latency base vs head
[latency] base(1ee8c8f8): sequential ?e(1,Y) x3000 p50=0.476ms p90=0.514ms p99=0.605ms

stdout | tests/zz_nm_latency.test.ts > latency base vs head
[latency] head(e9621371): sequential ?e(1,Y) x3000 p50=0.477ms p90=0.508ms p99=0.597ms

 ✓ tests/zz_nm_latency.test.ts (1 test) 6796ms
   ✓ latency base vs head  6795ms

 Test Files  1 passed (1)
      Tests  1 passed (1)
   Start at  12:27:04
   Duration  7.39s (transform 266ms, setup 0ms, collect 331ms, tests 6.80s, environment 0ms, prepare 74ms)
Evidence: Adversarial mock-server fixture tests

 RUN  v3.2.7 ~/.no-mistakes/worktrees/9eea230c3aae/01M43BFB8M18GTF3V2STQB1V68/packages/inputlayer-js

 ✓ tests/connection.test.ts (54 tests) 6501ms
   ✓ connection fixtures > overdue_stream_stalls  303ms
   ✓ connection fixtures > rate_limited_once  1003ms
   ✓ connection fixtures > rate_limited_twice  1002ms
   ✓ keepalive > a long call past the keepalive interval keeps a live connection  402ms
   ✓ cancellation > a queued call is sent with what is left of its deadline  305ms
   ✓ reconnect > a closed connection does not reopen  422ms

 Test Files  1 passed (1)
      Tests  54 passed (54)
   Start at  12:28:17
   Duration  7.01s (transform 203ms, setup 0ms, collect 239ms, tests 6.50s, environment 0ms, prepare 82ms)

Pipeline

Updates from git push no-mistakes

... (7 earlier update rounds omitted to keep the PR body within GitHub's 65536-char limit; full history is in the run log.)

🔧 **Review** - 7 issues found → auto-fixed (7) ✅

🔧 Fix applied.
6 issues (5 warnings, 1 info) still open:

  • ⚠️ packages/inputlayer-js/src/connection.ts:599 - probeServer() assumes that "a live server answers at once, even while computing". That is false when the engine's request pipeline is full. The engine reads client frames, including WebSocket pings, only while requests.has_capacity() (ws.rs:443, pipeline.rs:99), and an admitted cancel takes a pipeline slot until its cancel_ack is released (ws.rs:606-616). With the default bounds (client maxInFlight 15, server 16), 15 calls in flight plus one deadline cancel fill all 16 slots, so the probe's ping is not read and gets no pong. After timeoutGraceMs/2 (1 s) the client terminates a healthy connection. Example: Promise.all of a write that is slow to commit plus 14 queries pipelined behind it (replies come back in request order). At the write's deadline the client sends cancel (too_late) and the probe. The ping goes unread, and after 1 s every call fails with ConnectionLostError: the write with mayHaveCommitted, all the queries needlessly. The connection then reconnects and emits session_reset, so session state is lost. This probe code is from the original commit, and round 1 (the F2 fix) kept it. Fix: don't terminate on a missing pong while the server's window may be full (inFlight.size plus outstanding cancels >= maxInFlight), or count any frame received since the probe started as proof the server is alive. The overdue timer already bounds the call.
  • ⚠️ packages/inputlayer-js/src/connection.ts:556 - overdue() fails the call at deadline + grace even when its reply has already started arriving (call.stream is set by result_start). Example: a query with timeoutMs: 30000 finishes computing at 29 s and starts streaming a large result over a slow link. At 32 s the call rejects with DeadlineExceededError, which claims no reply came, and the rest of a successful result is thrown away. Each later result_chunk and the result_end is then counted separately in staleReplies, because the id is in neither inFlight nor abandoned. The operator's F2 decision limits local failure to a call where "neither the call's reply nor its cancel_ack has arrived", and this path came in with the round-1 F2 fix. Fix: in overdue, if call.stream is set, keep waiting for result_end (or re-arm the grace timer). When a call is failed locally, add its id to abandoned so its remaining frames are dropped as one stale reply.
  • ⚠️ packages/inputlayer-js/src/connection.ts:731 - finish() removes a call from inFlight as soon as the client fails it locally, and then pump() sends more requests, but the engine still holds the request until it replies. Replies come back in request order, so this is reachable: queries pipelined behind a slow write go overdue together and free their slots at once. The client then sends another full window, and the engine's unreleased count goes past ws_max_in_flight_requests. The engine then stops reading, so a later cancel (from an AbortSignal) or probe ping is not read until the head of the line finishes. That breaks the intent's invariant: "at most ws_max_in_flight_requests - 1 requests in flight ... so a cancel can always be sent". The round-1 F2 fix introduced this through overdue() (line 561); abandon() (line 742) has the same early release for broken streams. Fix: keep a locally failed id counted against maxInFlight (for example in abandoned, checked by pump) until its final frame (result, error or result_end) arrives or the connection closes.
  • ℹ️ packages/inputlayer-js/src/connection.ts:976 - After an AuthenticationError the reconnect loop stops at once (line 955), yet the give-up error always says Reconnecting failed after ${maxReconnectAttempts} attempt(s). Example: a credential revoked during an outage gives "failed after 10 attempt(s)" after one attempt. Fix: report the actual attempt count.
  • ⚠️ packages/inputlayer-js/src/connection.ts:577 - Introduced by the round-4 N1/N3 fixes, which followed the operator's instructions to the letter but together leave no way to get a connection back. A locally failed call's slot now stays in abandoned until its final frame arrives or the socket closes (line 577; and line 770 for broken streams). pump counts those slots (line 534). The probe no longer terminates while inFlight + abandoned + control &gt;= maxInFlight (line 625). The keepalive ping's failure is ignored (lines 1026-1029: 'A failed ping surfaces as a close', which never happens on a half-open socket).

Two paths leave the connection open but unable to send.
(a) The F2 case the operator required to stay usable: a live server that answers pings but never replies to a request (a lost reply). Each such call holds a slot for good. After maxInFlight (15) of them, pump never sends again. Every later call fails with DeadlineExceededError 'nothing was applied', or hangs forever with timeoutMs: 0. Pongs keep the socket alive, so it never reconnects. The silent_server tests only cover one slot.
(b) A half-open connection (host crash or partition) while 15 calls are in flight: at each deadline, outstanding >= maxInFlight, so the probe never terminates. All 15 go to abandoned, and the same wedge follows until the OS times out the TCP connection, which can take tens of minutes.

Fixing this means choosing a new liveness rule, which needs a decision. For example: drop and reconnect the connection when abandoned alone fills the window, or when an abandoned slot outlives a bound. Or treat a keepalive ping that times out as a dead connection. Without one of these, the N3 slot accounting turns a single lost reply per slot into a permanently dead handle.

  • ⚠️ packages/inputlayer-js/src/connection.ts:574 - Introduced by the round-4 N2 fix. When the grace period ends while a reply is streaming, overdue returns with call.deadline cleared and does not re-arm any timer, so the call has no bound left. If the connection goes half-open mid-stream, nothing ever notices:
  • the chunks that already arrived cleared the probe (onFrame calls clearProbe);
  • the keepalive skips whenever inFlight.size &gt; 0 (line 1026), so the client sends nothing and TCP never detects the dead peer.
    Example: a query with timeoutMs: 30000 starts streaming at 29 s, and the network partitions at 31 s. The call stays pending forever and holds its slot. Before round 4 it failed at deadline + grace. Fix, using the alternative the operator's N2 instruction allows ('or re-arm the grace timer'): in overdue, if call.stream is set, re-arm call.deadline = setTimeout(() =&gt; this.overdue(call), timeoutGraceMs) and reset it on each result_chunk. A stream that keeps arriving completes; a stalled one fails locally like a silent call.

🔧 Fix applied.
8 issues (1 error, 5 warnings, 2 infos) still open:

  • ⚠️ packages/inputlayer-js/src/connection.ts:599 - probeServer() assumes that "a live server answers at once, even while computing". That is false when the engine's request pipeline is full. The engine reads client frames, including WebSocket pings, only while requests.has_capacity() (ws.rs:443, pipeline.rs:99), and an admitted cancel takes a pipeline slot until its cancel_ack is released (ws.rs:606-616). With the default bounds (client maxInFlight 15, server 16), 15 calls in flight plus one deadline cancel fill all 16 slots, so the probe's ping is not read and gets no pong. After timeoutGraceMs/2 (1 s) the client terminates a healthy connection. Example: Promise.all of a write that is slow to commit plus 14 queries pipelined behind it (replies come back in request order). At the write's deadline the client sends cancel (too_late) and the probe. The ping goes unread, and after 1 s every call fails with ConnectionLostError: the write with mayHaveCommitted, all the queries needlessly. The connection then reconnects and emits session_reset, so session state is lost. This probe code is from the original commit, and round 1 (the F2 fix) kept it. Fix: don't terminate on a missing pong while the server's window may be full (inFlight.size plus outstanding cancels >= maxInFlight), or count any frame received since the probe started as proof the server is alive. The overdue timer already bounds the call.
  • ⚠️ packages/inputlayer-js/src/connection.ts:556 - overdue() fails the call at deadline + grace even when its reply has already started arriving (call.stream is set by result_start). Example: a query with timeoutMs: 30000 finishes computing at 29 s and starts streaming a large result over a slow link. At 32 s the call rejects with DeadlineExceededError, which claims no reply came, and the rest of a successful result is thrown away. Each later result_chunk and the result_end is then counted separately in staleReplies, because the id is in neither inFlight nor abandoned. The operator's F2 decision limits local failure to a call where "neither the call's reply nor its cancel_ack has arrived", and this path came in with the round-1 F2 fix. Fix: in overdue, if call.stream is set, keep waiting for result_end (or re-arm the grace timer). When a call is failed locally, add its id to abandoned so its remaining frames are dropped as one stale reply.
  • ⚠️ packages/inputlayer-js/src/connection.ts:731 - finish() removes a call from inFlight as soon as the client fails it locally, and then pump() sends more requests, but the engine still holds the request until it replies. Replies come back in request order, so this is reachable: queries pipelined behind a slow write go overdue together and free their slots at once. The client then sends another full window, and the engine's unreleased count goes past ws_max_in_flight_requests. The engine then stops reading, so a later cancel (from an AbortSignal) or probe ping is not read until the head of the line finishes. That breaks the intent's invariant: "at most ws_max_in_flight_requests - 1 requests in flight ... so a cancel can always be sent". The round-1 F2 fix introduced this through overdue() (line 561); abandon() (line 742) has the same early release for broken streams. Fix: keep a locally failed id counted against maxInFlight (for example in abandoned, checked by pump) until its final frame (result, error or result_end) arrives or the connection closes.
  • ℹ️ packages/inputlayer-js/src/connection.ts:976 - After an AuthenticationError the reconnect loop stops at once (line 955), yet the give-up error always says Reconnecting failed after ${maxReconnectAttempts} attempt(s). Example: a credential revoked during an outage gives "failed after 10 attempt(s)" after one attempt. Fix: report the actual attempt count.
  • ⚠️ packages/inputlayer-js/src/connection.ts:577 - Introduced by the round-4 N1/N3 fixes, which followed the operator's instructions to the letter but together leave no way to get a connection back. A locally failed call's slot now stays in abandoned until its final frame arrives or the socket closes (line 577; and line 770 for broken streams). pump counts those slots (line 534). The probe no longer terminates while inFlight + abandoned + control &gt;= maxInFlight (line 625). The keepalive ping's failure is ignored (lines 1026-1029: 'A failed ping surfaces as a close', which never happens on a half-open socket).

Two paths leave the connection open but unable to send.
(a) The F2 case the operator required to stay usable: a live server that answers pings but never replies to a request (a lost reply). Each such call holds a slot for good. After maxInFlight (15) of them, pump never sends again. Every later call fails with DeadlineExceededError 'nothing was applied', or hangs forever with timeoutMs: 0. Pongs keep the socket alive, so it never reconnects. The silent_server tests only cover one slot.
(b) A half-open connection (host crash or partition) while 15 calls are in flight: at each deadline, outstanding >= maxInFlight, so the probe never terminates. All 15 go to abandoned, and the same wedge follows until the OS times out the TCP connection, which can take tens of minutes.

Fixing this means choosing a new liveness rule, which needs a decision. For example: drop and reconnect the connection when abandoned alone fills the window, or when an abandoned slot outlives a bound. Or treat a keepalive ping that times out as a dead connection. Without one of these, the N3 slot accounting turns a single lost reply per slot into a permanently dead handle.

  • ⚠️ packages/inputlayer-js/src/connection.ts:574 - Introduced by the round-4 N2 fix. When the grace period ends while a reply is streaming, overdue returns with call.deadline cleared and does not re-arm any timer, so the call has no bound left. If the connection goes half-open mid-stream, nothing ever notices:
  • the chunks that already arrived cleared the probe (onFrame calls clearProbe);
  • the keepalive skips whenever inFlight.size &gt; 0 (line 1026), so the client sends nothing and TCP never detects the dead peer.
    Example: a query with timeoutMs: 30000 starts streaming at 29 s, and the network partitions at 31 s. The call stays pending forever and holds its slot. Before round 4 it failed at deadline + grace. Fix, using the alternative the operator's N2 instruction allows ('or re-arm the grace timer'): in overdue, if call.stream is set, re-arm call.deadline = setTimeout(() =&gt; this.overdue(call), timeoutGraceMs) and reset it on each result_chunk. A stream that keeps arriving completes; a stalled one fails locally like a silent call.
  • 🚨 packages/inputlayer-js/src/connection.ts:1043 - The round-5 L1 fix brought this in. The keepalive now sends an application {type:&#39;ping&#39;, id} while calls are in flight and drops the socket when no pong arrives within timeoutGraceMs. The engine, however, answers an application ping as an ordinary pipelined request (request.rs:100 Request::immediate, "released in order with the others"; ws.rs:137 "Replies come back in request order"). Its pong therefore waits behind every earlier request. Concrete sequence with the defaults (keepaliveMs 20 000, grace 2 000, defaultTimeoutMs 30 000): kg.execute(q) starts a query that takes 25 s on a healthy server. Nothing else is sent, so the keepalive tick at about 20 s sends a ping, and that pong cannot be released until the query's reply at 25 s. At about 22 s request() times out and ws.terminate() runs. The healthy query fails with ConnectionLostError, a write fails with mayHaveCommitted, and the reconnect emits session_reset. With timeoutMs: 0 (no deadline), any call that runs longer than keepaliveMs + grace is killed the same way. The same false positive occurs with inFlight 0 when the engine still holds an abandoned call: the round-5 change no longer counts abandoned calls, but the engine still orders their replies first. Sibling: the ping also takes an engine pipeline slot, so 15 calls in flight plus the keepalive ping fill the engine's 16 slots. The engine then stops reading, and a deadline or signal cancel can no longer be read, which breaks the intent's 'ws_max_in_flight_requests - 1 so a cancel can always be sent'. The tests cannot catch this because the mock server (connection.test.ts:87) answers application pings at once, out of order. Fix: check liveness with a transport ws.ping() and its pong (as probeServer does; that pong does not wait behind the request order and takes no pipeline slot). Keep the application ping only for idle keep-alive, with no terminate-on-timeout, or count any frame received as proof of life. Add a mock fixture in which a ping's pong waits behind a slow reply.
  • ℹ️ packages/inputlayer-js/src/client.ts:55 - The keepaliveMs docs in client.ts:55 and docs/content/docs/guides/js-sdk.mdx:66 still describe only idle keep-alive. They don't say that the round-5 keepalive now also drops and reconnects a connection whose ping goes unanswered within timeoutGraceMs, though connection.ts:88-93 was updated to say so. Align the wording.

🔧 Fix applied.
8 issues (1 error, 5 warnings, 2 infos) still open:

  • ⚠️ packages/inputlayer-js/src/connection.ts:599 - probeServer() assumes that "a live server answers at once, even while computing". That is false when the engine's request pipeline is full. The engine reads client frames, including WebSocket pings, only while requests.has_capacity() (ws.rs:443, pipeline.rs:99), and an admitted cancel takes a pipeline slot until its cancel_ack is released (ws.rs:606-616). With the default bounds (client maxInFlight 15, server 16), 15 calls in flight plus one deadline cancel fill all 16 slots, so the probe's ping is not read and gets no pong. After timeoutGraceMs/2 (1 s) the client terminates a healthy connection. Example: Promise.all of a write that is slow to commit plus 14 queries pipelined behind it (replies come back in request order). At the write's deadline the client sends cancel (too_late) and the probe. The ping goes unread, and after 1 s every call fails with ConnectionLostError: the write with mayHaveCommitted, all the queries needlessly. The connection then reconnects and emits session_reset, so session state is lost. This probe code is from the original commit, and round 1 (the F2 fix) kept it. Fix: don't terminate on a missing pong while the server's window may be full (inFlight.size plus outstanding cancels >= maxInFlight), or count any frame received since the probe started as proof the server is alive. The overdue timer already bounds the call.
  • ⚠️ packages/inputlayer-js/src/connection.ts:556 - overdue() fails the call at deadline + grace even when its reply has already started arriving (call.stream is set by result_start). Example: a query with timeoutMs: 30000 finishes computing at 29 s and starts streaming a large result over a slow link. At 32 s the call rejects with DeadlineExceededError, which claims no reply came, and the rest of a successful result is thrown away. Each later result_chunk and the result_end is then counted separately in staleReplies, because the id is in neither inFlight nor abandoned. The operator's F2 decision limits local failure to a call where "neither the call's reply nor its cancel_ack has arrived", and this path came in with the round-1 F2 fix. Fix: in overdue, if call.stream is set, keep waiting for result_end (or re-arm the grace timer). When a call is failed locally, add its id to abandoned so its remaining frames are dropped as one stale reply.
  • ⚠️ packages/inputlayer-js/src/connection.ts:731 - finish() removes a call from inFlight as soon as the client fails it locally, and then pump() sends more requests, but the engine still holds the request until it replies. Replies come back in request order, so this is reachable: queries pipelined behind a slow write go overdue together and free their slots at once. The client then sends another full window, and the engine's unreleased count goes past ws_max_in_flight_requests. The engine then stops reading, so a later cancel (from an AbortSignal) or probe ping is not read until the head of the line finishes. That breaks the intent's invariant: "at most ws_max_in_flight_requests - 1 requests in flight ... so a cancel can always be sent". The round-1 F2 fix introduced this through overdue() (line 561); abandon() (line 742) has the same early release for broken streams. Fix: keep a locally failed id counted against maxInFlight (for example in abandoned, checked by pump) until its final frame (result, error or result_end) arrives or the connection closes.
  • ℹ️ packages/inputlayer-js/src/connection.ts:976 - After an AuthenticationError the reconnect loop stops at once (line 955), yet the give-up error always says Reconnecting failed after ${maxReconnectAttempts} attempt(s). Example: a credential revoked during an outage gives "failed after 10 attempt(s)" after one attempt. Fix: report the actual attempt count.
  • ⚠️ packages/inputlayer-js/src/connection.ts:577 - Introduced by the round-4 N1/N3 fixes, which followed the operator's instructions to the letter but together leave no way to get a connection back. A locally failed call's slot now stays in abandoned until its final frame arrives or the socket closes (line 577; and line 770 for broken streams). pump counts those slots (line 534). The probe no longer terminates while inFlight + abandoned + control &gt;= maxInFlight (line 625). The keepalive ping's failure is ignored (lines 1026-1029: 'A failed ping surfaces as a close', which never happens on a half-open socket).

Two paths leave the connection open but unable to send.
(a) The F2 case the operator required to stay usable: a live server that answers pings but never replies to a request (a lost reply). Each such call holds a slot for good. After maxInFlight (15) of them, pump never sends again. Every later call fails with DeadlineExceededError 'nothing was applied', or hangs forever with timeoutMs: 0. Pongs keep the socket alive, so it never reconnects. The silent_server tests only cover one slot.
(b) A half-open connection (host crash or partition) while 15 calls are in flight: at each deadline, outstanding >= maxInFlight, so the probe never terminates. All 15 go to abandoned, and the same wedge follows until the OS times out the TCP connection, which can take tens of minutes.

Fixing this means choosing a new liveness rule, which needs a decision. For example: drop and reconnect the connection when abandoned alone fills the window, or when an abandoned slot outlives a bound. Or treat a keepalive ping that times out as a dead connection. Without one of these, the N3 slot accounting turns a single lost reply per slot into a permanently dead handle.

  • ⚠️ packages/inputlayer-js/src/connection.ts:574 - Introduced by the round-4 N2 fix. When the grace period ends while a reply is streaming, overdue returns with call.deadline cleared and does not re-arm any timer, so the call has no bound left. If the connection goes half-open mid-stream, nothing ever notices:
  • the chunks that already arrived cleared the probe (onFrame calls clearProbe);
  • the keepalive skips whenever inFlight.size &gt; 0 (line 1026), so the client sends nothing and TCP never detects the dead peer.
    Example: a query with timeoutMs: 30000 starts streaming at 29 s, and the network partitions at 31 s. The call stays pending forever and holds its slot. Before round 4 it failed at deadline + grace. Fix, using the alternative the operator's N2 instruction allows ('or re-arm the grace timer'): in overdue, if call.stream is set, re-arm call.deadline = setTimeout(() =&gt; this.overdue(call), timeoutGraceMs) and reset it on each result_chunk. A stream that keeps arriving completes; a stalled one fails locally like a silent call.
  • 🚨 packages/inputlayer-js/src/connection.ts:1043 - The round-5 L1 fix brought this in. The keepalive now sends an application {type:&#39;ping&#39;, id} while calls are in flight and drops the socket when no pong arrives within timeoutGraceMs. The engine, however, answers an application ping as an ordinary pipelined request (request.rs:100 Request::immediate, "released in order with the others"; ws.rs:137 "Replies come back in request order"). Its pong therefore waits behind every earlier request. Concrete sequence with the defaults (keepaliveMs 20 000, grace 2 000, defaultTimeoutMs 30 000): kg.execute(q) starts a query that takes 25 s on a healthy server. Nothing else is sent, so the keepalive tick at about 20 s sends a ping, and that pong cannot be released until the query's reply at 25 s. At about 22 s request() times out and ws.terminate() runs. The healthy query fails with ConnectionLostError, a write fails with mayHaveCommitted, and the reconnect emits session_reset. With timeoutMs: 0 (no deadline), any call that runs longer than keepaliveMs + grace is killed the same way. The same false positive occurs with inFlight 0 when the engine still holds an abandoned call: the round-5 change no longer counts abandoned calls, but the engine still orders their replies first. Sibling: the ping also takes an engine pipeline slot, so 15 calls in flight plus the keepalive ping fill the engine's 16 slots. The engine then stops reading, and a deadline or signal cancel can no longer be read, which breaks the intent's 'ws_max_in_flight_requests - 1 so a cancel can always be sent'. The tests cannot catch this because the mock server (connection.test.ts:87) answers application pings at once, out of order. Fix: check liveness with a transport ws.ping() and its pong (as probeServer does; that pong does not wait behind the request order and takes no pipeline slot). Keep the application ping only for idle keep-alive, with no terminate-on-timeout, or count any frame received as proof of life. Add a mock fixture in which a ping's pong waits behind a slow reply.
  • ℹ️ packages/inputlayer-js/src/connection.ts:639 - The round-6 K1/K2 doc updates (client.ts:55, connection.ts:92-94 and the header at connection.ts:24-28) say the connection is dropped when there is "no WebSocket pong and no frame within timeoutGraceMs". probeServer actually terminates after Math.floor(timeoutGraceMs / 2) (line 639). It also never terminates while inFlight.size + control.size &gt;= maxInFlight, which the option docs leave out. The probe now runs on every keepalive tick rather than only after a deadline, so this window decides when an idle listener counts as dead. With the defaults it gives up after 1 s, where the docs say 2 s. Fix: make the docs and the code agree. Either state the half-grace window, or use the full timeoutGraceMs as the operator's K1 wording describes ("no pong and no frame within timeoutGraceMs").

🔧 Fix applied.
1 info still open:

  • ℹ️ gui/lib/docs-bundle.ts:1482 - The committed GUI docs bundle is out of date. js-sdk.mdx was edited in later rounds, including this round's keepaliveMs comment at js-sdk.mdx:66-67, but the bundled guides/js-sdk content still has the older text keepaliveMs: 20_000, // ping a silent connection so it is never idle-closed (default: 20 000). Builds regenerate the bundle (gui/package.json prebuild), so the shipped GUI is not affected. The committed file still disagrees with its source, though. Fix: rerun node gui/scripts/bundle-docs.mjs and commit the result.

🔧 Fix applied.
✅ Re-checked - no issues remain.

✅ **Test** - passed

✅ No issues found.

  • Live validation: ✅ go - 14 of 15 scenarios driven live against the product
Scenario Result Live Evidence
40 pipelined queries on one handle each resolve with their own reply ✅ pass live make-js-test-live.log (connection.integration: pipelined calls)
Query with timeoutMs 50 fails DeadlineExceededError carrying iql; the connection stays usable (no CancelledError race over 10 repeats) ✅ pass live make-js-test-live.log, connection-live-x10.log
Aborting a running call yields CancelledError and the next call works ✅ pass live make-js-test-live.log
Two handles on different graphs, interleaved, never cross; each connection is bound to its own kg ✅ pass live make-js-test-live.log
Idle client receives persistent_update notifications with no call in flight ✅ pass live make-js-test-live.log
Socket terminated: the handle reconnects to the same graph and replays notifications missed meanwhile from the cursor ✅ pass live make-js-test-live.log
Keepalive: a handle idle 8 s on a server with a 3 s idle timeout gets no disconnect; the keepalive-off control is closed with code idle_timeout ✅ pass live live-driver.log (keepalive ON events: [], OFF: disconnected idle_timeout)
Server restart: reconnected, session_reset and notification_gap events fire, stats.reconnects increments, the handle still reads its graph ✅ pass live live-driver.log (restart events)
Server SIGKILLed mid-call: read fails ConnectionLostError mayHaveCommitted=false, write mayHaveCommitted=true, both carry iql ✅ pass live live-driver.log
Server gone for good: after 3 reconnect attempts the waiting call fails ConnectionLostError with iql and a closed event is emitted ✅ pass live live-driver.log (give-up error after 187 ms)
Rate limited at 5 msg/s: 8 burst calls all succeed via one retry; with 14 calls the excess fails RateLimitedError carrying iql ✅ pass live live-driver.log
R2: kg.execute with timeoutMs 500 during a reconnect fails DeadlineExceededError at about 500 ms ✅ pass live live-driver-r2-bound.log (501 ms)
In-flight bound: 30 long pipelined queries keep 15 in flight; aborting one cancels it and the other 29 complete correctly ✅ pass live live-driver-r2-bound.log
Millisecond latency does not regress versus the base SDK ✅ pass live latency-base-vs-new.log (p50 ~0.49 ms both)
Adversarial frames (out-of-order replies, missing or repeated chunks, count mismatch, unknown ids, malformed frames, invalid_request as ProtocolError, too_late cancel_ack, silent server leading to Out… ⏸️ untested no The prior payload did not establish a live result. This scenario was covered only by mock-server fixtures in npx vitest run tests/connection.test.ts, not against the real engine. A real inputlayer-s…
  • npx vitest run tests/connection.test.ts tests/notifications.test.ts tests/statement-errors.test.ts (mock-server frame fixtures, 83 passed)
  • make js-test-live (builds the release server and runs docs-queries plus connection live tests against it: 32 passed)
  • tests/connection.integration.test.ts repeated 10 times against a live server, checking the deadline/cancel race (R1) for flakiness: 10/10 passed
  • Custom live driver zz-nm-live-driver.integration.test.ts, which spawns disposable servers: keepalive vs a 3 s idle timeout, server restart, SIGKILL mid-call, server gone for good, rate limit at 5 msg/s, R2 deadline during reconnect, 30 pipelined calls with one abort
  • Live latency comparison of the base SDK (ea9d5d3c src) against the new SDK on the same server: 2000 sequential small queries x3 rounds

✅ No issues found.

  • Live validation: ✅ go - 9 of 9 scenarios driven live against the product
Scenario Result Live Evidence
40 pipelined queries on one handle each get their own reply; two handles on different graphs never cross ✅ pass live live-connection-integration.txt (pipelined calls, two handles interleaved)
A deadline fails with DeadlineExceeded carrying the program sent, and the connection stays usable; an aborted call is cancelled on the server ✅ pass live live-scenarios.txt S6 (DeadlineExceededError after 101 ms, iql set); live-connection-integration.txt (abort test)
A ~2 s query in flight across several 300 ms keepalive intervals completes without a drop or reconnect ✅ pass live live-scenarios.txt S1: 2048 ms, 1599 rows, no events
An idle listener with keepalive outlives the server's 3 s idle timeout and still receives notifications; with keepalive off it is idle-closed and reconnects ✅ pass live live-scenarios.txt S2
A half-open connection (proxy freezes traffic) is detected by the keepalive probe, reconnected to the same ?kg= graph, and emits session_reset and notification_gap ✅ pass live live-scenarios.txt S3: detected in 989 ms; query after reconnect returned [[5,6]]
Write whose reply is lost fails with ConnectionLost(mayHaveCommitted=true), and the write is present on the server; a read fails with mayHaveCommitted=false ✅ pass live live-scenarios.txt S4
Reconnect replays notifications missed during an outage; when reconnects are refused, the client gives up after maxReconnectAttempts with ConnectionLost ✅ pass live live-connection-integration.txt (reconnect replay); live-scenarios.txt S5 ('Reconnecting failed after 3 attempt(s)')
Bursting past the server rate limit: rate_limited is retried once, the remainder surface as RateLimitedError, and the connection stays usable ✅ pass live live-scenarios.txt S7: {ok:199, RateLimitedError:101}, no disconnect events
Millisecond query latency does not regress versus the base SDK ✅ pass live latency-base-vs-head.txt
  • cargo build --release --bin inputlayer-server, then isolated engines on 127.0.0.1:8197 (defaults), :8198 (INPUTLAYER_HTTP__WS_IDLE_TIMEOUT_MS=3000) and :8199 (rate limit raised), each with its own temp data dir
  • INPUTLAYER_TEST_SERVER=ws://127.0.0.1:8197/ws npx vitest run tests/connection.integration.test.ts (7 live connection-core tests)
  • Throwaway live scenario script (S1-S7) run against the real engines, with a controllable TCP proxy to simulate half-open, blackholed-reply and refused connections
  • Throwaway latency benchmark: base SDK (1ee8c8f8 src) vs head SDK, 3000 sequential queries x2 rounds against the real engine
  • npx vitest run tests/connection.test.ts (adversarial frame fixtures replayed through the mock server, 54 tests)
✅ **Document** - passed

✅ No issues found.

✅ No issues found.

✅ **Lint** - passed

✅ No issues found.

✅ No issues found.

✅ **Push** - passed

✅ No issues found.

✅ No issues found.

jsam added 9 commits October 4, 2026 12:03
…n core

Replace the single nextFrame resolver in connection.ts with one routed
reader: every request carries an id and its replies (including streamed
results) go back to the call that sent it, so calls pipeline on one
connection. Pushes route to the notification dispatcher or to a subscription
route by name and generation (stale generations are dropped and counted).

- in-flight bound (15, one below the engine's) with a client-side queue
- deadlines: timeout_ms on every call, then cancel and a transport ping
  probe; typed DeadlineExceededError, CancelledError (AbortSignal),
  ProtocolError, RateLimitedError (one resend after the rate window)
- pending calls fail with ConnectionLostError(code, mayHaveCommitted)
- idle delivery and a keepalive ping so a listening client is never
  idle-closed; notification iterators buffer instead of dropping
- reconnect with jittered backoff re-opens the same graph (?kg=) with the
  notification cursor (last_seq + epoch); queued calls wait for it;
  reconnected / session_reset / notification_gap events
- one connection per knowledge graph handle, bound with ?kg= and created
  on first use, instead of .kg use on a shared socket
- adversarial frame fixtures replayed through a mock server, and live
  tests in make js-test-live
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant