Repository navigation
Merged
Conversation
Deliver every result, snapshot and subscription delta whole, or report it, never cut: - A payload whose frame would exceed 1 MiB is streamed as one logical payload with byte-bounded chunks (<= 500 rows, ~1 MiB, never over the 16 MiB frame limit): result_start/chunk/end (the .subscribe reply now names its subscription on result_start) and the new subscription_delta_start/chunk/end, whose end frame carries the counts the chunks must add up to. Clients apply only at the end. - Rows are sized before the first frame is sent, so a row no frame can carry is reported up front: an error for a reply (and a .subscribe registers nothing), or the new subscription_reset push that removes the subscription. An ACL-withheld delta also resets instead of leaving the client behind a skipped seq. - Sizing serializes into a counter and the single-frame attempt stops at the budget, so large payloads cost no whole-payload copy; small frames are serialized once, as before. - The connection writes one frame at a time; a frame the client leaves unread for http.ws_send_timeout_ms (default 30 s) fails the connection and nothing more is written. The timer is armed only when the socket makes a frame wait. Protocol version 3. The testkit agent reassembles streamed deltas strictly (missing, duplicate, out-of-order chunks and count mismatches are violations), the W05 known-defect scenario becomes a required e2e pass, and the SDK/GUI frame types learn the new frames.
This was referenced Oct 4, 2026
Closed
jsam
added a commit
that referenced
this pull request
Oct 4, 2026
…ry request (#249) The release-checkpoint performance gate fails on main: cheap_query serial p50 is 1.110x the approved baseline e732cd5 on the 32-vCPU runner (115 -> 126 us). Concurrent latency and throughput did not move. Bisected to #214 (per-connection request pipelining), made worse by #222. Cause: the pipeline runs requests in a `FuturesUnordered`. Its `poll_next` wakes its own task and returns `Pending` after any poll that polled every future in the set. With one request in flight, which is every serial client, that happens on each request's first poll. The connection task wakes itself, tokio schedules that as a yield, and a yield wakes an idle worker to take the task. Each request therefore costs one more context switch (4.0 -> 5.0 per query) and a hand-off of the connection task to another core. #222 added no work per query but made the hand-off dearer: user cycles per query rose 15% at the same instruction count. Fix: keep one never-woken future in the set. After the pipeline's first poll it is never polled again, so no poll reaches every future and the forced yield no longer fires. The yield after futures that wake themselves still applies. Pipelining, ordering, deadlines, cancellation and the credential fence are unchanged. Nothing is added per request: no lock, atomic or allocation. Each connection gets one more set node. Bisect (cheap_query, `?edge(1, Y)`, serial). Server counters per query from perf stat. Ratios are against e732cd5, from an interleaved probe with one pinned server per commit, 40 rounds: | commit | PR | user instr/query | ctx switches/query | p50 ratio | |---|---|---|---|---| | e732cd5 | baseline | 1.282M | 4.0 | 1.000 | | d75ce6e | #208 atomic programs | 1.279M | 4.0 | 0.993 | | 4578fe6 | #215 notification stream | 1.271M | 4.0 | 0.991 | | 71a4d24 | #216 online backup | 1.275M | 4.0 | 1.016 | | 2f621c5 | #200 first-update latency | 1.272M | 4.0 | 1.009 | | e96df5c | #207 | 1.279M | 4.0 | 0.998 | | 4dd8fe9 | #209 shard index | 1.272M | 4.0 | 0.987 | | 277c7f0 | #206 | 1.274M | 4.0 | 1.008 | | 3554c98 | #214 pipelining, deadlines | 1.296M | 5.0 | 1.020 | | 3492a59 | #213 | 1.294M | 5.0 | 1.030 | | 2ba9a5a | #212 provenance | 1.297M | 5.0 | 1.029 | | 9ed61b4 | #204 key expiry | 1.297M | 5.0 | 1.027 | | be35ee3 | #221 | 1.297M | 5.0 | 1.013 | | c98a590 | #222 bounded streaming | 1.298M | 5.0 | 1.047 | | a6fd628 | #219 wildcard fix | 1.297M | 5.0 | 1.061 | | b7f7ad6 | #220 WAL durability | 1.300M | 5.0 | 1.056 | | de54807 | #223 shared standing eval | 1.299M | 5.0 | 1.071 | | 55ec34d | #225 checkpoint-1 fixes | 1.300M | 5.0 | 1.045 | | 23712b8 | main | 1.300M | 5.0 | 1.064 | | this change | | 1.300M | 4.0 | 1.033 | Confirming A/Bs with the gate's own fixture (fresh pinned servers each round, 40 rounds, ratio [95% interval]): - #214 vs its parent 277c7f0: 1.039 [1.027..1.049] - #222 vs its parent be35ee3: 1.025 [1.006..1.044] - A/A 3554c98: 1.010 [1.002..1.025] - e732cd5 -> main: 1.048 [1.028..1.066] - main -> this change: 0.978 [0.958..0.989] - e732cd5 -> this change: 1.021 [1.009..1.044], PASS (budget 1.05) User cycles per query (perf stat, same instruction count): e732cd5 958K, be35ee3 1053K, c98a590 1206K, this change 1006K. Full gate, all fixtures, 20 rounds, servers pinned to CPUs 2-15: - main -> this change: no metric worse. Every required metric is PASS, or INCONCLUSIVE with an interval that includes 1.0. cheap_query p50 0.979 [0.953..1.007]. - e732cd5 -> this change: cheap_query p50 1.029 [1.019..1.072], INCONCLUSIVE on a busy host (load average up to 22); bound_query p50 1.022, PASS. The insert and delta metrics improved since the baseline. The shared host is about 4x slower per query than the CI runner, so the same hand-off is a smaller share of each query here than there. The remaining ~1.7% of user instructions per query since #214 is the pipeline's own per-request bookkeeping: the boxed request future (one 4-8 KiB allocation), its set node, the request control and the in-flight maps.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Intent
Implement this work item fully and correctly. Always follow clean code principles and clean architecture. Millisecond query and update latency must not regress; no new locks on hot paths. Keep documentation current with the change. The work item is W05 only, bounded complete snapshot/delta transport over the engine /ws protocol (bounded chunked snapshot/delta transport, subscription_reset, ws_send_timeout_ms): a snapshot or subscription delta too large for one frame is chunked as one logical revision with begin/end frames, or the subscription is explicitly invalidated/reset before the client could adopt undeliverable state; the receiver applies only a complete revision and a partial send never masquerades as complete; bytes and queueing per client are bounded with explicit overflow behavior (a client that stops reading is disconnected after ws_send_timeout_ms). Chunking is linear in payload with no extra whole-result copy, slow readers are isolated from other connections, and small-frame delta latency is unchanged. Cases that must hold: a delta larger than MAX_MESSAGE_SIZE, a disconnect mid-chunk, a missing or duplicate chunk, the result cap, and no further writes after an error; a client is never left silently divergent.
What Changed
/wsserver now caps every frame at the 16 MiB message limit. Large results,.subscribesnapshots, and subscription deltas are streamed as one logical revision. Deltas use the newsubscription_delta_start/subscription_delta_chunk/subscription_delta_endframes (each chunk holds at most 500 rows and about 1 MiB), and the end frame carries the chunk and row counts the client checks against. A change that cannot be delivered whole fails the subscribe with anerror, or removes a live subscription and sends a newsubscription_resetpush. The same reset replaces the oldsubscription_errorwhen read access is revoked.protocol_versiongoes from 2 to 3.http.ws_send_timeout_mssetting (default 30000 ms, 0 = wait forever). A client that leaves a frame unread for longer is disconnected, so a slow reader can't block other connections. A byte-size estimate decides which pushes are offloaded to chunked streaming, so small deltas still go out as a single frame. The new transport lives inws/framing.rs,ws/push.rsandws/stream.rs.ws-protocoltypes, the JS and Python SDKs, and the GUI WS types now handle the new frames. The SDKs reject streamed results whose chunks are out of order or whose counts don't matchresult_end. The WebSocket guide, configuration docs, AsyncAPI spec and docs bundle are updated. Theknown_defectse2e cases are replaced by new delivery tests (tests/ws_delivery_tests.rs,tests/e2e_reactive/delivery.rs) and testkit support for streamed results.Risk Assessment
subscribedfrom streamed subscribe replies and don't check chunk order or counts, small deltas with over 256 rows now take a blocking-pool hop that conflicts with the 'latency unchanged' requirement, and large snapshots are still encoded inline on the connection loop.Testing
I started the real
inputlayer-serverbinary with disposable configs and drove every W05 transport case over/wswith a Python client that records each frame's type and size. Covered: a delta over 16 MiB, a large streamed snapshot (raw frames and the Python SDK, which keepssubscribed), an 18 MiB row that can't fit any frame, a raw-socket client that stops reading, a disconnect mid-stream, the result cap, and small-delta latency. All of these live scenarios passed, and the server logged the expected ws_subscription_reset / ws_row_too_large / ws_send_timeout events. I also ran the targeted in-process/wsdelivery tests, the push/stream/outbound unit tests (including the operator-requested 300-row inline and wide-row offload tests), the testkit broken-stream checks, and the Python and JS SDK stream-assembly tests; all passed. Detecting missing or duplicate chunks happens on the client, and the server never sends a broken stream, so I couldn't test that against the live server. It is marked untested here and is covered only by the testkit and SDK tests.make e2e-reactivewas not run: it needs a full release build and the shared host disk had about 2–5 GB free (the first debug build failed with ENOSPC), so the in-process tests plus the live driver stand in for it. Every server and temp dir I created is gone, and the worktree is clean. This is a protocol change with no UI, so the evidence is CLI transcripts, not screenshots..subscribeon a 3,000-row snapshot streams as result_start (withsubscribed), 6 chunks, result_end row_count=3000. Later deltas use the subscription, and the Python SDK's assembled result keeps `s….subscribeover the cap returns an error, and once back under the cap the next delta is relative to the las…Evidence: Live driver script (real server over /ws)
Evidence: Large delta >16 MiB streamed whole
stream ok: seq=1 chunks=20 inserted=4000 largest_frame=1047192B total=20041637B ~19.1 MiB delta (> 16 MiB frame limit) delivered in 3.48s follow-up: subscription_delta seq=2 inserted_len=1 (single frame)Evidence: Large snapshot streamed as .subscribe reply; Python SDK keeps subscribed
start: result_start subscribed={subscription: snap, generation: 1, revision: 22} end: result_end row_count=3000 chunk_count=6 python SDK: rows=2999 subscribed=Subscribed(subscription='sdk', generation=1, revision=23)Evidence: Oversized row leads to subscription_reset, no further pushes
push: subscription_reset big gen=1 "Delta 1 has a row of 18874377 bytes, over the 16777216-byte message limit. The subscription was removed; subscribe again." no further pushes after reset; .unsubscribe -> error; re-subscribe -> single error frame, nothing registeredEvidence: Stalled reader disconnected after ws_send_timeout_ms; reader isolated
reader: stream ok chunks=39 inserted=8000 (40 MiB); small delta latency while stalled peer pending: 6.9 ms stalled client: server closed the connection after 2.0 MiB; frames seen: start 1, chunk 2, no end server: ws_send_timeout timeout=Some(300ms)Evidence: Disconnect mid-stream
Evidence: Small-delta latency (single frames)
40 commits, each pushing a 1-row and a 300-row delta as single frames; write->both-deltas latency ms: p50=13.70 p90=17.63 max=19.91 (debug build)Evidence: Result cap fails closed
Evidence: Server key events
Evidence: ws_delivery_tests
Evidence: push/stream/outbound unit tests
Evidence: testkit + ws-protocol stream tests
Evidence: Python SDK stream tests
Evidence: JS SDK stream tests
Pipeline
Updates from git push no-mistakes
✅ **intent** - passed
✅ No issues found.
✅ **Rebase** - passed
✅ No issues found.
packages/inputlayer-py/src/inputlayer/connection.py:315- This change addssubscribedtoresult_startso a streamed.subscribereply can still name its subscription. Both SDKs now parse the field, but neither stream assembler copies it into the result it rebuilds. Concrete case:.subscribewith a snapshot over 1 MiB is streamed,ResultStartResponse.subscribedis parsed, and then_assemble_stream(connection.py:315) andassembleStream(packages/inputlayer-js/src/connection.ts:300) return aResultResponsewithoutsubscribed. The caller loses the generation and revision it needs to match later deltas, even though a single-frame reply to the same command carries them. The same assemblers also skip the integrity checks the protocol docs now require (chunk_indexin order from 0, andresult_end.row_count/chunk_countmatching what arrived), so a short or duplicated stream is accepted as complete. The testkit already does these checks (testkit/src/client.rs result_start branch). Fix: passsubscribed=start.subscribed/subscribed: start.subscribed, and check chunk order and end counts in both SDK assemblers.src/protocol/rest/handlers/ws/push.rs:72- The intent requires that 'small-frame delta latency is unchanged'. Before this change every push was serialized inline on the connection loop. Now any delta with more than INLINE_PUSH_ROWS (256) inserted+retracted rows goes tospawn_blocking, whatever its byte size. Example: a delta of 300 single-integer rows is a few KB and fits in one frame, but each push now pays a blocking-pool hop (thread handoff plus wakeup), and under load it queues behind other blocking work. The decision is made on row count, not bytes, so many small-frame deltas take the slower path. The remedy is a choice between latency and keeping work off runtime workers, which is why this is ask-user. Options: gate offloading on a cheap byte estimate, or tryencode_within(FRAME_BUDGET)inline first and only offload the chunking of payloads that don't fit.src/protocol/rest/handlers/ws.rs:758-release_replyruns synchronously in the connection's select loop and now callsstream::result_frameson the.subscribesnapshot. For a large snapshot (up to the result cap) this does a bounded 1 MiB trial encode, a full sizing pass withjson_lenper row, and full chunk serialization, all on a runtime worker. The old code did oneto_stringthere. Program results get the same treatment off-thread via INLINE_FRAME_ROWS/spawn_blocking in execute.rs, but snapshots don't, so a large subscribe can stall other connections scheduled on that worker. Consider encoding large snapshots off the runtime worker, as the execute path does.🔧 Fix applied.
3 issues (2 warnings, 1 info) still open:
src/protocol/rest/handlers/ws/push.rs:72- The intent requires that 'small-frame delta latency is unchanged'. Before this change every push was serialized inline on the connection loop. Now any delta with more than INLINE_PUSH_ROWS (256) inserted+retracted rows goes tospawn_blocking, whatever its byte size. Example: a delta of 300 single-integer rows is a few KB and fits in one frame, but each push now pays a blocking-pool hop (thread handoff plus wakeup), and under load it queues behind other blocking work. The decision is made on row count, not bytes, so many small-frame deltas take the slower path. The remedy is a choice between latency and keeping work off runtime workers, which is why this is ask-user. Options: gate offloading on a cheap byte estimate, or tryencode_within(FRAME_BUDGET)inline first and only offload the chunking of payloads that don't fit.src/protocol/rest/handlers/ws.rs:758-release_replyruns synchronously in the connection's select loop and now callsstream::result_frameson the.subscribesnapshot. For a large snapshot (up to the result cap) this does a bounded 1 MiB trial encode, a full sizing pass withjson_lenper row, and full chunk serialization, all on a runtime worker. The old code did oneto_stringthere. Program results get the same treatment off-thread via INLINE_FRAME_ROWS/spawn_blocking in execute.rs, but snapshots don't, so a large subscribe can stall other connections scheduled on that worker. Consider encoding large snapshots off the runtime worker, as the execute path does.src/protocol/rest/handlers/ws/push.rs:90- Round 1's fix (commit a99ece3) introduced this.estimated_bytestakes the width of the first row only (inserted.first().or(retracted.first())) and multiplies it by the row count. Deltas with variable-width rows are therefore underestimated and serialized inline on the connection loop, even when they span many frames. That breaks the operator's round-1 decision that only payloads fitting one FRAME_BUDGET stay inline and multi-frame payloads go to spawn_blocking. Concrete case: a delta of 10,000 rows on a text column whose first row is[""](estimated ~5 bytes) while the other rows hold 1 KB strings. The estimate is ~50 KB, below the 1 MiB budget, sostream::push_framesruns inline. That does the bounded trial encode, thejson_lensizing pass, and serialization of ~10 MB of chunk frames on a runtime worker, which stalls other connections on that worker. The old row-count gate would have offloaded this delta. Smaller gap:value_widthcountss.len() + 2for strings, but JSON escaping (quotes, backslashes, control chars as\u00XX) can grow a string up to 6x. Fix, still without serializing: sumrow_widthover all inserted and retracted rows, short-circuiting once the total passes FRAME_BUDGET. Optionally count escapable bytes too. The existing tests use uniform rows only, so they pass with the first-row shortcut. Add a test with a short first row followed by wide rows.🔧 Fix applied.
4 issues (2 warnings, 2 infos) still open:
src/protocol/rest/handlers/ws/push.rs:72- The intent requires that 'small-frame delta latency is unchanged'. Before this change every push was serialized inline on the connection loop. Now any delta with more than INLINE_PUSH_ROWS (256) inserted+retracted rows goes tospawn_blocking, whatever its byte size. Example: a delta of 300 single-integer rows is a few KB and fits in one frame, but each push now pays a blocking-pool hop (thread handoff plus wakeup), and under load it queues behind other blocking work. The decision is made on row count, not bytes, so many small-frame deltas take the slower path. The remedy is a choice between latency and keeping work off runtime workers, which is why this is ask-user. Options: gate offloading on a cheap byte estimate, or tryencode_within(FRAME_BUDGET)inline first and only offload the chunking of payloads that don't fit.src/protocol/rest/handlers/ws.rs:758-release_replyruns synchronously in the connection's select loop and now callsstream::result_frameson the.subscribesnapshot. For a large snapshot (up to the result cap) this does a bounded 1 MiB trial encode, a full sizing pass withjson_lenper row, and full chunk serialization, all on a runtime worker. The old code did oneto_stringthere. Program results get the same treatment off-thread via INLINE_FRAME_ROWS/spawn_blocking in execute.rs, but snapshots don't, so a large subscribe can stall other connections scheduled on that worker. Consider encoding large snapshots off the runtime worker, as the execute path does.src/protocol/rest/handlers/ws/push.rs:90- Round 1's fix (commit a99ece3) introduced this.estimated_bytestakes the width of the first row only (inserted.first().or(retracted.first())) and multiplies it by the row count. Deltas with variable-width rows are therefore underestimated and serialized inline on the connection loop, even when they span many frames. That breaks the operator's round-1 decision that only payloads fitting one FRAME_BUDGET stay inline and multi-frame payloads go to spawn_blocking. Concrete case: a delta of 10,000 rows on a text column whose first row is[""](estimated ~5 bytes) while the other rows hold 1 KB strings. The estimate is ~50 KB, below the 1 MiB budget, sostream::push_framesruns inline. That does the bounded trial encode, thejson_lensizing pass, and serialization of ~10 MB of chunk frames on a runtime worker, which stalls other connections on that worker. The old row-count gate would have offloaded this delta. Smaller gap:value_widthcountss.len() + 2for strings, but JSON escaping (quotes, backslashes, control chars as\u00XX) can grow a string up to 6x. Fix, still without serializing: sumrow_widthover all inserted and retracted rows, short-circuiting once the total passes FRAME_BUDGET. Optionally count escapable bytes too. The existing tests use uniform rows only, so they pass with the first-row shortcut. Add a test with a short first row followed by wide rows.src/protocol/rest/handlers/ws/push.rs:25- Round 2's estimator (commit 2b8b9eb) fixes the first-row underestimate:estimated_bytesnow sumsrow_widthover inserted then retracted rows and stops once the total passes FRAME_BUDGET, so the walk is bounded and variable-width rows are counted. One tradeoff is left. Every JSON number counts as NUMBER_BYTES (24), but typical integers serialize to 1–6 bytes. A wide numeric delta that really fits one frame can therefore be estimated over budget and sent to spawn_blocking. Example: 1,500 rows × 30 small-int columns is about 200 KB of JSON but is estimated at about 1.1 MB. This errs toward keeping runtime workers free, and it only affects payloads of hundreds of KB, where serialization cost outweighs the blocking-pool hop. Small deltas, which the latency criterion targets, stay inline: the 300-row test is estimated at ~8 KB. No action needed unless the operator wants the estimate tighter.🔧 Fix applied.
1 info still open:
src/protocol/rest/handlers/ws.rs:758-release_replyruns synchronously in the connection's select loop and now callsstream::result_frameson the.subscribesnapshot. For a large snapshot (up to the result cap) this does a bounded 1 MiB trial encode, a full sizing pass withjson_lenper row, and full chunk serialization, all on a runtime worker. The old code did oneto_stringthere. Program results get the same treatment off-thread via INLINE_FRAME_ROWS/spawn_blocking in execute.rs, but snapshots don't, so a large subscribe can stall other connections scheduled on that worker. Consider encoding large snapshots off the runtime worker, as the execute path does.✅ **Test** - passed
✅ No issues found.
.subscribeon a 3,000-row snapshot streams as result_start (withsubscribed), 6 chunks, result_end row_count=3000. Later deltas use the subscription, and the Python SDK's assembled result keeps `s….subscribeover the cap returns an error, and once back under the cap the next delta is relative to the las…Builttarget/debug/inputlayer-server(CARGO_INCREMENTAL=0,--config profile.dev.package.inputlayer.debug=0because the host disk was full) and ran two isolated instances on ports 18571/18572 with throwaway TOML configs (temp data dir,ws_send_timeout_ms=300, andmax_result_rows=10on the cap instance)python drive_ws.py 18571 large_delta s_delta: ~19 MiB delta, 4,000 rowspython drive_ws.py 18571 large_snapshot s_snap: streamed.subscribereply, raw frames and Python SDKConnection.executepython drive_ws.py 18571 oversized_row s_reset: 18 MiB row leads to subscription_resetpython drive_ws.py 18571 stalled_reader s_stall2: raw-socket client stops reading during a ~40 MiB deltapython stall_lib_with_reader.py/stall_lib_client.py: a websockets-library client as the slow peer (checked that a buffering client gets the full delta and a blocked one is closed)python drive_ws.py 18571 disconnect_mid_stream s_droppython drive_ws.py 18571 small_latency s_lat: 40 commits, each pushing a 1-row and a 300-row deltapython drive_ws.py 18572 result_cap s_capcargo test --test ws_delivery_tests(5 tests, real axum server over TCP)cargo test --lib -- handlers::ws::push handlers::ws::stream handlers::ws::outbound(18 tests, incl. 300-row-inline, wide small-int inline, short-first-row offload)cargo test -p inputlayer-testkit -p inputlayer-ws-protocol(streamed::tests missing/duplicate/foreign chunks, wire shapes)pytest tests/test_connection.py tests/test_protocol.py -k 'stream or subscri or reset or chunk'(Python SDK)npx vitest run tests/statement-errors.test.ts tests/protocol.test.ts(JS SDK)✅ **Document** - passed
✅ No issues found.
gateway/src/provider.rs:11-make doc-check(RUSTDOCFLAGS=-D warnings cargo doc --workspace) fails, but not because of this change. The two errors are a bare URL at gateway/src/provider.rs:11 and a public doc linking to the privateSCANS_BEFORE_INDEXat src/provenance/proof_relations.rs:6, and this change touches neither file. The changed Rust files pass cargo fmt --check, clippy -D warnings (workspace and e2e_reactive) and rustdoc. The JS SDK passestsc --noEmit, and the changed Python files pass ruff check. Pythonruff format/mypy issues in those files existed before this change (58 files repo-wide are unformatted), so they were left alone.✅ **Push** - passed
✅ No issues found.