Skip to content

feat(examples): add Postgres CDC and webhook ingestion recipe with reactive queries - #218

Merged
jsam merged 7 commits into
mainfrom
fm/il-ing1
Oct 3, 2026
Merged

feat(examples): add Postgres CDC and webhook ingestion recipe with reactive queries#218
jsam merged 7 commits into
mainfrom
fm/il-ing1

Conversation

@jsam

@jsam jsam commented Oct 3, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Intent

Ensure that we have perfectly ready streaming KG reworking on which AI agents can subscribe to queries and get results in the reactive mode of working. Always follow clean code principles and clean code architecture. Ensure that the computational real-time aspects, where results come back in milliseconds, stay the same; if locks get added, ensure this does not hurt performance. The performance policy should be done as a pipeline which is always run before a PR commit. Agents subscribing through the engine WebSocket and SDK is what should already be expected in the tests; make sure those expectations are part of the e2e quality checks pipeline. The engine is installed per tenant. Ignore draft PRs. Do not run CI redundantly; it is okay to move quickly and break things, then run QC at certain checkpoints. Call them release checkpoints, where the entire CI and QC verification runs. To move quickly, run and test things locally, make sure the relevant checks run in parallel before making the commit, and only run the checks that make sense, not everything.

What Changed

  • Adds examples/ingestion/, a runnable Docker Compose stack (Postgres, Debezium Server, a Python adapter and InputLayer) with demo.sh. The script generates credentials and ports bound to loopback, then runs a self-checking demo: Postgres row changes and signed billing webhooks drive the needs_attention rule, and a client subscribed over WebSocket receives every insert and retraction delta.
  • The adapter (adapter/ingest/) does the following:
    • Verifies Standard Webhooks signatures on /cdc and /webhooks/<source>.
    • Maps tables and webhook events to relations through mappings.py.
    • Applies changes with engine-side conditional updates guarded by revisions, so duplicate, replayed and out-of-order deliveries are safe. The applied and skipped counters come from committed updates.
    • Bounds engine requests with a configurable ENGINE_REQUEST_TIMEOUT and returns 503 on expiry.
    • Refuses engines that do not speak WebSocket protocol 2.
    • Includes tests for mapping, applier, engine protocol, startup and the demo.
  • Adds the guides/ingestion.mdx docs guide and its nav entry. The guide covers the walkthrough, ordering and replay guarantees, acknowledgment semantics, signatures, timeouts and production notes.

Risk Assessment

✅ Low: The change is confined to the authorized ingestion recipe and documentation, with no substantiated material defects after tracing authentication, conditional commits, retries, counters, and watcher failures.

Testing

I built the engine from this worktree and ran the adapter's engine-backed tests: all 95 passed, none skipped. I also ran the real Docker demo on an isolated stack. CDC and webhook changes reached the watching client as exact deltas in tens of milliseconds, and duplicates and late webhooks were skipped. Adversarial checks also passed: bad signatures got 401, a paused engine got 503 at the 10s deadline, and the adapter recovered with a safe skipped retry. The default published engine image speaks WebSocket protocol 1, so the adapter refuses it. I report that as an informational note. Everything here is CLI or service behaviour, so there is no visual evidence. The stack and caches were torn down.

  • Live validation: ✅ go - 7 of 7 scenarios driven live against the product
Scenario Result Live Evidence
Operator runs ./demo.sh; Postgres row changes and billing webhooks update the derived answer, and the watching client receives each delta in milliseconds ✅ pass live docker-e2e-demo.log: all 9 steps matched, deltas in 33-62 ms, exit 0 (engine image inputlayer-ing1-main:local)
A duplicate webhook and a late, lower-sequence webhook are skipped, with no spurious delta and a correct applied/skipped count ✅ pass live docker-e2e-demo.log steps 2 and 9: {'applied': 0, 'skipped': 1}; watcher quiet
No-op Postgres UPDATE and delete/rename of rows produce exactly the right retract/insert deltas ✅ pass live docker-e2e-demo.log steps 5-7
Adversarial: forged, missing, stale or tampered webhook signatures are rejected ✅ pass live adversarial-live.log: all four returned 401
Adversarial: engine stalls mid-request; the adapter replies 503 at the configured deadline instead of hanging, then recovers, and a retry of the same revision is skipped ✅ pass live adversarial-live.log: 503 after 10.02s while paused; 200 applied=1 after unpause; retry gives applied=0 skipped=1
Replay safety under an interrupted connection, with counters derived from committed conditional outcomes (in-flight newer revision) ✅ pass live adapter-pytest.log: test_disconnected_write_preserves_revision_order_and_committed_counts (32 cases) and test_batch_replay_delete_and_recreate passed against the real inputlayer-server binary
Adapter refuses an engine that speaks WebSocket protocol 1, with a clear error ✅ pass live docker-e2e-demo-ghcr-latest-protocol1.log and adapter log: 'engine speaks WebSocket protocol 1; this client needs 2 or later'
Evidence: Docker e2e demo transcript (CDC and webhooks to standing-query deltas) 
time="2026-10-03T22:09:02Z" level=warning msg="Docker Compose is configured to build using Bake, but buildx isn't installed"
#0 building with "default" instance using docker driver

#1 [adapter internal] load build definition from Dockerfile
#1 transferring dockerfile: 383B done
#1 DONE 0.0s

#2 [adapter internal] load metadata for docker.io/library/python:3.12-slim
#2 DONE 0.4s

#3 [adapter internal] load .dockerignore
#3 transferring context: 101B done
#3 DONE 0.0s

#4 [adapter 1/5] FROM docker.io/library/python:3.12-slim@sha256:dddfd7e07f9d15aeeca61529320492139d21cac7f0070c00609243e51e4e0016
#4 resolve docker.io/library/python:3.12-slim@sha256:dddfd7e07f9d15aeeca61529320492139d21cac7f0070c00609243e51e4e0016 0.0s done
#4 DONE 0.0s

#5 [adapter internal] load build context
#5 transferring context: 1.41kB done
#5 DONE 0.0s

#6 [adapter 4/5] COPY ingest/ ingest/
#6 CACHED

#7 [adapter 2/5] WORKDIR /app
#7 CACHED

#8 [adapter 3/5] COPY pyproject.toml ./
#8 CACHED

#9 [adapter 5/5] RUN pip install --no-cache-dir ".[demo]"
#9 CACHED

#10 [adapter] exporting to image
#10 exporting layers done
#10 exporting manifest sha256:faf31b40c482a72a5f0a7a14a165c44d9b79f699c5698ca605bdfc41b056e2f9 done
#10 exporting config sha256:23d4982e093581ef7ead23a45141f5965168d27b291eda187685648670281e8f done
#10 exporting attestation manifest sha256:b04898014f3734a7d09826a1fbe6e3e87745bcbfac12dac8d965c22aa191fe75 0.0s done
#10 exporting manifest list sha256:da2ce23e0c58e1d4e9353aa1be78627efbea703ff778570477b417a3a355206f done
#10 naming to docker.io/library/inputlayer-ingest-adapter:latest done
#10 unpacking to docker.io/library/inputlayer-ingest-adapter:latest done
#10 DONE 0.1s

#11 [adapter] resolving provenance for metadata file
#11 DONE 0.0s
 adapter  Built
 Network ilingtest_default  Creating
 Network ilingtest_default  Created
 Volume ilingtest_debezium-data  Creating
 Volume ilingtest_debezium-data  Created
 Volume ilingtest_inputlayer-data  Creating
 Volume ilingtest_inputlayer-data  Created
 Container ilingtest-postgres-1  Creating
 Container ilingtest-inputlayer-1  Creating
 Container ilingtest-inputlayer-1  Created
 Container ilingtest-adapter-1  Creating
 Container ilingtest-postgres-1  Created
 Container ilingtest-adapter-1  Created
 Container ilingtest-debezium-1  Creating
 Container ilingtest-debezium-1  Created
 Container ilingtest-postgres-1  Starting
 Container ilingtest-inputlayer-1  Starting
 Container ilingtest-postgres-1  Started
 Container ilingtest-inputlayer-1  Started
 Container ilingtest-inputlayer-1  Waiting
 Container ilingtest-inputlayer-1  Healthy
 Container ilingtest-adapter-1  Starting
 Container ilingtest-adapter-1  Started
 Container ilingtest-postgres-1  Waiting
 Container ilingtest-adapter-1  Waiting
 Container ilingtest-postgres-1  Healthy
 Container ilingtest-adapter-1  Healthy
 Container ilingtest-debezium-1  Starting
 Container ilingtest-debezium-1  Started
 Container ilingtest-inputlayer-1  Waiting
 Container ilingtest-postgres-1  Waiting
 Container ilingtest-adapter-1  Waiting
 Container ilingtest-debezium-1  Waiting
 Container ilingtest-postgres-1  Healthy
 Container ilingtest-adapter-1  Healthy
 Container ilingtest-inputlayer-1  Healthy
 Container ilingtest-debezium-1  Healthy
loading rules and waiting for the Postgres snapshot ...
watching ?needs_attention(Id, Name, Invoice); snapshot: []

[1] billing webhook: payment.failed inv_100 for Acme (enterprise)
    adapter: {'applied': 1, 'skipped': 0}
    watcher: seq=1 + [1, 'Acme', 'inv_100']  (33 ms)

[2] the same webhook delivered again (a SaaS retry)
    adapter: {'applied': 0, 'skipped': 1}
    watcher: no delta within 1.5s, as expected

[3] billing webhook: payment.failed inv_200 for Globex (free tier)
    adapter: {'applied': 1, 'skipped': 0}
    watcher: no delta within 1.5s, as expected

[4] Postgres: UPDATE customers SET tier = 'enterprise' WHERE id = 2
    watcher: seq=2 + [2, 'Globex', 'inv_200']  (62 ms)

[5] Postgres: UPDATE customers SET tier = tier WHERE id = 1 (no-op)
    watcher: no delta within 1.5s, as expected

[6] Postgres: UPDATE customers SET name = 'Globex Corp' WHERE id = 2
    watcher: seq=3 - [2, 'Globex', 'inv_200']  (48 ms)
    watcher: seq=3 + [2, 'Globex Corp', 'inv_200']  (48 ms)

[7] Postgres: DELETE FROM customers WHERE id = 1
    watcher: seq=4 - [1, 'Acme', 'inv_100']  (62 ms)

[8] billing webhook: payment.succeeded inv_200 (sequence 3)
    adapter: {'applied': 1, 'skipped': 0}
    watcher: seq=5 - [2, 'Globex Corp', 'inv_200']  (47 ms)

[9] late webhook: payment.failed inv_200 (sequence 2) arrives after it
    adapter: {'applied': 0, 'skipped': 1}
    watcher: no delta within 1.5s, as expected
    watcher: no delta within 1.5s, as expected

all deltas matched: CDC and webhooks drive the standing query.
Evidence: Live adversarial checks: signatures, engine stall deadline, recovery, retry 
== signature guard
forged signature -> ((401, 'missing, stale or bad webhook signature'), 0.02)
no signature     -> ((401, 'missing, stale or bad webhook signature'), 0.0)
stale timestamp  -> ((401, 'missing, stale or bad webhook signature'), 0.0)
tampered body    -> ((401, 'missing, stale or bad webhook signature'), 0.0)
== engine stalled (docker pause inputlayer), ENGINE_REQUEST_TIMEOUT=10
ilingtest-inputlayer-1
valid seq=1     -> ((503, 'engine request timed out after 10s'), 10.02)
valid seq=1     -> ((503, 'cannot connect to ws://inputlayer:8080/ws?kg=shop: '), 10.02)
== engine resumed
ilingtest-inputlayer-1
valid seq=1     -> ((200, '{"applied": 1, "skipped": 0}'), 0.02)
== sender retry of the same revision after recovery
valid seq=1     -> ((200, '{"applied": 0, "skipped": 1}'), 0.02)
== adapter log
2026-10-03 22:09:25,308 ingest.applier applied=1 skipped=0 in 5.3 ms
2026-10-03 22:09:25,370 ingest.applier applied=1 skipped=0 in 5.7 ms
2026-10-03 22:09:25,417 ingest.applier applied=1 skipped=0 in 45.0 ms
2026-10-03 22:09:25,460 ingest.applier applied=0 skipped=1 in 40.8 ms
2026-10-03 22:09:57,603 __main__ engine unavailable for /webhooks/billing: engine request timed out after 10s
2026-10-03 22:10:07,668 __main__ engine unavailable for /webhooks/billing: cannot connect to ws://inputlayer:8080/ws?kg=shop: 
2026-10-03 22:10:08,780 ingest.applier applied=1 skipped=0 in 6.4 ms
2026-10-03 22:10:08,847 ingest.applier applied=0 skipped=1 in 1.0 ms
Evidence: Adapter engine-backed pytest run 
============================= test session starts ==============================
platform linux -- Python 3.12.3, pytest-9.1.1, pluggy-1.6.0 -- /tmp/il-ing1-venv/bin/python
rootdir: ~/.no-mistakes/worktrees/9eea230c3aae/01M41QR6R67KDDMZKWKPT67D5E/examples/ingestion/adapter
configfile: pyproject.toml
testpaths: tests
plugins: asyncio-1.4.0
asyncio: mode=Mode.AUTO, debug=False, asyncio_default_fixture_loop_scope=None, asyncio_default_test_loop_scope=function
collecting ... collected 95 items

tests/test_applier.py::test_disconnected_write_preserves_revision_order_and_committed_counts[10-False-False-False-relation0-key0-old_row0-new_row0] PASSED [  1%]
tests/test_applier.py::test_disconnected_write_preserves_revision_order_and_committed_counts[10-False-False-False-relation1-key1-old_row1-new_row1] PASSED [  2%]
tests/test_applier.py::test_disconnected_write_preserves_revision_order_and_committed_counts[10-False-False-True-relation0-key0-old_row0-new_row0] PASSED [  3%]
tests/test_applier.py::test_disconnected_write_preserves_revision_order_and_committed_counts[10-False-False-True-relation1-key1-old_row1-new_row1] PASSED [  4%]
tests/test_applier.py::test_disconnected_write_preserves_revision_order_and_committed_counts[10-False-True-False-relation0-key0-old_row0-new_row0] PASSED [  5%]
tests/test_applier.py::test_disconnected_write_preserves_revision_order_and_committed_counts[10-False-True-False-relation1-key1-old_row1-new_row1] PASSED [  6%]
tests/test_applier.py::test_disconnected_write_preserves_revision_order_and_committed_counts[10-False-True-True-relation0-key0-old_row0-new_row0] PASSED [  7%]
tests/test_applier.py::test_disconnected_write_preserves_revision_order_and_committed_counts[10-False-True-True-relation1-key1-old_row1-new_row1] PASSED [  8%]
tests/test_applier.py::test_disconnected_write_preserves_revision_order_and_committed_counts[10-True-False-False-relation0-key0-old_row0-new_row0] PASSED [  9%]
tests/test_applier.py::test_disconnected_write_preserves_revision_order_and_committed_counts[10-True-False-False-relation1-key1-old_row1-new_row1] PASSED [ 10%]
tests/test_applier.py::test_disconnected_write_preserves_revision_order_and_committed_counts[10-True-False-True-relation0-key0-old_row0-new_row0] PASSED [ 11%]
tests/test_applier.py::test_disconnected_write_preserves_revision_order_and_committed_counts[10-True-False-True-relation1-key1-old_row1-new_row1] PASSED [ 12%]
tests/test_applier.py::test_disconnected_write_preserves_revision_order_and_committed_counts[10-True-True-False-relation0-key0-old_row0-new_row0] PASSED [ 13%]
tests/test_applier.py::test_disconnected_write_preserves_revision_order_and_committed_counts[10-True-True-False-relation1-key1-old_row1-new_row1] PASSED [ 14%]
tests/test_applier.py::test_disconnected_write_preserves_revision_order_and_committed_counts[10-True-True-True-relation0-key0-old_row0-new_row0] PASSED [ 15%]
tests/test_applier.py::test_disconnected_write_preserves_revision_order_and_committed_counts[10-True-True-True-relation1-key1-old_row1-new_row1] PASSED [ 16%]
tests/test_applier.py::test_disconnected_write_preserves_revision_order_and_committed_counts[12-False-False-False-relation0-key0-old_row0-new_row0] PASSED [ 17%]
tests/test_applier.py::test_disconnected_write_preserves_revision_order_and_committed_counts[12-False-False-False-relation1-key1-old_row1-new_row1] PASSED [ 18%]
tests/test_applier.py::test_disconnected_write_preserves_revision_order_and_committed_counts[12-False-False-True-relation0-key0-old_row0-new_row0] PASSED [ 20%]
tests/test_applier.py::test_disconnected_write_preserves_revision_order_and_committed_counts[12-False-False-True-relation1-key1-old_row1-new_row1] PASSED [ 21%]
tests/test_applier.py::test_disconnected_write_preserves_revision_order_and_committed_counts[12-False-True-False-relation0-key0-old_row0-new_row0] PASSED [ 22%]
tests/test_applier.py::test_disconnected_write_preserves_revision_order_and_committed_counts[12-False-True-False-relation1-key1-old_row1-new_row1] PASSED [ 23%]
tests/test_applier.py::test_disconnected_write_preserves_revision_order_and_committed_counts[12-False-True-True-relation0-key0-old_row0-new_row0] PASSED [ 24%]
tests/test_applier.py::test_disconnected_write_preserves_revision_order_and_committed_counts[12-False-True-True-relation1-key1-old_row1-new_row1] PASSED [ 25%]
tests/test_applier.py::test_disconnected_write_preserves_revision_order_and_committed_counts[12-True-False-False-relation0-key0-old_row0-new_row0] PASSED [ 26%]
tests/test_applier.py::test_disconnected_write_preserves_revision_order_and_committed_counts[12-True-False-False-relation1-key1-old_row1-new_row1] PASSED [ 27%]
tests/test_applier.py::test_disconnected_write_preserves_revision_order_and_committed_counts[12-True-False-True-relation0-key0-old_row0-new_row0] PASSED [ 28%]
tests/test_applier.py::test_disconnected_write_preserves_revision_order_and_committed_counts[12-True-False-True-relation1-key1-old_row1-new_row1] PASSED [ 29%]
tests/test_applier.py::test_disconnected_write_preserves_revision_order_and_committed_counts[12-True-True-False-relation0-key0-old_row0-new_row0] PASSED [ 30%]
tests/test_applier.py::test_disconnected_write_preserves_revision_order_and_committed_counts[12-True-True-False-relation1-key1-old_row1-new_row1] PASSED [ 31%]
tests/test_applier.py::test_disconnected_write_preserves_revision_order_and_committed_counts[12-True-True-True-relation0-key0-old_row0-new_row0] PASSED [ 32%]
tests/test_applier.py::test_disconnected_write_preserves_revision_order_and_committed_counts[12-True-True-True-relation1-key1-old_row1-new_row1] PASSED [ 33%]
tests/test_applier.py::test_batch_replay_delete_and_recreate PASSED      [ 34%]
tests/test_demo.py::test_self_check_observes_collector_termination[None-expected] PASSED [ 35%]
tests/test_demo.py::test_self_check_observes_collector_termination[None-quiet] PASSED [ 36%]
tests/test_demo.py::test_self_check_observes_collector_termination[None-final] PASSED [ 37%]
tests/test_demo.py::test_self_check_observes_collector_termination[failure1-expected] PASSED [ 38%]
tests/test_demo.py::test_self_check_observes_collector_termination[failure1-quiet] PASSED [ 40%]
tests/test_demo.py::test_self_check_observes_collector_termination[failure1-final] PASSED [ 41%]
tests/test_demo.py::test_self_check_observes_collector_termination[failure2-expected] PASSED [ 42%]
tests/test_demo.py::test_self_check_observes_collector_termination[failure2-quiet] PASSED [ 43%]
tests/test_demo.py::test_self_check_observes_collector_termination[failure2-final] PASSED [ 44%]
tests/test_demo.py::test_self_check_awaits_collector_shutdown PASSED     [ 45%]
tests/test_engine_protocol.py::test_notices_and_pushes_before_a_reply_are_set_aside PASSED [ 46%]
tests/test_engine_protocol.py::test_statement_errors_raise PASSED        [ 47%]
tests/test_engine_protocol.py::test_protocol_1_engines_are_refused PASSED [ 48%]
tests/test_engine_protocol.py::test_watcher_drops_stale_generations_and_raises_on_a_lost_delta PASSED [ 49%]
tests/test_engine_protocol.py::test_silent_engine_times_out_and_closes_connection[authentication] PASSED [ 50%]
tests/test_engine_protocol.py::test_silent_engine_times_out_and_closes_connection[execution] PASSED [ 51%]
tests/test_engine_protocol.py::test_silent_engine_times_out_and_closes_connection[stream] PASSED [ 52%]
tests/test_engine_protocol.py::test_stream_uses_one_deadline_through_final_frame[0.1] PASSED [ 53%]
tests/test_engine_protocol.py::test_stream_uses_one_deadline_through_final_frame[1.0] PASSED [ 54%]
tests/test_engine_protocol.py::test_deadline_includes_sending PASSED     [ 55%]
tests/test_engine_protocol.py::test_startup_retries_after_timeout[authentication] PASSED [ 56%]
tests/test_engine_protocol.py::test_startup_retries_after_timeout[declaration] PASSED [ 57%]
tests/test_engine_protocol.py::test_http_timeout_releases_lock_and_reconnects[revision-cdc] PASSED [ 58%]
tests/test_engine_protocol.py::test_http_timeout_releases_lock_and_reconnects[revision-billing] PASSED [ 60%]
tests/test_engine_protocol.py::test_http_timeout_releases_lock_and_reconnects[write-cdc] PASSED [ 61%]
tests/test_engine_protocol.py::test_http_timeout_releases_lock_and_reconnects[write-billing] PASSED [ 62%]
tests/test_engine_protocol.py::test_request_timeout_environment_and_default PASSED [ 63%]
tests/test_engine_protocol.py::test_invalid_request_deadlines_are_rejected[0] PASSED [ 64%]
tests/test_engine_protocol.py::test_invalid_request_deadlines_are_rejected[-1] PASSED [ 65%]
tests/test_engine_protocol.py::test_invalid_request_deadlines_are_rejected[nan] PASSED [ 66%]
tests/test_engine_protocol.py::test_invalid_request_deadlines_are_rejected[inf] PASSED [ 67%]
tests/test_engine_protocol.py::test_invalid_request_deadlines_are_rejected[-inf] PASSED [ 68%]
tests/test_engine_protocol.py::test_invalid_request_deadlines_are_rejected[invalid] PASSED [ 69%]
tests/test_engine_protocol.py::test_idle_subscription_wait_is_not_a_request_timeout PASSED [ 70%]
tests/test_mapping.py::test_debezium_upserts_and_deletes_by_key PASSED   [ 71%]
tests/test_mapping.py::test_debezium_accepts_lists_schema_wrapped_events_and_tombstones PASSED [ 72%]
tests/test_mapping.py::test_debezium_rejects_what_it_cannot_map[event0] PASSED [ 73%]
tests/test_mapping.py::test_debezium_rejects_what_it_cannot_map[event1] PASSED [ 74%]
tests/test_mapping.py::test_debezium_rejects_what_it_cannot_map[event2] PASSED [ 75%]
tests/test_mapping.py::test_debezium_rejects_what_it_cannot_map[event3] PASSED [ 76%]
tests/test_mapping.py::test_debezium_rejects_what_it_cannot_map[event4] PASSED [ 77%]
tests/test_mapping.py::test_debezium_rejects_what_it_cannot_map[event5] PASSED [ 78%]
tests/test_mapping.py::test_debezium_rejects_what_it_cannot_map[not an object] PASSED [ 80%]
tests/test_mapping.py::test_webhook_upsert_retract_and_unmapped_types PASSED [ 81%]
tests/test_mapping.py::test_webhook_requires_an_integer_revision PASSED  [ 82%]
tests/test_mapping.py::test_signatures_reject_tampering_staleness_and_other_secrets PASSED [ 83%]
tests/test_mapping.py::test_latest_per_key_keeps_highest_revision PASSED [ 84%]
tests/test_mapping.py::test_schema_program_declares_owned_relations_and_revisions PASSED [ 85%]
tests/test_mapping.py::test_literals PASSED                              [ 86%]
tests/test_mapping.py::test_relation_rejects_non_finite_floats PASSED    [ 87%]
tests/test_mapping.py::test_relation_validates_its_key PASSED            [ 88%]
tests/test_mapping.py::test_debezium_rejects_unavailable_mapped_values[r-id] PASSED [ 89%]
tests/test_mapping.py::test_debezium_rejects_unavailable_mapped_values[r-name] PASSED [ 90%]
tests/test_mapping.py::test_debezium_rejects_unavailable_mapped_values[r-tier] PASSED [ 91%]
tests/test_mapping.py::test_debezium_rejects_unavailable_mapped_values[c-id] PASSED [ 92%]
tests/test_mapping.py::test_debezium_rejects_unavailable_mapped_values[c-name] PASSED [ 93%]
tests/test_mapping.py::test_debezium_rejects_unavailable_mapped_values[c-tier] PASSED [ 94%]
tests/test_mapping.py::test_debezium_rejects_unavailable_mapped_values[u-id] PASSED [ 95%]
tests/test_mapping.py::test_debezium_rejects_unavailable_mapped_values[u-name] PASSED [ 96%]
tests/test_mapping.py::test_debezium_rejects_unavailable_mapped_values[u-tier] PASSED [ 97%]
tests/test_mapping.py::test_debezium_delete_rejects_unavailable_key PASSED [ 98%]
tests/test_startup.py::test_startup_generates_private_credentials_and_loopback_ports PASSED [100%]

============================= 95 passed in 25.12s ==============================
Evidence: demo.sh with default ghcr latest image (protocol 1 refused) 
time="2026-10-03T22:07:16Z" level=warning msg="Docker Compose is configured to build using Bake, but buildx isn't installed"
#0 building with "default" instance using docker driver

#1 [adapter internal] load build definition from Dockerfile
#1 transferring dockerfile: 383B done
#1 DONE 0.0s

#2 [adapter internal] load metadata for docker.io/library/python:3.12-slim
#2 DONE 0.6s

#3 [adapter internal] load .dockerignore
#3 transferring context: 101B done
#3 DONE 0.0s

#4 [adapter 1/5] FROM docker.io/library/python:3.12-slim@sha256:dddfd7e07f9d15aeeca61529320492139d21cac7f0070c00609243e51e4e0016
#4 resolve docker.io/library/python:3.12-slim@sha256:dddfd7e07f9d15aeeca61529320492139d21cac7f0070c00609243e51e4e0016 0.0s done
#4 DONE 0.0s

#5 [adapter internal] load build context
#5 transferring context: 131.96kB done
#5 DONE 0.0s

#6 [adapter 2/5] WORKDIR /app
#6 CACHED

#7 [adapter 3/5] COPY pyproject.toml ./
#7 CACHED

#8 [adapter 4/5] COPY ingest/ ingest/
#8 DONE 0.2s

#9 [adapter 5/5] RUN pip install --no-cache-dir ".[demo]"
#9 2.020 Processing /app
#9 2.022   Installing build dependencies: started
#9 2.843   Installing build dependencies: finished with status 'done'
#9 2.844   Getting requirements to build wheel: started
#9 2.917   Getting requirements to build wheel: finished with status 'done'
#9 2.918   Preparing metadata (pyproject.toml): started
#9 3.052   Preparing metadata (pyproject.toml): finished with status 'done'
#9 3.448 Collecting aiohttp<4,>=3.9 (from inputlayer-ingest-example==0.1.0)
#9 3.464   Downloading aiohttp-3.14.3-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl.metadata (8.3 kB)
#9 3.481 Collecting psycopg<4,>=3.1 (from psycopg[binary]<4,>=3.1; extra == "demo"->inputlayer-ingest-example==0.1.0)
#9 3.485   Downloading psycopg-3.3.6-py3-none-any.whl.metadata (4.4 kB)
#9 3.499 Collecting aiohappyeyeballs>=2.5.0 (from aiohttp<4,>=3.9->inputlayer-ingest-example==0.1.0)
#9 3.503   Downloading aiohappyeyeballs-2.7.1-py3-none-any.whl.metadata (5.9 kB)
#9 3.509 Collecting aiosignal>=1.4.0 (from aiohttp<4,>=3.9->inputlayer-ingest-example==0.1.0)
#9 3.515   Downloading aiosignal-1.4.0-py3-none-any.whl.metadata (3.7 kB)
#9 3.524 Collecting attrs>=17.3.0 (from aiohttp<4,>=3.9->inputlayer-ingest-example==0.1.0)
#9 3.527   Downloading attrs-26.1.0-py3-none-any.whl.metadata (8.8 kB)
#9 3.566 Collecting frozenlist>=1.1.1 (from aiohttp<4,>=3.9->inputlayer-ingest-example==0.1.0)
#9 3.569   Downloading frozenlist-1.8.0-cp312-cp312-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl.metadata (20 kB)
#9 3.731 Collecting multidict<7.0,>=4.5 (from aiohttp<4,>=3.9->inputlayer-ingest-example==0.1.0)
#9 3.735   Downloading multidict-6.9.1-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl.metadata (5.3 kB)
#9 3.778 Collecting propcache>=0.2.0 (from aiohttp<4,>=3.9->inputlayer-ingest-example==0.1.0)
#9 3.782   Downloading propcache-0.5.4-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl.metadata (25 kB)
#9 3.795 Collecting typing_extensions>=4.4 (from aiohttp<4,>=3.9->inputlayer-ingest-example==0.1.0)
#9 3.799   Downloading typing_extensions-4.16.0-py3-none-any.whl.metadata (3.3 kB)
#9 3.976 Collecting yarl<2.0,>=1.17.0 (from aiohttp<4,>=3.9->inputlayer-ingest-example==0.1.0)
#9 3.979   Downloading yarl-1.25.1-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl.metadata (111 kB)
#9 4.091 Collecting psycopg-binary==3.3.6 (from psycopg[binary]<4,>=3.1; extra == "demo"->inputlayer-ingest-example==0.1.0)
#9 4.094   Downloading psycopg_binary-3.3.6-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl.metadata (2.7 kB)
#9 4.112 Collecting idna>=2.0 (from yarl<2.0,>=1.17.0->aiohttp<4,>=3.9->inputlayer-ingest-example==0.1.0)
#9 4.115   Downloading idna-3.20-py3-none-any.whl.metadata (7.2 kB)
#9 4.130 Downloading aiohttp-3.14.3-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl (1.8 MB)
#9 4.137    ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.8/1.8 MB 503.4 MB/s eta 0:00:00
#9 4.141 Downloading psycopg-3.3.6-py3-none-any.whl (215 kB)
#9 4.150 Downloading psycopg_binary-3.3.6-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (5.2 MB)
#9 4.164    ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 5.2/5.2 MB 436.8 MB/s eta 0:00:00
#9 4.168 Downloading aiohappyeyeballs-2.7.1-py3-none-any.whl (15 kB)
#9 4.171 Downloading aiosignal-1.4.0-py3-none-any.whl (7.5 kB)
#9 4.174 Downloading attrs-26.1.0-py3-none-any.whl (67 kB)
#9 4.177 Downloading frozenlist-1.8.0-cp312-cp312-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl (242 kB)
#9 4.180 Downloading multidict-6.9.1-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl (336 kB)
#9 4.184 Downloading propcache-0.5.4-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl (250 kB)
#9 4.187 Downloading typing_extensions-4.16.0-py3-none-any.whl (45 kB)
#9 4.190 Downloading yarl-1.25.1-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl (117 kB)
#9 4.193 Downloading idna-3.20-py3-none-any.whl (69 kB)
#9 4.208 Building wheels for collected packages: inputlayer-ingest-example
#9 4.209   Building wheel for inputlayer-ingest-example (pyproject.toml): started
#9 4.237   Building wheel for inputlayer-ingest-example (pyproject.toml): finished with status 'done'
#9 4.238   Created wheel for inputlayer-ingest-example: filename=inputlayer_ingest_example-0.1.0-py3-none-any.whl size=21123 sha256=f4c9057d90163103f7726e38ec90c4b733b9b7629a29818b47bec573587b5dad
#9 4.238   Stored in directory: /tmp/pip-ephem-wheel-cache-j0k32wue/wheels/54/1b/b7/aa63e25c8f14f4f2ae7b04e6097bdecb770e455c5c1ee0a600
#9 4.240 Successfully built inputlayer-ingest-example
#9 4.261 Installing collected packages: typing_extensions, psycopg-binary, propcache, multidict, idna, frozenlist, attrs, aiohappyeyeballs, yarl, psycopg, aiosignal, aiohttp, inputlayer-ingest-example
#9 4.850 Successfully installed aiohappyeyeballs-2.7.1 aiohttp-3.14.3 aiosignal-1.4.0 attrs-26.1.0 frozenlist-1.8.0 idna-3.20 inputlayer-ingest-example-0.1.0 multidict-6.9.1 propcache-0.5.4 psycopg-3.3.6 psycopg-binary-3.3.6 typing_extensions-4.16.0 yarl-1.25.1
#9 4.851 WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.
#9 4.902 
#9 4.902 [notice] A new release of pip is available: 25.0.1 -> 26.2.1
#9 4.902 [notice] To update, run: pip install --upgrade pip
#9 DONE 5.1s

#10 [adapter] exporting to image
#10 exporting layers
#10 exporting layers 1.5s done
#10 exporting manifest sha256:faf31b40c482a72a5f0a7a14a165c44d9b79f699c5698ca605bdfc41b056e2f9 done
#10 exporting config sha256:23d4982e093581ef7ead23a45141f5965168d27b291eda187685648670281e8f 0.0s done
#10 exporting attestation manifest sha256:745b76f336f052ca2f3b2ce4b562bdbe6dc7b823483a921190f5a7aab38cb208 0.0s done
#10 exporting manifest list sha256:23ea2f42c6710534be0b4f013f9ebd92e818409bd5a0291364ca89390621acce 0.0s done
#10 naming to docker.io/library/inputlayer-ingest-adapter:latest done
#10 unpacking to docker.io/library/inputlayer-ingest-adapter:latest
#10 unpacking to docker.io/library/inputlayer-ingest-adapter:latest 0.4s done
#10 DONE 1.9s

#11 [adapter] resolving provenance for metadata file
#11 DONE 0.0s
 adapter  Built
 Network ilingtest_default  Creating
 Network ilingtest_default  Created
 Volume ilingtest_inputlayer-data  Creating
 Volume ilingtest_inputlayer-data  Created
 Volume ilingtest_debezium-data  Creating
 Volume ilingtest_debezium-data  Created
 Container ilingtest-postgres-1  Creating
 Container ilingtest-inputlayer-1  Creating
 Container ilingtest-postgres-1  Created
 Container ilingtest-inputlayer-1  Created
 Container ilingtest-adapter-1  Creating
 Container ilingtest-adapter-1  Created
 Container ilingtest-debezium-1  Creating
 Container ilingtest-debezium-1  Created
 Container ilingtest-postgres-1  Starting
 Container ilingtest-inputlayer-1  Starting
 Container ilingtest-inputlayer-1  Started
 Container ilingtest-inputlayer-1  Waiting
 Container ilingtest-postgres-1  Started
 Container ilingtest-inputlayer-1  Healthy
 Container ilingtest-adapter-1  Starting
 Container ilingtest-adapter-1  Started
 Container ilingtest-adapter-1  Waiting
 Container ilingtest-postgres-1  Waiting
 Container ilingtest-postgres-1  Healthy
 Container ilingtest-adapter-1  Error
dependency failed to start: container ilingtest-adapter-1 is unhealthy
Evidence: Demo excerpt 
[1] billing webhook: payment.failed inv_100 for Acme (enterprise)
adapter: {'applied': 1, 'skipped': 0}
watcher: seq=1 + [1, 'Acme', 'inv_100'] (33 ms)
[2] the same webhook delivered again (a SaaS retry)
adapter: {'applied': 0, 'skipped': 1}
[6] Postgres: UPDATE customers SET name = 'Globex Corp' WHERE id = 2
watcher: seq=3 - [2, 'Globex', 'inv_200'] (48 ms)
watcher: seq=3 + [2, 'Globex Corp', 'inv_200'] (48 ms)
[9] late webhook: payment.failed inv_200 (sequence 2) arrives after it
adapter: {'applied': 0, 'skipped': 1}
all deltas matched: CDC and webhooks drive the standing query.
Evidence: Engine stall excerpt 
valid seq=1 -> ((503, 'engine request timed out after 10s'), 10.02)
== engine resumed
valid seq=1 -> ((200, '{"applied": 1, "skipped": 0}'), 0.02)
== sender retry of the same revision after recovery
valid seq=1 -> ((200, '{"applied": 0, "skipped": 1}'), 0.02)
- Outcome: ⚠️ 1 info across 2 runs (16m33s)

Pipeline

Updates from git push no-mistakes

✅ **intent** - passed

✅ No issues found.

✅ **Rebase** - passed

✅ No issues found.

🔧 **Review** - 8 issues found → auto-fixed (3) ✅
  • 🚨 examples/ingestion/adapter/ingest/demo.py:177 - The required SDK subscription path remains absent. The intent requires agents subscribing through “the engine WebSocket and SDK” in the “e2e quality checks pipeline,” but the added Watcher.open(...) uses a separate example client. Public SDKs still discard subscription pushes (packages/inputlayer-js/src/connection.ts:149; packages/inputlayer-py/src/inputlayer/connection.py:338), and TESTING.md explicitly excludes SDK agents. Neither the new demo nor its tests is included in the release-checkpoint pipeline. Implement and exercise the public SDK path through that pipeline.
  • 🚨 examples/ingestion/adapter/ingest/applier.py:50 - Replay safety fails across an interrupted connection: revision 10's write can remain executing on the engine after the adapter detects disconnection; a reconnected request commits revision 11; the older write then commits unconditionally, restoring revision 10 and stale data. The engine awaits execution before reading socket closure. The invariant must cover applier.py:50 (separate revision read), applier.py:77 (reconnect), and iql.py:60–66 (data deletion/insertion and watermark replacement), for both CDC and webhooks. The supported boundary is the engine's atomic program commit, including revision validation. This defect belongs to the unrequired ingestion subsystem identified in R8; removing that subsystem is the smallest scope-compliant remedy.
  • 🚨 examples/ingestion/adapter/ingest/debezium.py:55 - With the configured default replica identity, updating the tier of a customer whose unchanged name is stored out-of-line produces Debezium's unavailable-value placeholder. row_from accepts that string, and the adapter overwrites the real name, advances the revision and returns 200. Both mapped text fields, name and tier, lack handling for unavailable values. Related sites: postgres/init.sql:5–6 defines unrestricted text; mappings.py:15 maps both fields; docs/content/docs/guides/ingestion.mdx:113 incorrectly says default replica identity suffices. Debezium documents this behavior. This is inside R8's unrequired CDC component; removal is the smallest scope-compliant remedy.
  • 🚨 examples/ingestion/docker-compose.yml:15 - Running the documented demo publishes the engine on all host interfaces with the public bootstrap credential from line 6. Any reachable peer can authenticate and read or modify its knowledge graphs. The same exposure occurs at line 43: the public secrets at lines 7–8 let peers forge CDC and billing writes. This contradicts the file's instruction to replace secrets before exposing ports; exposure happens immediately. These publications belong to R8's unrequired Compose stack, whose removal is the smallest scope-compliant remedy.
  • ⚠️ examples/ingestion/adapter/ingest/demo.py:187 - The self-check can report success after its watcher fails. If the connection closes or a subscription_error arrives after the final expected retraction, collect() terminates with an exception, but expect_quiet() treats the empty queue as success and main prints “all deltas matched.” The collector is never awaited or checked. Related sites: demo.py:49 collects, :145 interprets silence, :184 starts the task, and :189 cancels without observing failure. This belongs to R8's unrequired ingestion demo; removing that demo is the smallest scope-compliant remedy.
  • 🚨 examples/ingestion/demo.sh:12 - The intent requires performance policy as a pipeline “always run before a PR commit.” That source-level integration is absent: make perf-gate remains a standalone manual target, and the added demo.sh:10–12 only builds and runs the example. Its demo.py:28 timeout permits 15-second delivery and does not enforce the millisecond requirement. Add the required pre-commit performance integration and coverage for the introduced serialized ingestion path, or explicitly amend the criterion. This concerns missing implementation, not this run's deferred test or CI results.
  • 🚨 .github/workflows/ci.yml:10 - The explicit criterion “Ignore draft PRs” remains unimplemented. The branch leaves the pull_request workflows without draft guards: .github/workflows/ci.yml:10 runs the fast gate for drafts, .github/workflows/full-suite.yml:39 permits release-labeled drafts, and .github/workflows/docker.yml:26 permits their image builds. The added examples also match the existing examples/** fast-gate filter. Add draft exclusion to these entry points or obtain an explicit amendment to the criterion.
  • ⚠️ examples/ingestion/docker-compose.yml:22 - Simplification: the intent requires reactive engine/SDK subscriptions and quality gates, not a new ingestion product. The added Postgres/Debezium service and mapper, billing webhook endpoint and signing scheme, configurable relation/type/key mappings, batch/schema-wrapper/tombstone acceptance paths, persistent ingest_revision ledger, retrying adapter service, separate WebSocket client/watcher, Docker stack, demo and ingestion guide exceed that scope. Primary sites: docker-compose.yml:22, :38, :52; debezium.py:21; webhooks.py:38; signing.py:28; changes.py:30; iql.py:17; server.py:89; engine.py:59; watch.py:41; demo.py:36; docs/content/docs/guides/ingestion.mdx:1. Remove these components and implement the required subscription path in the existing SDKs and quality pipeline.

🔧 Fix applied.
6 issues (4 errors, 2 warnings) still open:

  • 🚨 examples/ingestion/adapter/ingest/demo.py:177 - The required SDK subscription path remains absent. The intent requires agents subscribing through “the engine WebSocket and SDK” in the “e2e quality checks pipeline,” but the added Watcher.open(...) uses a separate example client. Public SDKs still discard subscription pushes (packages/inputlayer-js/src/connection.ts:149; packages/inputlayer-py/src/inputlayer/connection.py:338), and TESTING.md explicitly excludes SDK agents. Neither the new demo nor its tests is included in the release-checkpoint pipeline. Implement and exercise the public SDK path through that pipeline.
  • 🚨 examples/ingestion/adapter/ingest/applier.py:50 - Replay safety fails across an interrupted connection: revision 10's write can remain executing on the engine after the adapter detects disconnection; a reconnected request commits revision 11; the older write then commits unconditionally, restoring revision 10 and stale data. The engine awaits execution before reading socket closure. The invariant must cover applier.py:50 (separate revision read), applier.py:77 (reconnect), and iql.py:60–66 (data deletion/insertion and watermark replacement), for both CDC and webhooks. The supported boundary is the engine's atomic program commit, including revision validation. This defect belongs to the unrequired ingestion subsystem identified in R8; removing that subsystem is the smallest scope-compliant remedy.
  • 🚨 examples/ingestion/demo.sh:12 - The intent requires performance policy as a pipeline “always run before a PR commit.” That source-level integration is absent: make perf-gate remains a standalone manual target, and the added demo.sh:10–12 only builds and runs the example. Its demo.py:28 timeout permits 15-second delivery and does not enforce the millisecond requirement. Add the required pre-commit performance integration and coverage for the introduced serialized ingestion path, or explicitly amend the criterion. This concerns missing implementation, not this run's deferred test or CI results.
  • 🚨 .github/workflows/ci.yml:10 - The explicit criterion “Ignore draft PRs” remains unimplemented. The branch leaves the pull_request workflows without draft guards: .github/workflows/ci.yml:10 runs the fast gate for drafts, .github/workflows/full-suite.yml:39 permits release-labeled drafts, and .github/workflows/docker.yml:26 permits their image builds. The added examples also match the existing examples/** fast-gate filter. Add draft exclusion to these entry points or obtain an explicit amendment to the criterion.
  • ⚠️ examples/ingestion/docker-compose.yml:22 - Simplification: the intent requires reactive engine/SDK subscriptions and quality gates, not a new ingestion product. The added Postgres/Debezium service and mapper, billing webhook endpoint and signing scheme, configurable relation/type/key mappings, batch/schema-wrapper/tombstone acceptance paths, persistent ingest_revision ledger, retrying adapter service, separate WebSocket client/watcher, Docker stack, demo and ingestion guide exceed that scope. Primary sites: docker-compose.yml:22, :38, :52; debezium.py:21; webhooks.py:38; signing.py:28; changes.py:30; iql.py:17; server.py:89; engine.py:59; watch.py:41; demo.py:36; docs/content/docs/guides/ingestion.mdx:1. Remove these components and implement the required subscription path in the existing SDKs and quality pipeline.
  • ⚠️ examples/ingestion/adapter/ingest/applier.py:53 - Round 1's R2 fix left the outcome counters based on the pre-read. Concrete sequence: revision 12 remains in flight after disconnection; a reconnected revision-11 request reads watermark 9; revision 12 commits; the engine correctly skips revision 11, but this returns applied=1, skipped=0. Derive both counters from the committed conditional outcome. Related sites: examples/ingestion/adapter/ingest/applier.py:50–53 selects and counts candidates, :55–57 logs incorrect counts; examples/ingestion/adapter/ingest/iql.py:70 conditionally skips writes; examples/ingestion/adapter/ingest/server.py:74 exposes the counts for both CDC and webhooks.

🔧 Fix applied.
5 issues (3 errors, 2 warnings) still open:

  • 🚨 examples/ingestion/adapter/ingest/demo.py:177 - The required SDK subscription path remains absent. The intent requires agents subscribing through “the engine WebSocket and SDK” in the “e2e quality checks pipeline,” but the added Watcher.open(...) uses a separate example client. Public SDKs still discard subscription pushes (packages/inputlayer-js/src/connection.ts:149; packages/inputlayer-py/src/inputlayer/connection.py:338), and TESTING.md explicitly excludes SDK agents. Neither the new demo nor its tests is included in the release-checkpoint pipeline. Implement and exercise the public SDK path through that pipeline.
  • 🚨 examples/ingestion/demo.sh:12 - The intent requires performance policy as a pipeline “always run before a PR commit.” That source-level integration is absent: make perf-gate remains a standalone manual target, and the added demo.sh:10–12 only builds and runs the example. Its demo.py:28 timeout permits 15-second delivery and does not enforce the millisecond requirement. Add the required pre-commit performance integration and coverage for the introduced serialized ingestion path, or explicitly amend the criterion. This concerns missing implementation, not this run's deferred test or CI results.
  • 🚨 .github/workflows/ci.yml:10 - The explicit criterion “Ignore draft PRs” remains unimplemented. The branch leaves the pull_request workflows without draft guards: .github/workflows/ci.yml:10 runs the fast gate for drafts, .github/workflows/full-suite.yml:39 permits release-labeled drafts, and .github/workflows/docker.yml:26 permits their image builds. The added examples also match the existing examples/** fast-gate filter. Add draft exclusion to these entry points or obtain an explicit amendment to the criterion.
  • ⚠️ examples/ingestion/docker-compose.yml:22 - Simplification: the intent requires reactive engine/SDK subscriptions and quality gates, not a new ingestion product. The added Postgres/Debezium service and mapper, billing webhook endpoint and signing scheme, configurable relation/type/key mappings, batch/schema-wrapper/tombstone acceptance paths, persistent ingest_revision ledger, retrying adapter service, separate WebSocket client/watcher, Docker stack, demo and ingestion guide exceed that scope. Primary sites: docker-compose.yml:22, :38, :52; debezium.py:21; webhooks.py:38; signing.py:28; changes.py:30; iql.py:17; server.py:89; engine.py:59; watch.py:41; demo.py:36; docs/content/docs/guides/ingestion.mdx:1. Remove these components and implement the required subscription path in the existing SDKs and quality pipeline.
  • ⚠️ examples/ingestion/adapter/ingest/engine.py:139 - Engine requests have no response deadline. If the engine stalls after accepting a request while its TCP connection remains open, this await holds Applier's shared lock indefinitely. Debezium's 30-second HTTP timeout does not release that lock, so retries and billing requests accumulate behind it instead of receiving 503. Bound each complete engine request, close the connection on expiry, and raise EngineUnavailable so existing reconnect handling applies. Cover sibling waits at engine.py:88 (authentication), engine.py:111 (execution), engine.py:143 (stream completion), and lock holders at applier.py:43 (declaration) and applier.py:50 (revision reads and writes); both HTTP feeds share server.py:67, and startup retries depend on server.py:94 returning.

🔧 Fix applied.
4 issues (3 errors, 1 warning) still open:

  • 🚨 examples/ingestion/adapter/ingest/demo.py:177 - The required SDK subscription path remains absent. The intent requires agents subscribing through “the engine WebSocket and SDK” in the “e2e quality checks pipeline,” but the added Watcher.open(...) uses a separate example client. Public SDKs still discard subscription pushes (packages/inputlayer-js/src/connection.ts:149; packages/inputlayer-py/src/inputlayer/connection.py:338), and TESTING.md explicitly excludes SDK agents. Neither the new demo nor its tests is included in the release-checkpoint pipeline. Implement and exercise the public SDK path through that pipeline.
  • 🚨 examples/ingestion/demo.sh:12 - The intent requires performance policy as a pipeline “always run before a PR commit.” That source-level integration is absent: make perf-gate remains a standalone manual target, and the added demo.sh:10–12 only builds and runs the example. Its demo.py:28 timeout permits 15-second delivery and does not enforce the millisecond requirement. Add the required pre-commit performance integration and coverage for the introduced serialized ingestion path, or explicitly amend the criterion. This concerns missing implementation, not this run's deferred test or CI results.
  • 🚨 .github/workflows/ci.yml:10 - The explicit criterion “Ignore draft PRs” remains unimplemented. The branch leaves the pull_request workflows without draft guards: .github/workflows/ci.yml:10 runs the fast gate for drafts, .github/workflows/full-suite.yml:39 permits release-labeled drafts, and .github/workflows/docker.yml:26 permits their image builds. The added examples also match the existing examples/** fast-gate filter. Add draft exclusion to these entry points or obtain an explicit amendment to the criterion.
  • ⚠️ examples/ingestion/docker-compose.yml:22 - Simplification: the intent requires reactive engine/SDK subscriptions and quality gates, not a new ingestion product. The added Postgres/Debezium service and mapper, billing webhook endpoint and signing scheme, configurable relation/type/key mappings, batch/schema-wrapper/tombstone acceptance paths, persistent ingest_revision ledger, retrying adapter service, separate WebSocket client/watcher, Docker stack, demo and ingestion guide exceed that scope. Primary sites: docker-compose.yml:22, :38, :52; debezium.py:21; webhooks.py:38; signing.py:28; changes.py:30; iql.py:17; server.py:89; engine.py:59; watch.py:41; demo.py:36; docs/content/docs/guides/ingestion.mdx:1. Remove these components and implement the required subscription path in the existing SDKs and quality pipeline.

✅ No issues found.

⚠️ **Test** - 1 info

✅ No issues found.

  • Live validation: ✅ go - 10 of 10 scenarios driven live against the product
Scenario Result Live Evidence
Postgres inserts, updates, deletes, and primary-key changes reach subscribed queries ✅ pass live Postgres CDC and webhook demo; Primary-key changes and restart persistence
Billing events change derived answers while duplicates, stale events, and no-op updates produce no delta ✅ pass live Postgres CDC and webhook demo
Disconnected writes cannot overwrite newer revisions or misreport committed applied/skipped counts ✅ pass live test_applier.py::test_disconnected_write_preserves_revision_order_and_committed_counts
Batched duplicates, deletion tombstones, and recreation preserve facts and revision counts ✅ pass live test_applier.py::test_batch_replay_delete_and_recreate
Startup recovers after authentication or schema-declaration replies stall ✅ pass live Live timeout, retry, stream, and request-validation results
Both HTTP feeds return bounded 503s on stalled engine requests, release the shared lock, reconnect, and retry safely ✅ pass live Live timeout, retry, stream, and request-validation results
A missing final stream frame triggers the deadline; reconnect retrieves all 1,201 rows ✅ pass live Live timeout, retry, stream, and request-validation results
An idle subscription survives multiple request deadlines and receives the next delta ✅ pass live Live timeout, retry, stream, and request-validation results
Both feeds reject missing, stale, or tampered signatures and malformed data without preventing valid retries ✅ pass live Live timeout, retry, stream, and request-validation results
Restarting the engine, adapter, and Debezium preserves facts and prevents stale events from resurrecting deleted payments ✅ pass live Primary-key changes and restart persistence

  • cargo build --bin inputlayer-server

  • PYTHONPATH=examples/ingestion/adapter tmp/ingestion-test/venv/bin/python -m pytest examples/ingestion/adapter/tests --basetemp=tmp/ingestion-test/pytest -v --log-cli-level=INFO

  • INPUTLAYER_PORT=18080 ADAPTER_PORT=18090 docker compose --project-name nm-ingestion-test --env-file tmp/ingestion-test/stack.env -f examples/ingestion/docker-compose.yml -f tmp/ingestion-test/override.json up -d --no-build --wait inputlayer postgres adapter debezium

  • INPUTLAYER_PORT=18080 ADAPTER_PORT=18090 docker compose --project-name nm-ingestion-test --env-file tmp/ingestion-test/stack.env -f examples/ingestion/docker-compose.yml -f tmp/ingestion-test/override.json run --rm --no-deps demo

  • PYTHONPATH=examples/ingestion/adapter tmp/ingestion-test/venv/bin/python ~/.no-mistakes/evidence/01M41QR6R67KDDMZKWKPT67D5E/live_checks.py

  • PYTHONPATH=examples/ingestion/adapter tmp/ingestion-test/venv/bin/python ~/.no-mistakes/evidence/01M41QR6R67KDDMZKWKPT67D5E/restart_checks.py

  • Removed disposable containers, volumes, environment, and generated data; verified a clean worktree.

  • ℹ️ examples/ingestion/docker-compose.yml:11 - ./demo.sh with its default engine image (ghcr.io/inputlayer/inputlayer:latest, built 2026-10-03 11:18) fails: that image speaks WebSocket protocol 1, which the adapter refuses as designed, so the adapter never becomes healthy. The demo passes with a protocol-2 engine image passed through INPUTLAYER_IMAGE, which the guide documents. Out-of-the-box use depends on publishing a newer engine image.

  • Live validation: ✅ go - 7 of 7 scenarios driven live against the product

Scenario Result Live Evidence
Operator runs ./demo.sh; Postgres row changes and billing webhooks update the derived answer, and the watching client receives each delta in milliseconds ✅ pass live docker-e2e-demo.log: all 9 steps matched, deltas in 33-62 ms, exit 0 (engine image inputlayer-ing1-main:local)
A duplicate webhook and a late, lower-sequence webhook are skipped, with no spurious delta and a correct applied/skipped count ✅ pass live docker-e2e-demo.log steps 2 and 9: {'applied': 0, 'skipped': 1}; watcher quiet
No-op Postgres UPDATE and delete/rename of rows produce exactly the right retract/insert deltas ✅ pass live docker-e2e-demo.log steps 5-7
Adversarial: forged, missing, stale or tampered webhook signatures are rejected ✅ pass live adversarial-live.log: all four returned 401
Adversarial: engine stalls mid-request; the adapter replies 503 at the configured deadline instead of hanging, then recovers, and a retry of the same revision is skipped ✅ pass live adversarial-live.log: 503 after 10.02s while paused; 200 applied=1 after unpause; retry gives applied=0 skipped=1
Replay safety under an interrupted connection, with counters derived from committed conditional outcomes (in-flight newer revision) ✅ pass live adapter-pytest.log: test_disconnected_write_preserves_revision_order_and_committed_counts (32 cases) and test_batch_replay_delete_and_recreate passed against the real inputlayer-server binary
Adapter refuses an engine that speaks WebSocket protocol 1, with a clear error ✅ pass live docker-e2e-demo-ghcr-latest-protocol1.log and adapter log: 'engine speaks WebSocket protocol 1; this client needs 2 or later'
  • cargo build --bin inputlayer-server (engine from this worktree for engine-backed regressions)
  • python -m pytest -v -rs in examples/ingestion/adapter with the real engine binary (95 passed, 0 skipped: revision guard and disconnect race, committed-count derivation, request deadlines, mapping, signatures, startup)
  • COMPOSE_PROJECT_NAME=ilingtest INPUTLAYER_PORT=18080 ADAPTER_PORT=18090 ./demo.sh with default ghcr.io/inputlayer/inputlayer:latest (adapter refused protocol-1 engine)
  • COMPOSE_PROJECT_NAME=ilingtest INPUTLAYER_IMAGE=inputlayer-ing1-main:local INPUTLAYER_PORT=18080 ADAPTER_PORT=18090 ./demo.sh (full 9-step self-checking demo, exit 0)
  • Live POSTs to /webhooks/billing with forged, missing, stale and tampered signatures
  • docker pause ilingtest-inputlayer-1 then a signed webhook POST (503 deadline), docker unpause, then the same POST followed by its retry
  • docker compose -p ilingtest down -v teardown and removal of local caches
✅ **Document** - passed

✅ No issues found.

✅ No issues found.

🔧 **Lint** - 1 issue found → auto-fixed ✅
  • ⚠️ examples/ingestion/adapter/pyproject.toml:15 - Python lint, formatting, and static-analysis checks remain blocked: Ruff and mypy are absent from PATH, and python3 reports neither module installed. No repository-local tool path was supplied. ShellCheck and git diff --check passed.

🔧 Fix applied.
✅ Re-checked - no issues remain.

✅ No issues found.

✅ **Push** - passed

✅ No issues found.

✅ No issues found.

jsam added 6 commits October 3, 2026 22:01
@jsam
feat(examples): ingestion recipe for Postgres CDC and webhooks
b413ac3 
Adds examples/ingestion: Postgres -> Debezium Server (HTTP sink) and signed
SaaS webhooks -> a small adapter -> InputLayer facts, plus a docker-compose
stack and a self-checking demo in which a watching client receives every
change to a derived answer.

Each change becomes one atomic program that replaces the fact by key and
records its source revision (Postgres LSN, webhook sequence) in
ingest_revision, so replays and late out-of-order deliveries are no-ops and
deletes become retractions with a tombstone revision. Every write endpoint
verifies Standard Webhooks signatures. Documented as the "Ingestion (CDC &
Webhooks)" guide.
@jsam
no-mistakes(review): Fix ingestion ordering, placeholders, secrets, a…
934ee11 
…nd watcher failures
@jsam
no-mistakes(review): Derive ingestion counters from committed conditi…
e315771 
…onal updates
@jsam
no-mistakes(review): Bound engine requests with configurable deadlines
771b469
@jsam
no-mistakes(document): Correct ingestion batching, acknowledgment, an…
f8d501d 
…d timeout documentation
@jsam
no-mistakes(lint): Fix ingestion adapter formatting and import spacing
f61cffc
@jsam jsam changed the title feat(examples): add CDC and webhook ingestion with reactive queries feat(examples): add Postgres CDC and webhook ingestion recipe with reactive queries Oct 3, 2026
@jsam
jsam merged commit da8e147 into main Oct 3, 2026
14 checks passed
@jsam
jsam deleted the fm/il-ing1 branch October 3, 2026 22:25
@jsam jsam mentioned this pull request Oct 4, 2026
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

See More

Successfully merging this pull request may close these issues.

1 participant

@jsam