Repository navigation
fix(inputlayer-py): timestamps, meta commands, in_(), negation and literals the engine accepts - #239
Merged
Merged
Conversation
jsam
marked this pull request as ready for review
October 4, 2026 12:36
…terals the engine accepts
Phase-1 item P1-C2 of the SDK redesign, compile path v2 part 2.
- R-TYPE: `Timestamp` and `datetime` columns are declared `int` and hold
Unix milliseconds (the engine's schema parser has no timestamp type);
`datetime` values are written as milliseconds (naive ones as UTC) and
typed rows read them back as UTC datetimes. `Vector[N]` is declared
`vector(N)`, the form the parser reads (it rejected `vector[N]`).
Migrations declare the same types.
- R-META: meta commands come from one table (`_meta`) rendered against the
shared fixture packages/conformance/meta-commands.json, like the JS SDK.
`session.list_rules()` sent `.session list` and `drop_rule(name, index=)`
sent `.session remove`, both rejected; `rule_definition()` sent
`.rule show`, which the engine answers with an empty query result. They
now send `.session`, `.session drop <n>` and `.rule def` and parse the
replies with the fixture's grammar; `list_rules()` no longer returns the
`Rules:` header as a rule.
- R-IN: `a.in_(b)` compiled to `rel(..., X, ...)`, which is not IQL. It is
now an atom of b's relation with a in b's column and `_` elsewhere
(`manager(_, Id)`); `~a.in_(b)` negates it. `matches()` fills unbound
columns with `_` instead of emitting a short atom.
- R-NEG: `~` is pushed down to atoms and comparisons (`!(X = 1)` was a
parse error), every negated atom must share a variable with a positive
atom (a CompileError otherwise, before anything is sent), and one linked
to the body by a constant only binds it through an SDK-owned constant
relation: a session fact in the query's own program
(`il_const_s("x")` then `..., il_const_s(K), K = "x", !r(K)`), a
persistent row written with a view's clauses. View bodies are canonical:
positive atoms, negated atoms, then comparisons.
- R-LIT: one encoder (`_literal`) builds every literal: the engine's five
string escapes (control characters such as NUL are written raw, since
the engine keeps `\0` and `\xHH` verbatim), 64-bit ints, finite floats,
vectors; None, NaN, infinities, out-of-range ints and lone surrogates are
refused with CompileError. The LangChain and LangGraph helpers, the
migration recorder, why_not() and vector_search() use it.
tests/compile_rules_live.py pins each rule against a live engine (every
command of the shared fixture, the rejected spellings, the T-B2 negation
form, and a fuzz test that round-trips random values: encode, insert, read
back, compare) and runs in `make python-sdk-live`.
This was referenced Oct 4, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Intent
So fix the SDKs by improving them. (Context: the captain asked to rework the SDKs so agents and agentic loops use InputLayer without writing Datalog, and to fix the SDK release blockers by improving the SDKs rather than patching. The agreed design, signed off by an independent challenger after three rounds, is ~/firstmate/data/sdk-design/report.md; this item is its phase-1 PR P1-C2 P1-C2 (Py). Part 1 (P1-C1) merged as #235.)
The referenced item, P1-C2 of the agreed design (Python SDK, compile path v2 part 2), is: R-TYPE, R-META (shared command fixture,
.rule defparsing), R-IN, R-NEG check andil_constbinding, and the R-LIT encoder with the fuzz round-trip, each pinned by unit tests on compiled text and live tests against a real engine that run in CI. The design's rules:datetime/Date/Timestampcolumns declareintand carry Unix milliseconds; values are converted both ways..session,.rule def,.rule drop,.subscribe,.unsubscribe,.why,.why_not), the same JSON fixture in both SDKs (packages/conformance/meta-commands.json, already used by the JS SDK), so a wrong spelling (.session list) cannot exist in one SDK only; the.rule defreply is parsed (rule_definition() returned an empty list for a live rule).col.in_(Other.col)emits a positive atomother(_, V, _)with V unified to col; not-in emits the negated atom under R-NEG. Neverrel(..., X, ...).~on an atom emits!r(...). A negated atom must share a variable with a positive atom (otherwise a CompileError before anything is sent, because the engine accepts such a rule and then fails every query of it). It never adds a filter to a positive atom. A negated atom linked to the body only by a constant binds the constant through an SDK-owned typed constant relation: on the read path a program-local session fact in the same program (il_const_s("x")then..., il_const_s(K), K = "x", !r(K)); for views persistentil_const_<type>rows written in the same program as the clauses (never deleted, rows may be shared). In a view each clause body is ordered positive atoms, then negated atoms, then comparisons and equalities. The write-path binding (stagedil_txn_constrows) belongs to a later item (P1-I), so a conditional delete needing it is refused.Other phase-1 items and phases 2-4 are out of scope.
What Changed
datetimeandTimestampcolumns are now declared asintand stored as Unix milliseconds. Values are converted on the way in and in result columns on the way out. Meta commands (.session,.rule def,.rule drop,.rule remove, and so on) now come from a new_metatable that renders the sharedpackages/conformance/meta-commands.jsonfixture, so the Python and JS SDKs send the same text. The SDK also parses the replies:rule_definition()returns a live rule's clauses.Session.list_rules()returns one entry per clause.Session.drop_rule(name, index=)works out the clause's position among all session rules.col.in_(Other.col)emits a positive atom that unifies withcol. A negatedin_emits a negated atom.~on an atom emits!r(...). If a negated atom shares no variable with a positive atom, compiling raises aCompileError.il_const_*relation: a session fact in queries and session rules, persistent rows in views._literalencoder is now the only place values become program text. It covers strings (escapes, newlines, unicode), ints, floats, bools and vectors, and raises aCompileErrorfor NaN and Infinity. The compiler, vector-search radius and limit values, and the LangChain/LangGraph params all use it. New unit tests check the compiled text (test_compile_rules.py,test_meta.py). A new live suite,compile_rules_live.py, runs inmake python-sdk-liveand includes a round-trip fuzz of random literals through a real engine.Risk Assessment
✅ Low: The fix-round commits correctly close the round-1 findings: datetime conversion now happens once in QueryPlan for whole-relation, projected, aggregate min/max and .why rows, including Optional[datetime], while Timestamp stays an int. Radius and limit/offset now go through the encoder, and the fixture text is updated. All of this is pinned by unit tests and by live tests that CI runs through make python-sdk-live.
Testing
I built and ran the real inputlayer-server on an isolated port and data directory. Against it I ran the new compile_rules_live suite (the shared meta-command fixture, rule_definition, datetime, in_/not-in, ground-negation constants in queries and views, and the literal fuzz), the docs-guide live suite, four extra fuzz seeds, and a custom adversarial driver that records the exact program text sent. Everything passed: 48/48 live tests, 27/27 driver checks, and all four seeds. I tore down the server and data directory afterwards. One thing outside this change's scope:describe()sends.rel <name>and returns data rows instead of the schema. This code was not touched by the diff. There is no UI surface, so the evidence is CLI transcripts..session list) fail.rule def lead-> ['lead(Name) <- emp(Id, Name, ), boss(, Id)']At > 1767225600000?emp(Id, Name, Dept), boss(_, Id)/!boss(_, Id)Dist <= 0.5finds doc 1;Dist <= "0.5), evil(X"-> []; nan/inf/limit=-1/offset=-5 refused with nothing sent;limit(2, 1)-> [[2],[3]]Evidence: Adversarial live driver transcript (program text sent + PASS/FAIL per check)
Evidence: Adversarial live driver script
Evidence: Live pytest run (compile_rules_live + docs_queries_live) against real server
Evidence: Fuzz round-trip with fixed seeds
Evidence: Program text for in_ / not-in
Evidence: Literal encoder on hostile input
Pipeline
Updates from git push no-mistakes
✅ **intent** - passed
✅ No issues found.
✅ **Rebase** - passed
✅ No issues found.
🔧 **Review** - 3 issues found → auto-fixed ✅
packages/inputlayer-py/src/inputlayer/result.py:109- R-TYPE says a datetime column's values are 'converted both ways', and the updated guide (docs/content/docs/guides/python-sdk.mdx:162) says a datetime is 'read back as a UTC datetime'. In practice the ms→datetime conversion (_from_engine) only runs in_row_to_obj, and only when the query selects one whole relation (_relation_clsis set; knowledge_graph.py:415). Concrete case:kg.query(Reading.at)orkg.query(Reading.sensor, Reading.at)givesrows/to_dicts()/to_df()/scalar()values like 1791117000250 (int), not datetime. Evenkg.query(Reading).rows/to_dicts()returns raw ints, because only iteration and indexing go through_from_engine. Aggregates over a datetime column (min/max) are also left as ints. Also in result.py:121, theisinstance(info.annotation, type)check skipsdatetime | None/Optional[datetime]fields, so Pydantic's seconds-vs-ms heuristic decides those. Fix: convert in one place using the plan's output column types (outputs that map to a datetime-annotated field) when buildingResultSet.rows, and unwrap Optional annotations. Otherwise, narrow the docs claim to typed rows.packages/inputlayer-py/src/inputlayer/knowledge_graph.py:600- R-LIT requires the encoder to be 'the only place program text is built from values'.vector_searchstill writes the caller'sradiusstraight into the program (f"Dist <= {radius}"). Soradius=float('nan')sendsDist <= nan, andfloat('inf')sendsDist <= inf: text the engine misreads as an atom or variable, where the encoder would raise CompileError. A non-numeric radius is interpolated unchecked too. The query vector one line above was switched toencode_literal, but this line was not. Fix:f"Dist <= {encode_literal(radius)}". A sibling that must hold the same invariant: compiler.py:1081_limit_atominterpolateslimit/offsetwithout validation. Route them through the int encoder, or check they are non-negative ints.packages/conformance/meta-commands.json:2- The shared fixture's description still says the Python SDK 'is planned to' rendercommandsfrom its table. This change makes it do so (tests/test_meta.py renders every fixture command through_meta.COMMANDSand checks the table pins exactly the fixture's names). Update the sentence so the shared contract doesn't misdescribe the Python SDK.🔧 Fix applied.
✅ Re-checked - no issues remain.
✅ No issues found.
✅ **Test** - passed
✅ No issues found.
manager(_, Id)and returns Alice and Bob; ~in_ sends!manager(_, Id)and returns Cara; ~in_ in a conditional delete removes only non-managerscargo build --release --bin inputlayer-serverthen started the server with a disposable --data-dir on 127.0.0.1:8097INPUTLAYER_TEST_SERVER=ws://127.0.0.1:8097/ws uv run --extra dev pytest tests/compile_rules_live.py tests/docs_queries_live.py -v -s(48 live tests)INPUTLAYER_FUZZ_SEED={1,42,7777,123456,99999999} pytest tests/compile_rules_live.py -k round_trip(literal fuzz round-trip with more seeds)uv run --extra dev python drive_scenarios.py: a scenario driver that spies on every program the SDK sends to the live engine and checks results (S1-S10)uv run --extra dev pytest tests/test_compile_rules.py tests/test_meta.py tests/test_escape_iql.py(targeted unit pins on the compiled text)✅ No issues found.
.session list) fail.rule def lead-> ['lead(Name) <- emp(Id, Name, ), boss(, Id)']At > 1767225600000?emp(Id, Name, Dept), boss(_, Id)/!boss(_, Id)Dist <= 0.5finds doc 1;Dist <= "0.5), evil(X"-> []; nan/inf/limit=-1/offset=-5 refused with nothing sent;limit(2, 1)-> [[2],[3]]cargo build --release --bin inputlayer-server, then the server started on 127.0.0.1:18093 with a mktemp data directory (asmake python-sdk-livedoes)uv run --extra dev pytest tests/compile_rules_live.py tests/docs_queries_live.py -vagainst the live server (48 passed)INPUTLAYER_FUZZ_SEED={1,42,31337,2024} pytest tests/compile_rules_live.py -k round_trip: literal round-trip fuzz, 200 rows per seedpython drive_sdk.py(evidence dir): adversarial live driver that wraps the KG connection's execute() to record the program text sent, covering in_/~in_, unlinked negation, injection-shaped and unicode/escaped string literals, NaN/±inf in conditions, vector_search radius (0.5, nan, inf, injection string), invalid limit/offset, datetime whole/projected/condition reads, rule_definition via.rule def, and.sessionvs the rejected.session listuv run --extra dev pytest tests/test_compile_rules.py tests/test_meta.py tests/test_escape_iql.py -q(unit pins on compiled text, 106 passed)✅ **Document** - passed
✅ No issues found.
✅ No issues found.
🔧 **Lint** - 1 issue found ✅
packages/inputlayer-py/src/inputlayer/result.py:85- ruff format and mypy --strict report many issues across packages/inputlayer-py, but almost all were already there at the base commit and neither tool is enforced in CI or the Makefile. ruff check reports one E501 on changed files, at result.py:85 (an existing install-hint string). I left these alone; cleaning them up across the package would be a separate follow-up.✅ No issues found.
✅ **Push** - passed
✅ No issues found.
✅ No issues found.