Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
45 changes: 45 additions & 0 deletions .github/workflows/full-suite.yml
Original file line number Diff line number Diff line change
Expand Up @@ -291,6 +291,51 @@ jobs:
if-no-files-found: ignore
retention-days: 7

python-sdk-live:
name: Python SDK query examples (live server)
needs: gate
if: ${{ !inputs.perf_gate_only }}
runs-on: blacksmith-4vcpu-ubuntu-2404
timeout-minutes: 25
steps:
- uses: actions/checkout@v4

- name: Install Rust toolchain
uses: dtolnay/rust-toolchain@stable

- name: Cache cargo
uses: useblacksmith/cache@v5
with:
path: |
~/.cargo/registry
~/.cargo/git
target/release
key: ${{ runner.os }}-cargo-py-sdk-live-${{ hashFiles('**/Cargo.lock') }}
restore-keys: |
${{ runner.os }}-cargo-py-sdk-live-

- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: "3.12"

- name: Install uv
uses: astral-sh/setup-uv@v3
with:
enable-cache: true

- name: Run the Python SDK guide's query examples against a live server
run: make python-sdk-live

- name: Upload server log on failure
if: failure()
uses: actions/upload-artifact@v4
with:
name: inputlayer-py-sdk-live-server-log
path: /tmp/il_py_sdk_live_server.log
if-no-files-found: ignore
retention-days: 7

# Lint and unit-test the performance gate tool.
perf-gate-tool:
name: Performance gate tool
Expand Down
31 changes: 30 additions & 1 deletion Makefile
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
.PHONY: all ci fmt fmt-check lint pre-pr pre-pr-snapshots pre-pr-js pre-pr-selftest perf-gate perf-gate-check secret-check install-gitleaks install-hooks uninstall-hooks hooks-test bench-genbi test test-fast test-release unit-test integration-test oracle-test e2e-test e2e-reactive e2e-update test-affected doc doc-check check build build-release clean fix release snapshot-test test-all ci-test-all flush-dev docker docker-run docker-deploy docker-deploy-no-tls docker-logs docker-stop k8s-check deny python-test python-test-live python-test-examples vc-gate js-test js-test-live front-build front-deploy gui-build run run-server demo coverage view-coverage static-analysis
.PHONY: all ci fmt fmt-check lint pre-pr pre-pr-snapshots pre-pr-js pre-pr-selftest perf-gate perf-gate-check secret-check install-gitleaks install-hooks uninstall-hooks hooks-test bench-genbi test test-fast test-release unit-test integration-test oracle-test e2e-test e2e-reactive e2e-update test-affected doc doc-check check build build-release clean fix release snapshot-test test-all ci-test-all flush-dev docker docker-run docker-deploy docker-deploy-no-tls docker-logs docker-stop k8s-check deny python-test python-test-live python-test-examples vc-gate js-test js-test-live python-sdk-live front-build front-deploy gui-build run run-server demo coverage view-coverage static-analysis

SHELL := /bin/bash

Expand Down Expand Up @@ -456,6 +456,35 @@ js-test-live:
rm -rf $$DATA_DIR; \
exit $$TEST_EXIT

# python-sdk-live runs every query example in the Python SDK guide
# (docs/content/docs/guides/python-sdk.mdx) against a live server, with the
# pins of the SDK's query compile rules, so a query form the engine rejects
# fails here rather than in a user's hands.
PY_LIVE_PORT ?= 8092
python-sdk-live:
@cargo build --release --bin inputlayer-server
@cd packages/inputlayer-py && uv sync --extra dev
@DATA_DIR=$$(mktemp -d -t il-py-sdk-live-XXXXXX); \
INPUTLAYER_ADMIN_PASSWORD=admin ./target/release/inputlayer-server \
--host 127.0.0.1 --data-dir $$DATA_DIR --port $(PY_LIVE_PORT) \
> /tmp/il_py_sdk_live_server.log 2>&1 & \
SERVER_PID=$$!; \
for i in $$(seq 1 40); do \
if curl -sf http://127.0.0.1:$(PY_LIVE_PORT)/health > /dev/null 2>&1; then break; fi; \
sleep 0.5; \
done; \
cd packages/inputlayer-py && \
INPUTLAYER_TEST_SERVER=ws://127.0.0.1:$(PY_LIVE_PORT)/ws \
INPUTLAYER_TEST_USER=admin \
INPUTLAYER_TEST_PASSWORD=admin \
uv run --extra dev pytest tests/docs_queries_live.py -v; \
TEST_EXIT=$$?; \
cd - > /dev/null; \
kill $$SERVER_PID 2>/dev/null || true; \
wait $$SERVER_PID 2>/dev/null || true; \
rm -rf $$DATA_DIR; \
exit $$TEST_EXIT

# Coverage & Static Analysis

# Generate test coverage report (requires cargo-tarpaulin)
Expand Down
23 changes: 13 additions & 10 deletions docs/content/docs/guides/python-sdk.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -232,7 +232,7 @@ engineers = await kg.query(

### Selecting Specific Columns

Instead of fetching full rows, you can select just the columns you need:
Instead of fetching full rows, you can select just the columns you need. The result has exactly those columns, in that order, and like every query result it is a set: two employees with the same name and salary give one row.

```python
result = await kg.query(
Expand Down Expand Up @@ -290,6 +290,8 @@ for row in result:

### Ordering and Pagination

Ordering and pagination run in the engine, so the engine sends only the requested page. To page a projection, select the column you order by: ordering a page by a column you do not select is ambiguous, so the SDK raises `CompileError`.

```python
# Top 10 highest paid
result = await kg.query(
Expand Down Expand Up @@ -323,22 +325,27 @@ result = await kg.query(
join=[Employee],
)
for row in result:
print(f"{row.department}: {row.count} employees, avg ${row.avg}")
print(f"{row.department}: {row.count_id} employees, avg ${row.avg_salary}")
```

Result columns are named after what you select: a column by its name (`row.department`), a computed column by its keyword, and an aggregate as `<function>_<column>` (`count_id`, `avg_salary`, `max_salary`; `count()` without a column is `count`). A repeated name gets a `_2` suffix. Pass an aggregate as a keyword to name it yourself: `kg.query(Employee.department, n=count(Employee.id), join=[Employee])`.

For more specialized aggregation, `top_k` lets you find the top entries per group:

```python
from inputlayer import top_k

# Top 3 highest-paid employees per department
result = await kg.query(
Employee.department, Employee.name, Employee.salary,
Employee.department,
top_k(3, Employee.name, order_by=Employee.salary, desc=True),
join=[Employee],
top_k(3, Employee.department, order_by=Employee.salary, desc=True),
)
# columns: department, name, salary
```

The columns you select beside `top_k` are the groups; the columns passed to `top_k` before `order_by` are kept for each of the top rows.

### Working with Results

Every query returns a `ResultSet` with several ways to access the data:
Expand Down Expand Up @@ -368,7 +375,7 @@ df = result.to_df() # pandas DataFrame (requires pandas)

### Query Plans

To understand how a query will execute without running it, use `debug()`:
To understand how a query will execute without running it, use `debug()`. It takes the same arguments as `query()`, and so does `why()`, which returns the query's rows with a proof tree for each. The engine does not paginate `.why`, so `why()` fetches every row and applies `order_by`, `limit` and `offset` in the SDK:

```python
plan = await kg.debug(
Expand Down Expand Up @@ -621,7 +628,7 @@ from inputlayer import functions as fn
# Rows from the last hour
result = await kg.query(
SensorReading,
where=lambda r: fn.within_last(r.timestamp, fn.time_now(), 3600000),
where=lambda r: r.timestamp > fn.time_sub(fn.time_now(), 3600000),
)

# Time-decayed scoring
Expand Down Expand Up @@ -776,10 +783,6 @@ This is a large topic with its own dedicated guide. See [Migrations](migrations)

A few rough edges to be aware of in the current pre-alpha SDK:

- **Aggregates are session rules under the hood**. `kg.query(count(Employee.id))` works by registering a temporary session rule and querying it. The rule is best-effort dropped after the call, so if your process dies between rule registration and cleanup the rule lingers until the WebSocket session ends. This is invisible to application code but worth knowing if you read server logs.

- **`order_by` and `offset` are applied client-side** for plain queries. IQL only supports ordering inside aggregate heads (`top_k`, `within_radius`, etc.), so the SDK sorts and slices the result rows in Python after they come back. This is fine for typical result sizes but is not appropriate for streaming over millions of rows.

- **`kg.relations()` only sees relations after data exists**. The server's `.rel` listing surfaces a relation only after at least one row has been inserted; defining the schema is not enough. If you need a registry of declared schemas that's independent of data, track them in your application code.

## Next Steps
Expand Down
2 changes: 1 addition & 1 deletion packages/inputlayer-py/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -454,7 +454,7 @@ The autodetector diffs your current Python models against the last migration's s
| `grant_access(username, role)` | Grant per-KG access |
| `revoke_access(username)` | Revoke per-KG access |
| `list_acl()` | List access control entries |
| `explain(*select, ...)` | Show query plan without executing |
| `debug(*select, ...)` | Show query plan without executing (same arguments as `query`) |
| `execute(iql)` | Execute raw IQL |
| `status()` | Get server status |
| `compact()` | Trigger storage compaction |
Expand Down
2 changes: 2 additions & 0 deletions packages/inputlayer-py/src/inputlayer/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@
from inputlayer.exceptions import (
AuthenticationError,
CannotDropError,
CompileError,
ConnectionError,
IndexNotFoundError,
InputLayerConnectionError,
Expand Down Expand Up @@ -94,6 +95,7 @@
"CannotDropError",
"ClearResult",
"ColumnInfo",
"CompileError",
"ConnectionError",
"DebugResult",
"DeleteResult",
Expand Down
Loading
Loading