Skip to content

Rebuild koshell as a hybrid Rust + Node monorepo#1

Merged
sjdhome merged 13 commits into
mainfrom
rewrite/hybrid-rust-node
Jul 2, 2026
Merged

Rebuild koshell as a hybrid Rust + Node monorepo#1
sjdhome merged 13 commits into
mainfrom
rewrite/hybrid-rust-node

Conversation

@sjdhome

@sjdhome sjdhome commented Jul 1, 2026

Copy link
Copy Markdown
Contributor

Context

The previous single-package TypeScript koshell was a half-finished shell wrapper: the timeline, terminal-context, ai-context, and screen-diff modules were only test-driven and never wired into the running process. This PR rebuilds the repository around the hybrid architecture recorded in the internal workspace — a native Rust foreground terminal process (koshell-rs) plus a Node.js AI daemon (koshell-ai-daemon) that communicate over newline-delimited JSON on a Unix domain socket.

Scope for this stage (agreed): complete the Rust terminal-core and a minimal Node receiver that logs and acks #? requests. pi integration, provider configuration, the tool loop, and streaming AI responses are the next stage.

Confirmed decisions: single monorepo (Cargo + pnpm workspaces); alacritty_terminal for terminal emulation; #? via OSC shell-integration markers only.

What's included

  • Phase 0 — monorepo scaffolding (crates/koshell-rs, crates/koshell-proto, packages/ai-daemon); the old TypeScript prototype and its docs are frozen under reference/; refreshed README/AGENTS/docs; CI workflow.
  • Phase 1 — Rust terminal-core: shell resolution, portable-pty session, stdin/stdout forwarding, resize, signals, nested-start guard (transparent shell-wrapper parity).
  • Phase 2alacritty_terminal-backed mirror with plain-text snapshots and alt-screen detection, the timeline store, line-level screen diffing, and terminal context selection (semantics ported from the prototype, unit-tested).
  • Phase 3 — shell integration (temporary bash/zsh rc emitting OSC 777 markers), a chunk-boundary-safe marker scanner, #? detection, command spans, and live recording into the timeline.
  • Phase 4 — Rust IPC client: lazy connect + graceful degrade, sends hello + ai_request (with the assembled context package).
  • Phase 5-min — minimal Node daemon: NDJSON decoder, protocol parser, logs #? requests with their context package and replies ack.

Verification

  • Rust: cargo test (35), cargo clippy -D warnings, cargo fmt --check — all green.
  • Node: pnpm check (format, lint, typecheck, 8 tests) — green.
  • Real PTY smoke: koshell runs a real zsh as a transparent wrapper; echo hi #? explain this output prints hi, the OSC marker never leaks.
  • End-to-end: with the daemon running, koshell detects #?, connects, and sends hello + ai_request; the daemon logs the question and full context package, then disconnects cleanly.

Known limits (next stage)

  • visible_output is not produced yet, so primaryText/recentPtyOutput still contain raw ANSI (currentScreen is clean text).
  • One snapshot per PTY chunk; no debounce/ring retention yet.
  • IPC covers hello/ai_request/ack only; the daemon is started manually (no auto-launch).
  • pi session, provider XDG/TOML config, koshell_* tool round-trips, and AI streaming (ai_delta/ai_tool_call/ai_response_end) are deferred.

🤖 Generated with Claude Code

sjdhome and others added 13 commits July 1, 2026 11:13
Rebuild koshell around the hybrid architecture: a Cargo workspace for the
Rust terminal-core (crates/koshell-rs, crates/koshell-proto) alongside a pnpm
workspace for the Node AI daemon (packages/ai-daemon). The pre-rewrite
TypeScript prototype and its docs are frozen under reference/. Adds
rust-toolchain pin, CI workflow, and refreshed README/AGENTS/docs.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Add the koshell-rs interactive session: shell resolution, PTY env construction,
and nested-start guard ported from the frozen prototype, plus a portable-pty
session that forwards stdin/stdout, propagates SIGWINCH resizes, and exits with
the child's code. Reaches transparent shell-wrapper parity.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Add the terminal-core data layer: an alacritty_terminal-backed headless mirror
with plain-text snapshots and alt-screen detection, the append-only timeline
store with its queries, line-level screen diffing, and terminal context
selection. Semantics match the frozen prototype, covered by ported unit tests.
Restructure koshell-rs as a lib plus thin binary.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Generate temporary bash/zsh rc files that emit OSC 777 command-boundary
markers, and parse them out of the PTY stream with a chunk-boundary-safe
scanner so they never leak to the terminal. Rework the session around a single
processor thread that records PTY output, human input, and screen snapshots
into the timeline, tracks command spans, and on a #? command assembles the
terminal context package (placeholder inline ack until the daemon is wired).
Clamp zero-sized PTYs to avoid an emulator panic.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Send #? requests to the AI daemon over newline-delimited JSON on a Unix domain
socket, connecting lazily and degrading gracefully when the daemon is absent.
On a #? the session sends a hello handshake then the ai_request carrying the
assembled context package, and acknowledges inline whether it reached the
daemon. Socket path follows XDG conventions.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Implement the JSONL Unix-socket receiver: an incremental NDJSON decoder, a
protocol parser for hello/ai_request/bye, and a server that logs #? requests
with their context package and acks them. Socket path resolution mirrors the
Rust side. Covered by framing and protocol unit tests. pi integration is the
next stage.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
License the project under Apache-2.0 (LICENSE plus license metadata in the
workspace manifests) and add contributor-facing files for public contribution:
CONTRIBUTING, a Code of Conduct, a pull request template, and bug/feature issue
templates.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
pnpm/action-setup needs an explicit version because the repo omits the
packageManager field. Pin to major 11 to match the engines floor.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
A #? question asked twice in a row only triggered on the first attempt. The
shell-integration hooks detected #? from shell history: bash deduped by command
text, and zsh's history read was defeated entirely by hist_ignore_dups (identical
repeats collapse to one entry). bash now dedups by history entry number; zsh stops
consulting history and instead captures the submitted line via an accept-line
widget (delegating to any pre-existing widget) plus a preexec-captured command
line, immune to hist_ignore_dups/hist_ignore_space. Add real-PTY regression tests
driving interactive bash and zsh (the latter reproducing the reported
hist_ignore_dups config), since this class of bug only manifests in an interactive
shell's line editor and cannot be reached by unit tests.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Document how #? will work inside foreground CLI programs (REPLs) where shell
integration is dormant and no OSC markers exist. Records the research: PTY probes
show no single universal in-band completion signal (PyREPL emits bracketed-paste
and termios toggles per command; node emits neither and stays raw). Specifies a
layered, confidence-ranked completion detector gated by shell command spans and
non-alt-screen: bracketed-paste edges (S1) with an output-quiescence plus
learned-prompt floor (S3), driving deferred #? launches. Captures prompt learning,
nested-prompt pairing, timeout/degradation, known limits, and the S1+S3 MVP scope
with S0/S2/OSC 133/per-program integration as future work.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Implement the S1+S3 prototype from design-0001 so #? works inside foreground REPLs
where shell integration is dormant. A ReplDetector owned by SessionState is gated to
run only inside a shell command span and off the alternate screen, so it never
overlaps the shell OSC path. It captures a submitted #? from keystrokes, then defers
the trigger until the command completes: S1 reads bracketed-paste ESC[?2004h/l edges
from the output stream (readline/libedit/PyREPL), and S3 falls back to output
quiescence for programs that emit no such edges (node). The deferred trigger reuses
build_context_package, so the AI sees the post-completion screen. Add real-PTY
regression tests exercising python (S1) and node (S3), asserting #? fires after the
command's output. #? capture uses keystrokes rather than the mirror-read for now; see
the design's Prototype status for that and other deferred hardening.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Design-0001 is revised after a first-principles review: #? is pinned as a
marker token with one firing rule (every #? fires at the next
completion/stabilization point of its line; no-op lines complete
immediately), armed by echo plus non-alt-screen instead of any whitelist,
with quote-parity suppression. Bracketed-paste (S1) is dropped as a
load-bearing signal in favor of output stabilization with escalating
debounce, a prompt-shape modulator, and a bounded max-wait; shell OSC
command_end stays authoritative with no short timeout. Pending-trigger
interaction is defined (delayed receipt, Ctrl+C and bare-Esc cancel, FIFO
serialization) along with accepted line-semantics corners (background
commands, REPL multi-line, the "#? /" command namespace). The prototype
still implements the earlier S1+S3 iteration; the deltas are listed in the
doc.

New design-0002 records the AI-stage boundaries: the mirror consumes the
full byte stream written to the user's terminal (PTY plus AI presentation
output), presentation buffers whichever side is bounded, the context
package pushes the question anchor plus a pullable-material inventory
while deeper history stays behind tools, and the daemon holds one
conversation per terminal session. architecture.md and rewrite-0001 are
annotated to match (in-program #? detection supersedes the
OSC-markers-only decision).

Co-Authored-By: Claude Fable 5 <[email protected]>
Replace the S1+S3 prototype with the stabilization-based design of
design-0001: mirror-read capture of the cursor's logical line at submit
(doubling as the echo-verification arming check), quote-parity
suppression, escalating debounce tiers with a prompt-shape modulator and
a bounded max-wait, still-running stabilization firing for
non-terminating commands, and completion metadata (form, completion
authority, stillRunning, exit code) in the context package. The
pending-trigger interaction lands too: a ~1s delayed receipt notice,
Ctrl+C cancel with command_end re-fire suppression, and bare-Esc cancel
with a 40ms continuation-timeout disambiguation in the stdin thread.
Presentation output now feeds the terminal mirror (the design-0002
mirror-feed invariant, for the lines that exist today).

Dogfooding immediately caught a false positive through fzf's Ctrl+R
widget (rendered `#?` UI text captured at the prompt, double-firing at
command_end), so the prompt layer is marker-owned: mirror capture is
armed only inside command spans and in shells without integration, and
the command_start marker now carries the full typed line (zsh preexec
already does; the bash DEBUG trap reads it back from history), emitted
once per accepted line. Real-PTY regressions cover echo arming, quote
parity, still-running single-fire, and both cancel paths.

Co-Authored-By: Claude Fable 5 <[email protected]>
@sjdhome sjdhome merged commit e0e6218 into main Jul 2, 2026
4 checks passed
@sjdhome sjdhome deleted the rewrite/hybrid-rust-node branch July 2, 2026 06:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant