Repository navigation
Proxy: support stateful chat chains (LM Studio native / OpenAI Responses API) #44
Description
Scope
Add Ragger-proxy support for stateful chat chains — backends where conversation context lives server-side and clients chain turns via `response_id` / `previous_response_id`. Applies to:
- LM Studio native: `POST /api/v1/chat`
- OpenAI Responses API: `POST /v1/responses`
Today Ragger only proxies OpenAI-compat stateless `/v1/chat/completions`. When the backend holds state, the current capture-and-inject model needs adjustments — not a rewrite, but enough that it's worth its own issue rather than sneaking into FM.
Why this is tractable
Despite being "stateful," both protocols still round-trip plaintext each turn:
- Request carries the new user message (+ `previous_response_id`)
- Response carries assistant text (+ a new `response_id`)
So Ragger continues to see both sides of every turn. What shifts is where context gets assembled and when injection happens.
What changes from the stateless path
| Concern | Stateless (today) | Stateful (this issue) |
|---|---|---|
| Turn capture | per-request body pair | unchanged — still pair user→assistant on each request |
| Chain reconstruction | N/A (full history in each request) | persist `previous_response_id → response_id` links in session rows |
| Memory injection | inject into `messages[]` per turn | inject at chain start (system prompt), or via tool-call, or via periodic refresh turn |
| Summarization trigger | context-window pressure | time / turn-count / topic-shift |
| Context visibility | full thread always | only the delta we captured — reconstruct via chain |
| Recall-quality config (#40) | per-turn decision | chain-level declaration at start |
Implementation sketch
- New format file: `formats/lmstudio-stateful.json` (and `openai-responses.json`) with upstream path, request transformer name, response extractor paths including `response_id`.
- Chain persistence: extend chat sessions schema with `upstream_response_id` + `previous_response_id` columns; link turns into a chain table.
- Proxy logic branch: when the selected endpoint's format declares `stateful: true`, skip history assembly, honor `previous_response_id` passthrough, capture `response_id` on reply.
- Memory injection mode: add `injection = start | every_turn | tool` option per endpoint; stateful endpoints default to `start`.
- Client-side stateless facade (optional): accept `/v1/chat/completions` from clients, translate to stateful chain calls upstream using Ragger's own chain tracking. Lets OC-style OpenAI-compat clients talk to stateful backends without knowing.
Acceptance
- Pointing `lm_proxy_url` at `http://host:1234/api/v1\` with `format = lmstudio-stateful` produces working chat via `POST /v1/chat/completions` or `POST /v1/responses`.
- Session rows contain chain IDs; turn history is reconstructible without querying upstream.
- Memory injection fires at chain start per config and doesn't duplicate per turn.
- Existing stateless `/v1/chat/completions` path continues to work unchanged.
Priority
After Fading Memory lands. FM on stateless OpenAI-compat is the primary target; stateful support is a compatibility expansion once the memory model is stable.
Activity
diskerror commentedon Apr 24, 2026
Architectural framing: stateful is arguably the better fit
Worth recording the rationale rather than treating stateful as just a compatibility feature. The division of labor is actually cleaner when upstream is stateful:
In the stateless path, Ragger rebuilds the full `messages[]` on every turn — duplicating work the upstream could cache, and paying context cost for history the upstream already has. Stateful flips it: Ragger injects memory once at chain start (or on triggered refresh), upstream handles turn-to-turn continuity in its own KV cache, and Ragger only steps back in when cadence says it's time to summarize or inject fresh memory.
That mapping also mirrors how humans work — working memory (the chain) is fast and local; long-term memory (Ragger) is deliberate and retrieved on demand.
Implication for scope
This issue should explicitly support both modes, not replace stateless. Per-endpoint `protocol` in #46 picks:
Clients remain unaware — they see the same OpenAI-compat `/v1/chat/completions` surface from Ragger. The protocol difference is entirely upstream-side.
Priority
Still post-FM, but elevating from "compatibility expansion" to "architecturally preferred mode where available." FM design should not assume stateless upstream — write it so memory injection policy is pluggable (start-of-chain vs per-turn) and the stateful path drops in naturally.
diskerror commentedon Apr 24, 2026
Two axes, four cells
Statefulness is really two independent choices — downstream (what Ragger exposes to clients) and upstream (what Ragger speaks to the backend). The matrix:
The interesting cell
Downstream stateful + upstream stateless is the one that's actually a new capability rather than a pass-through. Any OpenAI-compat backend (no chain support) becomes stateful-from-the-client's-perspective because Ragger holds the chain. Clients get the payload-reduction win of the Responses API shape even when the backend doesn't natively support it. That's a real differentiator.
Downstream stateful + upstream stateful is the ideal architecture — minimum data over the wire, each layer owning what it's best at — but depends on both sides supporting it.
Implication for the ongoing work
The `format`/`protocol` mechanism from #46 needs a per-direction statefulness flag, not just a single `stateful: true`. Possible shape:
```ini
[inference]
url = http://host:1234
protocol = openai # upstream protocol
stateful_upstream = false # upstream doesn't track chains
downstream exposure (server-wide, not per-endpoint):
[server]
expose_stateful = true # downstream clients can use /v1/responses
```
When `expose_stateful` is on and `stateful_upstream` is off, Ragger synthesizes chains in its own DB and reconstructs history upstream. When both are on, Ragger maps IDs. When `expose_stateful` is off, behavior is today's stateless.
Worth folding into the issue body as the design matures.
diskerror commentedon Jun 7, 2026
No proxy. Now relies on agent hooks.
diskerror commentedon Sep 5, 2026
Status note (2026-09-04): current direction is to solve this class of problem without Ragger proxying connections where possible. This issue depends on proxy/chat infrastructure that either doesn't exist yet or was removed (the old LM proxy was replaced by capture_turn — see commit 2b55ee9). Not closing — proxying remains a possible fallback if the non-proxy approach doesn't pan out — but deprioritized pending that exploration. Tagged
chat(Ragger Chat / proxy) to track this cluster.