feat!: forward what can be forwarded, convert only what must be - #26

The cache key was model + messages + max_tokens, hashed. Everything
else a caller sends — tools, tool_choice, top_p, stop, seed,
response_format — was left out, so two requests differing only there
landed in the same slot and the second got the first one's answer.

It has not bitten yet only because tools never reached an upstream:
the provider adapters dropped them (ThinkWatch-Core#50). The moment
the conversion layer is fixed, "same question, different tools" turns
into a served tool call for the wrong tool.

The key is now a fingerprint of the entire request. `extra` is
flattened into ChatCompletionRequest, so every field the caller sent is
in the bytes by construction — there is no list of fields to forget to
extend. `stream` is cleared first: it changes framing, not the answer.

It is still computed after redaction, which is right: the stored
response carries placeholders and each caller restores with their own
context, so two callers asking the same thing about their own e-mail
share one slot and each gets their own value back.

`request_for_cache` in the post-invoke snapshot becomes
`cache_fingerprint` — the deps carried the whole request only for the
key to pick three fields back out of it.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

Groundwork for moving the handlers onto tw-dialect. Nothing calls these
from a request path yet; each has tests and each is the piece a handler
needs once it holds an intermediate representation instead of a
ChatCompletionRequest.

`proxy::transport` sends a converted request. A Prepared already is
the bytes the upstream should see, so all that is left is spelling the
URL — the dialect's path for most, a deployment in the URL for Azure,
a region-derived host plus a SigV4 signature for Bedrock. Signing
happens last, over the final body.

`ContentFilter::check_request` and `PiiRedactor::redact_request` do
what their Value-based siblings do, over a structure that is known
rather than guessed. The guessing versions look for a `text` field on
array elements and so never see what sits inside a tool result — which
is where an injected instruction, or a customer's data pulled in by a
tool, actually lives. Both new entry points recurse into it.

The system prompt is deliberately not redacted: it is written by the
operator, not typed by the caller, and redacting it rewrites the
operator's instructions.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

Collapsing get/set onto a fingerprint dropped the temperature check that
used to open both of them. Requests sampled at a nonzero temperature
started being cached — asking for a fresh draw and getting someone
else's answer. The integration suite caught it
(temperature_nonzero_request_is_not_cached).

The check now decides whether a fingerprint exists at all. No
fingerprint, no key, nothing to look up or store — so the next refactor
of get/set cannot lose it again.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

`prefix_balancer` (routes by prompt-prefix hash for KV-cache reuse on
self-hosted backends) and `channel` (named provider endpoints with
priority and weight) are declared in lib.rs and used nowhere — not in
the gateway, not in the server, not in any test. Routing lives in
`router` and `proxy::routing`.

Both implement `DynAiProvider`, the trait the dialect migration is
retiring. Porting 813 lines of decorator that no request passes through
would be work spent on keeping dead code compiling.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

The dialect migration forwards a same-format request untouched — that
is the only way `cache_control`, server tools and metadata survive,
since the intermediate representation carries none of them. So
redaction and restoration can no longer assume a typed request and
response. These are the pieces that work on what actually goes over
the wire.

Redaction: PII is still found on the intermediate representation,
where the structure is known, and `RedactionContext::apply_to` carries
the value→placeholder mapping onto the raw request. It works on the
parsed Value rather than the bytes, because a client may send `@` as
an escape sequence; it replaces longer values first; it leaves `data`
and `bytes` alone, since changing a digit run inside base64 changes an
image, not PII.

That mapping has to be a function, so the same value now gets the same
placeholder. It also means a model no longer sees one e-mail address
as two people.

Restoration of a whole response happens on the bytes, with each
original JSON-escaped — a value containing a quote would otherwise
break the document.

Streams cannot be restored on bytes: a placeholder split across two
frames is not contiguous, the frame boundary sits in the middle of it.
`StreamShaper` works per frame on the text field of whichever format
it is, holding back an unclosed `{{` until the rest arrives, and
releasing a held tail as its own delta before the block closes rather
than after. It also puts the caller's model name back on every frame:
all formats keep it at the top level, under `message`, or under
`response`, so it needs no per-format branch.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

…e protocol enum

Three things lived in tw-provider only because the adapters did, and
none of them is about converting anything:

- the HTTP client policy: 10s to connect, 300s overall, no redirects.
  Refusing redirects is the SSRF guard — `base_url` is typed in by an
  admin, and a provider answering 302 to the metadata address would
  otherwise walk gateway traffic there.
- the mapping from an upstream status to the caller's error, including
  truncating error bodies that have carried stack traces and account
  ids.
- `UpstreamProtocol`: the strings `model_routes.upstream_protocol`
  stores, keyed on this gateway's `provider_type` values. It gains a
  mapping to the conversion layer's dialect.

The transport now sends raw bytes to a path rather than a Prepared, so
a same-format request forwarded untouched goes through the same door
as a converted one.

Bedrock's host is built from the region the provider row keeps in
`base_url`; an earlier draft treated that field as a host suffix, and
its test was written to the same wrong assumption.

Header templating uses `tw_types::substitute_template` instead of a
second hand-written copy.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

The three generation endpoints now share one pipeline, and it no
longer rebuilds every request as a chat-shaped DTO.

A request whose route speaks the caller's own format goes out as the
caller sent it — the model name changed, PII swapped for placeholders,
nothing else. A request crossing formats is decoded and re-encoded by
tw-dialect, which reports what the target cannot carry.

## What the DTO was losing

Anthropic to Anthropic, with the request Claude Code actually sends,
the upstream received:

    {"max_tokens":16,"messages":[{"content":"hi","role":"user"}],
     "model":"…","stream":false}

`system` was read with `as_str()`, which is `None` for the array form,
so Claude Code's whole system prompt was dropped. So were its tools,
`tool_choice`, `metadata` and every `cache_control` breakpoint — the
last one turning each cached prefix back into full-price input. The
`/v1/messages` and `/v1/responses` handlers also hardcoded
`extra: json!({})`, discarding everything the DTO did not model, and
the chat handler's `extra` reached the adapters only to be dropped
there (ThinkWatch-Core#50).

Same-format requests cannot go through the conversion layer either:
its intermediate representation has no place for `cache_control`,
server tools or `metadata`. Hence forwarding.

## What moved where

- The request is still decoded once, to know where the caller's text
  is. The content filter and PII detection read that — including text
  inside tool results, which the Value-guessing versions never saw. The
  found PII is carried back onto the raw request.
- A same-format request carries the caller's `anthropic-*` headers: its
  body may use a beta feature, and without the header the upstream
  refuses what used to work. Anthropic-bound requests always get
  `anthropic-version`, which the old adapter hardcoded and the API
  requires — the mock does not check it, so a test pins it.
- Responses are handled as the caller's bytes. Usage is sniffed off
  the upstream's own bytes by tw-wire, so a streamed response no longer
  keeps every chunk in memory for an accounting pass at the end — the
  field doing that was documented as unused. The model name goes back
  to the caller's alias, whole and per frame. PII is restored on a
  whole body in one pass, and per frame on the text field for a stream,
  since a placeholder split across two frames is not contiguous.
- The upstream call of a stream happens on the stream's first poll, so
  headers go out at once and a caller who leaves during the wait is
  still recorded as cancelled. A rejected dialect is retried inside
  that same call, before any byte reaches the caller — which made the
  old "peek the first item" machinery unnecessary.
- Routes share one upstream per provider instead of one adapter per
  (provider, dialect): only the format changes between alternates.
- The protocol probe encodes its request with the same layer as live
  traffic, so "the probe passed, forwarding fails" has nothing to hide
  behind.
- Output guardrails read the assistant text in whichever format the
  caller asked for; `max_length` still counts bytes, as it always has.

## Accounting

Prompt tokens are now counted the same for every upstream: plain input
plus cache reads and writes, OpenAI's definition. Anthropic's own
`input_tokens` excludes cached tokens, so routes to Anthropic will
record more prompt tokens than before for the same work. The price
model still charges every prompt token alike; pricing cache tokens
separately is a decision of its own.

## Removed

`providers/` and the tw-provider dependency, the three handlers,
`streaming.rs`, `token_counter.rs` and the character-count fallback
that used it, `redact_messages` / `restore_response`, the old
`ContentFilter::check`.

Integration suite (`make test-it`, run locally against Postgres, Redis
and ClickHouse): 225 passed, 22 failed — the same 22 that fail on dev
before this change.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

Seven files did nothing but re-export core (`crypto`, `json_secret`,
`retry`, `cb_registry` in common; `metrics_labels`, `sse_parser`,
`transform` in gateway), plus a `pub use` of `retry` in gateway's lib.
They let the tree keep compiling while code moved to core. It has
moved; every call site now names the core crate it uses, and the shims
are gone. `sse_parser` and `transform` had no callers at all, so
tw-protocol is no longer a dependency.

The core crates are declared once, in `[workspace.dependencies]`,
pinned to one tag. Each crate says `{ workspace = true }`, so a core
release is a one-line bump instead of six.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

Bedrock's ConverseStream is AWS eventstream, not SSE. The provider
adapter used to unframe it by hand; since the pipeline started
forwarding bytes, nothing did, and the converter, the usage sniffer
and the collector were all reading binary frames as if they were SSE.
A streamed Bedrock answer came out empty.

The pump now unframes a Bedrock stream at the door with core's
`tw_upstream::eventstream::Transcoder` (CRC checked, frames cut
anywhere held until whole). Everything after it reads the same SSE it
reads from every other upstream. An exception Bedrock sends mid-stream
(throttling, say) ends the stream with an error in the caller's
format, as a broken connection already did.

Core v0.35.0 also teaches the usage sniffer Converse's camelCase
counts. Before, every Bedrock call, streamed or not, found no usage
and was billed on an estimate.

Pins core v0.35.0; tw-upstream's `sigv4` feature is now `bedrock`.

Co-Authored-By: Claude Opus 5.5 <[email protected]>