Skip to content

feat: one circuit-breaker state machine, and hidden characters in requests - #31

Merged
fylorn merged 1 commit into
devfrom
refactor/shared-breaker
Sep 24, 2026
Merged

fylorn merged 1 commit into
devfrom
refactor/shared-breaker

Conversation

@fylorn

@fylorn fylorn commented Sep 24, 2026

Copy link
Copy Markdown
Contributor

The AI gateway's route health, the MCP gateway's server breaker and the desktop gateway each carried their own circuit-breaker state machine. They now all run thinkwatch-core's tw-breaker (v0.40.0, ThinkWatchProject/ThinkWatch-Core#155). Where the state lives and what trips it stay with each caller.

Route health (Redis, shared by replicas, tripped by an error rate)

  • One Lua call inserts the sample, trims and tallies the window, and reads the breaker. The transition is computed from that tally with the shared machine. It is written back only when the breaker changes, by compare-and-set, so two replicas that read the same state cannot both write.
  • Fixed: a tripped route came back only when its Redis key expired.
    • A route moved from open to half-open only when a request on it completed, but an open route is never picked.
    • Now a cooled breaker reads as half-open, the next request probes it, and a success closes it.
    • On dev, the new integration test gets 502 All routes failed for model after the cooldown; here it passes.
  • The router and the route-health page read one CircuitBreakerConfig. With the breaker disabled, a route reads as closed instead of keeping whatever state it last had.
  • RouteHealth.state is the shared State; its JSON (closed / open / half_open) is unchanged.

MCP breaker

The same machine, in process, keyed by server id. Transitions are mirrored to the dashboard registry as before, and a rename shows on the next state change. Its API is now synchronous (no await on a mutex), and its callers were updated.

Dashboard

Fixed: every AI provider read Closed. The registry it looked in is process-local, and only the MCP breaker wrote to it. AI rows now read their routes' real state from Redis; a provider shows its worst route. The registry holds the shared State directly, and its doc now says what it holds.

Hidden characters in requests

  • Unicode tag characters carry an instruction invisibly into the model's context, and bidi overrides make text read differently on screen than it is.
  • security.hidden_text (off / log / warn / block, default warn) checks the caller's messages and the tool results inside them, which is where a fetched page smuggles one in. It uses tw-guard's scanner.
  • Only those two kinds are flagged. Zero-width joiners build emoji, Persian needs the non-joiner, and Cyrillic is Russian.
  • Warn writes gateway.hidden_text_flagged to the audit log. Block refuses with 403 and writes gateway.hidden_text_blocked.
  • The security page gets a card, in en and zh.

Verification

  • cargo fmt --check and cargo clippy --workspace --all-targets -D warnings are clean.
  • cargo nextest run --workspace --lib --bins --tests: 644 passed.
  • Web: tsc -b, eslint, check-i18n and vitest (103 passed).
  • New integration tests:
    • a_tripped_route_is_probed_again_once_the_cooldown_is_over (fails on dev);
    • the_dashboard_shows_a_tripped_ai_provider_as_open;
    • hidden text: block refuses a tool result with tag characters without reaching the upstream; warn is the default and writes the audit row; emoji, Russian, Persian and Arabic pass even in block mode; an unknown setting value is refused.
  • Full integration suite on the final code: 237 passed, 22 failed. The 22 are identical to dev's baseline; the 237 include the 6 new tests.
  • UI: the hidden-characters card was checked in a local preview (zh) with real clicks.

No release.

🤖 Generated with Claude Code

…uests

The AI gateway's route health and the MCP gateway's server breaker each
carried their own state machine, and the desktop gateway a third. They
now all run thinkwatch-core's tw-breaker; what differs between them —
where the state lives, what trips it — stays with each.

Route health (Redis, shared by replicas, tripped by an error rate):

- One Lua call inserts the sample, trims and tallies the window and reads
  the breaker; the transition is computed from that tally with the shared
  machine and written back only when it changes, by compare-and-set, so
  two replicas that read the same state cannot both write.
- **A tripped route came back only when its Redis key expired.** It moved
  from open to half-open only when a request on it completed, and an open
  route is never picked. A cooled breaker now reads as half-open, the
  next request probes it, and a success closes it. The new integration
  test fails on dev with "All routes failed" after the cooldown.
- The router and the route-health page read one `CircuitBreakerConfig`.
  With the breaker disabled, a route reads as closed instead of keeping
  whatever state it last had.

MCP breaker: the same machine in process, keyed by server id, transitions
mirrored to the dashboard registry as before. Its API is synchronous now.

Dashboard: **every AI provider read `Closed`** — the registry it looked
in is process-local and only the MCP breaker wrote to it. AI rows now
read their routes' real state from Redis, a provider showing its worst
route.

Hidden characters: Unicode tag characters carry an instruction invisibly
into the model's context, and bidi overrides make text read differently
on screen than it is. `security.hidden_text` (off / log / warn / block,
default warn) checks the caller's messages and the tool results inside
them — where a fetched page smuggles one in — using tw-guard's scanner.
Only those two kinds are flagged: zero-width joiners make emoji, Persian
needs the non-joiner, Cyrillic is Russian. Warn writes
`gateway.hidden_text_flagged` to the audit log; block refuses with 403
and writes `gateway.hidden_text_blocked`. The security page gets a card.

Pins core v0.40.0.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
@fylorn
fylorn merged commit 05f52e8 into dev Sep 24, 2026
@fylorn
fylorn deleted the refactor/shared-breaker branch September 24, 2026 03:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant