feat: one circuit-breaker state machine, and hidden characters in requests - #31
Merged
Merged
Conversation
…uests The AI gateway's route health and the MCP gateway's server breaker each carried their own state machine, and the desktop gateway a third. They now all run thinkwatch-core's tw-breaker; what differs between them — where the state lives, what trips it — stays with each. Route health (Redis, shared by replicas, tripped by an error rate): - One Lua call inserts the sample, trims and tallies the window and reads the breaker; the transition is computed from that tally with the shared machine and written back only when it changes, by compare-and-set, so two replicas that read the same state cannot both write. - **A tripped route came back only when its Redis key expired.** It moved from open to half-open only when a request on it completed, and an open route is never picked. A cooled breaker now reads as half-open, the next request probes it, and a success closes it. The new integration test fails on dev with "All routes failed" after the cooldown. - The router and the route-health page read one `CircuitBreakerConfig`. With the breaker disabled, a route reads as closed instead of keeping whatever state it last had. MCP breaker: the same machine in process, keyed by server id, transitions mirrored to the dashboard registry as before. Its API is synchronous now. Dashboard: **every AI provider read `Closed`** — the registry it looked in is process-local and only the MCP breaker wrote to it. AI rows now read their routes' real state from Redis, a provider showing its worst route. Hidden characters: Unicode tag characters carry an instruction invisibly into the model's context, and bidi overrides make text read differently on screen than it is. `security.hidden_text` (off / log / warn / block, default warn) checks the caller's messages and the tool results inside them — where a fetched page smuggles one in — using tw-guard's scanner. Only those two kinds are flagged: zero-width joiners make emoji, Persian needs the non-joiner, Cyrillic is Russian. Warn writes `gateway.hidden_text_flagged` to the audit log; block refuses with 403 and writes `gateway.hidden_text_blocked`. The security page gets a card. Pins core v0.40.0. Co-Authored-By: Claude Opus 5.5 <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The AI gateway's route health, the MCP gateway's server breaker and the desktop gateway each carried their own circuit-breaker state machine. They now all run thinkwatch-core's tw-breaker (v0.40.0, ThinkWatchProject/ThinkWatch-Core#155). Where the state lives and what trips it stay with each caller.
Route health (Redis, shared by replicas, tripped by an error rate)
dev, the new integration test gets502 All routes failed for modelafter the cooldown; here it passes.CircuitBreakerConfig. With the breaker disabled, a route reads as closed instead of keeping whatever state it last had.RouteHealth.stateis the sharedState; its JSON (closed/open/half_open) is unchanged.MCP breaker
The same machine, in process, keyed by server id. Transitions are mirrored to the dashboard registry as before, and a rename shows on the next state change. Its API is now synchronous (no await on a mutex), and its callers were updated.
Dashboard
Fixed: every AI provider read
Closed. The registry it looked in is process-local, and only the MCP breaker wrote to it. AI rows now read their routes' real state from Redis; a provider shows its worst route. The registry holds the sharedStatedirectly, and its doc now says what it holds.Hidden characters in requests
security.hidden_text(off / log / warn / block, default warn) checks the caller's messages and the tool results inside them, which is where a fetched page smuggles one in. It uses tw-guard's scanner.gateway.hidden_text_flaggedto the audit log. Block refuses with 403 and writesgateway.hidden_text_blocked.Verification
cargo fmt --checkandcargo clippy --workspace --all-targets -D warningsare clean.cargo nextest run --workspace --lib --bins --tests: 644 passed.tsc -b, eslint,check-i18nand vitest (103 passed).a_tripped_route_is_probed_again_once_the_cooldown_is_over(fails ondev);the_dashboard_shows_a_tripped_ai_provider_as_open;dev's baseline; the 237 include the 6 new tests.No release.
🤖 Generated with Claude Code