Sentinel: discover and recover without blocking thread-pool workers - #3301

What we saw in production

A .NET 9 service on 6 CPUs froze for 0.5–5 s at a time: liveness probes (an endpoint with no I/O) took up to 4.9 s, and readiness failed ~15 times a day on one pod. It connects in Sentinel mode (ServiceName) through libraries that create their own multiplexers (DotNetCore.CAP's Redis Streams transport and the SignalR Redis backplane): up to 11 in one process, each with its own reconnect timer.

An EventPipe trace (counters, GC/contention/thread pool events, then a continuous sample profile) ruled out GC (max pause 224 ms) and locks (2.1 s of contention in 15 min, none ≥ 200 ms). During every freeze the process CPU was ~0, the pool ran 0–5 work items/s instead of thousands, and a ThreadPool Starvation event followed within ~1 s. In each freeze window all 6–8 busy pool threads were in:

TimerQueueTimer.Fire
 → OnManagedConnectionFailed (reconnect timer)
 → SwitchPrimary → GetConfiguredPrimaryForService
 → PLINQ First() → RedisServer.ExecuteSync   // SENTINEL get-master-addr-by-name

The timer was active in 283 of the 300 traced seconds.

The problem: in Sentinel mode the reconnect timer can run forever, and each tick blocks one pool thread per sentinel endpoint on synchronous calls whose replies need a pool thread themselves. With several multiplexers in one process this starves the thread pool and freezes the whole application for seconds. This PR removes the blocking from that path and stops the timer once the connection is back on the primary that sentinel reports.

Mechanism

1. Any ConnectionFailed on a managed connection starts the reconnect timer: every second, until ConnectionRestored. If no connection of that multiplexer is established again (a replica that came back at a new address is never added, because the switch only rebuilds for an unknown primary), or if Restored is handled before Failed (both go through the pool, in no guaranteed order), it never stops.
2. Each tick asks every sentinel through PLINQ with synchronous commands, one pool thread per sentinel endpoint. Configured DNS names and the IPs from SENTINEL sentinels are not merged, so 3 sentinels are 6 endpoints, and on a 6-CPU box that is the whole pool.
3. Socket completions are pool work items too, so the replies cannot be read until the pool injects a thread (~1 s per injection). The sentinels are not slow; the process blocks itself. A freeze lasts until enough threads have been injected.

ThreadPool.SetMinThreads(64) only masks it: with 11 multiplexers, 11 × 6 blocked threads exceed 64 and the freezes return (up to 2.55 s on the test bench).

This is the same starvation as #2790, which was closed as likely fixed by the 3.0 IO rewrite. It still reproduces on 3.4.20. The serviceName = "sentinel" in that issue's stack is a second bug, fixed here too (commit 3).

Fix

Four commits, each builds and tests on its own:

1. No blocking in the background path. Sentinel is queried with the async API, all sentinels at once, one deadline, first valid answer wins (Step 1 tries sentinels one by one with a short timeout each; one deadline for all keeps that bound for the whole round). SwitchPrimaryAsync has a per-connection gate, and a switch requested during another one is replayed. The reconnect timer is one-shot (next attempt one second after the previous finished) and stops on state: connected to the sentinel-reported primary, which is not a replica, with no other connected primary. The public sync Connect is the only synchronous wait left.
   Spec: Redis Step 1, Step 2, reconnections; Valkey Step 1, Step 2, reconnections.
2. Take only a healthy primary answer. With every sentinel racing, the answer must be checked: SENTINEL MASTER with master and no s_down/o_down (get-master-addr-by-name only when SENTINEL MASTER is unsupported); an answer flagged down is treated like the null reply in Step 2, and the next sentinel's answer is used. Replicas are read from the same sentinel, every switch refreshes the replica set, and live nodes are not retired on a partial view.
   Spec: Redis Step 2, replicas, failover disconnection; Valkey Step 2, replicas, failover disconnection.
3. Sentinel list. +sentinel passed the literal sentinel as the service name, so it never worked; the service is now read after @. The list stays append-only, as the spec describes, discoveries are not lost when a reconfigure is busy, and routes sentinel reports down are asked last.
   Spec: Redis Step 1, list refresh, events; Valkey Step 1, list refresh, events.
4. READONLY on a sentinel-managed connection marks the node as a replica immediately (no reconfigure on the IO path); the demotion heartbeat from Sentinel: re-resolve the primary when it is demoted on a healthy connection #3281 then asks sentinel. This covers the demoted primary from "failover disconnection" when the connection was not killed.
   Spec: Redis Step 3, failover disconnection; Valkey Step 3, failover disconnection.

Behaviour that changes beyond the starvation:

• ConnectionRestored no longer runs a switch when the connection is already on a healthy sentinel-reported primary;
• the fastest valid sentinel answer wins, instead of the first sentinel in the list;
• the sentinel list is also refreshed after the first switch of a managed connection.

Results

Test bench with the production shape (6 sentinel endpoints: 3 DNS names plus 3 announced IPs, default pool minimum, 6 CPUs), 180 s per run. The metric is the delay of a probe queued to the pool every 50 ms, with async SET/GET load. N = number of sentinel-managed multiplexers.

No episode over 500 ms in any fixed run. Production results will follow as a comment once there is enough data.

Tests

• In-process sentinel tests (SentinelPrimaryDemotionUnitTests, SentinelDiscoveryUnitTests): event order, timer spacing, replayed switches, dispose during recovery, SENTINEL MASTER flags and the legacy fallback, replicas from the same sentinel, partial views, the sentinel list, +sentinel parsing, READONLY, malformed addresses.
• All Sentinel* tests against the docker topology with RunLongRunning, plus ManagedPrimaryConnectionEndToEndWithFailoverTest three times. SentinelDiscoveryKeepsConfiguredNamesAndPreviouslyKnownRoutes now checks that the list does not shrink; it asserted that it does not change, which races on which sentinel answers first (a sentinel never lists itself).

Refs #2790

Checklist

• I fully and freely contribute this code in accordance with the project license (and am legally able to do so)
• I take responsibility for this contribution's quality and correctness, including any portions produced with AI assistance (see CONTRIBUTING.md).