Skip to content

ConnectionMultiplexer consistently timing out while other servers work fine #2888

Closed

Description

@Issung

We are having an issue with one of our services. It seems the connection multiplexer keeps breaking. It was fine once, we just recylced that pod. But it has kept coming back.

We have 1 cluster (in AWS) with 2 nodes (1 replica).
We keep 1 connection multiplexer alive for the entire pod lifetime, stored in dependency injection.
We have 4 pods (servers) all using the same Redis, but only 1 is getting the timeouts, on every single call.
So I believe the connection multiplexer is breaking, but why could this be? And what is the best path of action to take? Dispose and make a new one? We don't really see this problem on any other services that have their own Redis instances.

StackExchange.Redis.RedisTimeoutException: Timeout awaiting response (outbound=0KiB, inbound=1KiB, 5352ms elapsed, timeout is 5000ms), command=INFO, next: INFO, inst: 0, qu: 0, qs: 1, aw: False, bw: SpinningDown, rs: ReadAsync, ws: Idle, in: 0, in-pipe: 0, out-pipe: 0, last-in: 1396, cur-in: 0, sync-ops: 5182, async-ops: 7745870, serverEndpoint: redacted.0001.usw2.cache.amazonaws.com:6379, conn-sec: 337520.93, aoc: 1, mc: 1/1/0, mgr: 10 of 10 available, clientName: redacted-queue-development-849c7d7985-f6649(SE.Redis-v2.8.16.12844), IOCP: (Busy=0,Free=1000,Min=1,Max=1000), WORKER: (Busy=3,Free=32764,Min=16,Max=32767), POOL: (Threads=6,QueuedItems=0,CompletedItems=195372535,Timers=251), v: 2.8.16.12844 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts)
   at Redacted.Service.Diagnostics.RedisMonitor.Monitor() in /src/Redacted.Service/Diagnostics/RedisMonitor.cs:line 29
   at Redacted.Service.Diagnostics.MonitorBase`1.Monitor(CancellationToken stoppingToken) in /src/Redacted.Service/Diagnostics/MonitorBase.cs:line 58
Redis-Message: INFO
Redis-Message-Next: INFO
Redis-OpsSinceLastHeartbeat: 0
Redis-Queue-Awaiting-Write: 0
Redis-Queue-Awaiting-Response: 1
Redis-Active-Writer: False
Redis-Backlog-Writer: SpinningDown
Redis-Read-State: ReadAsync
Redis-Write-State: Idle
Redis-Inbound-Bytes: 0
Redis-Inbound-Pipe-Bytes: 0
Redis-Outbound-Pipe-Bytes: 0
Redis-Last-Result-Bytes: 1396
Redis-Inbound-Buffer-Bytes: 0
Redis-Sync-Ops: 5182
Redis-Async-Ops: 7745870
Redis-Server-Endpoint: redacted.0001.usw2.cache.amazonaws.com:6379
Redis-Server-Connected-Seconds: 337520.93
Redis-Abort-On-Connect: 1
Redis-Multiplexer-Connects: 1/1/0
Redis-Manager: 10 of 10 available
Redis-Client-Name: redacted-queue-development-849c7d7985-f6649(SE.Redis-v2.8.16.12844)
Redis-ThreadPool-IO-Completion: (Busy=0,Free=1000,Min=1,Max=1000)
Redis-ThreadPool-Workers: (Busy=3,Free=32764,Min=16,Max=32767)
Redis-ThreadPool-Items: (Threads=6,QueuedItems=0,CompletedItems=195372535,Timers=251)
Redis-Busy-Workers: 3
Redis-Version: 2.8.16.12844
redis-command: INFO
request-sent-status: Sent
redis-server: redacted.0001.usw2.cache.amazonaws.com:6379
Commandstatus: Sent

I believe the timeouts aren't valid because they only happen on 1 pod and the Redis is relatively underutilised
Image

Activity

  1. mgravell commented on Jul 22, 2026

    @mgravell
    Collaborator

    Thanks for the report. We believe this is very likely addressed by the low-level IO rewrite in StackExchange.Redis 3.0 (a new RESPite-based core that replaces Pipelines.Sockets.Unofficial and the old backlog/write loop).

    Closing on that basis — but we'd value confirmation: if you can still reproduce on 3.0.x, please comment with details and we'll reopen and dig in.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions