Dropped connection: some commands fail with TaskCanceledException (no token) or a raw IOException / InvalidOperationException instead of RedisConnectionException #3283

Summary

When connections are killed under load, almost every affected command fails with RedisConnectionException. A small number fail with something else instead:

• TaskCanceledException ("A task was canceled."), although no CancellationToken is involved anywhere (the API does not take one).
  • Callers that treat OperationCanceledException as "the caller cancelled" then report a Redis outage as a cancellation. For example, ASP.NET Core logs it as an aborted request, and a background loop may stop.
• A raw InvalidOperationException "Output has been completed with fault: Unable to write data to the transport connection …", with an inner IOException → SocketException, thrown from the RESPite write buffer.
• A raw IOException with an inner SocketException.

Seen with 3.0.17 and still with 3.4.5. On 3.4.5 the cancellations are more frequent than on 3.0.17.

Environment

• StackExchange.Redis 3.0.17 and 3.4.5, default protocol (RESP2), BacklogPolicy.FailFast, AsyncTimeout / SyncTimeout = 1000 ms
• Redis 7.4 (redis:7-alpine Docker image), standalone
• .NET 10 (SDK 10.0.301); seen on Linux (GitHub Actions ubuntu-latest) and on Windows 11
• Connections dropped with CLIENT KILL TYPE normal SKIPME yes from a separate admin multiplexer, while about 400 single commands are in flight

Repro

repro.csproj:

<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>net10.0</TargetFramework>
    <Nullable>enable</Nullable>
    <ImplicitUsings>enable</ImplicitUsings>
    <SERedisVersion Condition="'$(SERedisVersion)' == ''">3.4.5</SERedisVersion>
  </PropertyGroup>
  <ItemGroup>
    <PackageReference Include="StackExchange.Redis" Version="$(SERedisVersion)" />
  </ItemGroup>
</Project>

Program.cs:

// On a dropped connection some commands fail with TaskCanceledException (no token involved) or with a raw
// IOException / InvalidOperationException instead of RedisConnectionException.
// usage: dotnet run -c Release -- <host:port> [seconds]
using System.Diagnostics;
using System.Reflection;
using StackExchange.Redis;

var endpoint = args.Length > 0 ? args[0] : "localhost:6379";
var seconds = args.Length > 1 ? int.Parse(args[1]) : 60;

var options = ConfigurationOptions.Parse(endpoint);
options.AbortOnConnectFail = false;
options.AsyncTimeout = 1000;
options.SyncTimeout = 1000;
options.BacklogPolicy = BacklogPolicy.FailFast;
await using var mux = await ConnectionMultiplexer.ConnectAsync(options);
await using var admin = await ConnectionMultiplexer.ConnectAsync(endpoint + ",allowAdmin=true,abortConnect=false");
var server = admin.GetServer(admin.GetEndPoints()[0]);
var db = mux.GetDatabase();
Console.WriteLine("StackExchange.Redis " + typeof(ConnectionMultiplexer).Assembly.GetCustomAttribute<AssemblyInformationalVersionAttribute>()?.InformationalVersion);

var outcomes = new SortedDictionary<string, int>();
var firstOfEach = new Dictionary<string, Exception>();
var watch = Stopwatch.StartNew();
while (watch.Elapsed < TimeSpan.FromSeconds(seconds))
{
    var calls = new List<Task>();
    for (var i = 0; i < 200; i++) calls.Add(Call(i));
    try { await server.ExecuteAsync("CLIENT", "KILL", "TYPE", "normal", "SKIPME", "yes"); }
    catch (RedisException) { /* the admin connection may be caught by the kill too */ }
    for (var i = 200; i < 400; i++) calls.Add(Call(i));

    foreach (var call in calls)
    {
        string outcome;
        try
        {
            await call; // no CancellationToken anywhere
            outcome = "ok";
        }
        catch (Exception ex)
        {
            outcome = ex.GetType().Name;
            if (ex is not RedisConnectionException and not RedisTimeoutException) firstOfEach.TryAdd(outcome, ex);
        }
        outcomes[outcome] = outcomes.GetValueOrDefault(outcome) + 1;
    }
}

foreach (var (outcome, count) in outcomes) Console.WriteLine($"{outcome}: {count}");
foreach (var (type, ex) in firstOfEach) Console.WriteLine($"--- first {type}:{Environment.NewLine}{ex}");
return firstOfEach.Count == 0 ? 0 : 1;

Task Call(int i) => (i % 3) switch
{
    0 => db.StringGetAsync($"repro:cancel:{i % 10}"),
    1 => db.SortedSetRangeByScoreAsync($"repro:cancel:z{i % 10}"),
    _ => db.StringIncrementAsync($"repro:cancel:{i % 10}"),
};

docker run -d --rm -p 6379:6379 redis:7-alpine
dotnet run -c Release -- localhost:6379 180

Expected

Every failure caused by the dropped connection is a RedisConnectionException, with the transport exception as InnerException, or a RedisTimeoutException. A TaskCanceledException should only appear when a caller-supplied token is cancelled, and there is none here.

Actual

3.4.5, one of two 180 s runs on Windows 11 (the other: 24 TaskCanceledException, 0 InvalidOperationException):

StackExchange.Redis 3.4.5+d59ff9ef16
InvalidOperationException: 39
ok: 21452
RedisConnectionException: 9482482
TaskCanceledException: 27
--- first TaskCanceledException:
System.Threading.Tasks.TaskCanceledException: A task was canceled.
   at Program.<Main>$(String[] args) in ...\Program.cs:line 37
--- first InvalidOperationException:
System.InvalidOperationException: Output has been completed with fault: Unable to write data to the transport connection: An established connection was aborted by the software in your host machine..
 ---> System.IO.IOException: Unable to write data to the transport connection: An established connection was aborted by the software in your host machine..
 ---> System.Net.Sockets.SocketException (10053): An established connection was aborted by the software in your host machine.
   at RESPite.Streams.SwitchableBufferedStreamWriter.CopyOutAsync(Boolean alreadyActive) in /_/src/RESPite/Streams/BufferedStreamWriter.Switchable.cs:line 218
   --- End of inner exception stack trace ---
   at RESPite.Streams.CycleBufferStreamWriter.ThrowCompleteOrFaulted() in /_/src/RESPite/Streams/BufferedStreamWriter.cs:line 287
   at StackExchange.Redis.Message.CommandKeyMessage.WriteImpl(MessageWriter& writer) in /_/src/StackExchange.Redis/Message.cs:line 1272
   at StackExchange.Redis.Message.WriteTo(PhysicalConnection physical) in /_/src/StackExchange.Redis/Message.cs:line 990
   at StackExchange.Redis.PhysicalBridge.WriteMessageToServerInsideWriteLock(PhysicalConnection connection, Message message) in /_/src/StackExchange.Redis/PhysicalBridge.cs:line 1736

Observed frequencies

The counts below come from a heavier workload than the repro: reads (GET, and ZREMRANGEBYSCORE + ZRANGEBYSCORE) mixed with 4-command writes, 400 commands per kill. They count failed read commands only.

Analysis (3.4.5 source)

I have not instrumented which path produced each occurrence. From reading the source, these look like the likely sources.

1. PhysicalConnection.RecordConnectionFailed (PhysicalConnection.cs ~L549) fails the drained messages with whatever exception it was given.

   • outerException starts as innerException. It is replaced by a RedisConnectionException only inside if (isCurrent && Interlocked.CompareExchange(ref failureReported, 1, 0) == 0) (~L593 / ~L681).
   • The messages drained from _writtenAwaitingResponse are then failed with var ex = innerException is RedisException ? innerException : outerException; (~L698 → RecordMessageFailed → next.SetExceptionAndComplete(ex!, …), ~L742).
   • When that branch is not taken, the drained messages get the raw inner exception (IOException, or the write pipe's InvalidOperationException). This happens on a second report for the same connection (for example the read loop and the write path both report it), or when the bridge has already moved to a new physical connection, so isCurrent is false.
   • If innerException is null (for example RecordConnectionFailed(ConnectionFailureType.SocketClosed) from the read loop on EOF), they get null. TaskResultBox<T>.SetException(null) turns that into SimpleResultBox.CancelledException, a TaskCanceledException (ResultBox.cs ~L130).

2. A write failure races the drain.

   • In WriteMessageToServerInsideWriteLock, the message is enqueued (EnqueueInsideWriteLock, isQueued = true) before WriteTo throws.
   • The catch calls message.Fail(...), which sets a wrapped RedisConnectionException, and then connection.RecordConnectionFailed(failureType, ex).
   • That drains the same message again with the raw ex (point 1). The result box keeps only the last exception set, so the raw one can win. This matches the stack trace above.

3. message.Cancel() in PhysicalBridge (FailDueToNoConnection, TimedOutBeforeWrite, and the "anything else goes in the bin" path, ~L196 / ~L204 / ~L1365) completes the message as cancelled.

   • On the async path, ExecuteAsyncImpl then also tries to fault it with GetException(result, …), but the box may already be completed as cancelled.

Suggested direction: always fail drained or unwritable messages with a RedisConnectionException that carries the original as InnerException, as the first report already does. Never fail them with null or Cancel() unless a caller token is involved.

Workaround

Wrap the awaits:

• a TaskCanceledException without a cancelled caller token → RedisConnectionException;
• an IOException / SocketException, or an InvalidOperationException wrapping one → RedisConnectionException.

Related: #3282