Repository navigation
Synchronous scan timeout can leave an internal task exception unobserved in 2.13.17 #3260
Description
Summary
We observed TaskScheduler.UnobservedTaskException after a synchronous scan had already thrown a timeout that the caller caught. A representative case uses IDatabase.HashScan; the later exception includes CursorEnumerable<T>.Enumerator.AwaitedNextAsync in its stack.
The timeout during a Redis outage is expected. The additional unobserved exception from an internal task is the behavior we are reporting. We have not observed a process crash or measured a memory leak.
Environment
- StackExchange.Redis 2.13.17, informational version
2.13.17+6e0df970c7. - Windows, .NET 8.0.31.
- Six-node Redis Cluster using Redis 8.4, three primaries and three replicas in local containers.
SyncTimeoutandAsyncTimeoutwere both 1000 ms in the original reproduction.
Observed reproduction
- Connect to the healthy cluster and create a hash on a known shard.
- Stop that shard's primary and replica in the disposable test cluster.
- Enumerate
IDatabase.HashScansynchronously, catching the caller-visible exception and disposing the enumerator. - Restore the stopped nodes and keep listening for
TaskScheduler.UnobservedTaskException.
One representative run produced a caught RedisTimeoutException after approximately 1022 ms. Its library frames were:
CursorEnumerable<T>.Enumerator.ThrowTimeout
CursorEnumerable<T>.Enumerator.Wait
CursorEnumerable<T>.Enumerator.SlowNextSync
CursorEnumerable<T>.Enumerator.MoveNext
Roughly 2.5 seconds after the caught timeout, the event contained an AggregateException wrapping a RedisConnectionException with SocketClosed. This is the finalizer event timing, not a measurement of when the internal task faulted. The inner RedisConnectionException stack included:
StackExchange.Redis.CursorEnumerable`1+Enumerator+<AwaitedNextAsync>d__30.MoveNext
These are selected frames from recorded exceptions, not a complete raw stack trace.
Across seven isolated processes exercising synchronous scan overloads directly through StackExchange.Redis, six recorded this event. The remaining process received an AggregateException at the synchronous call and recorded no later event. Four asynchronous scan processes under the same outage condition and one healthy synchronous HashScan process recorded no such events. These controls do not establish that every asynchronous scan path is unaffected.
The direct-client path did not instantiate our application wrapper, although the original test harness references application assemblies for shared helpers. Every recorded event occurred during restoration, before client disposal and forced GC. We have not isolated whether restoration itself affects event timing.
Suspected mechanism
In the version-matched CursorEnumerable source, SlowNextSync converts the result of SlowNextAsync to a task. Wait throws when TryWait returns false. TryWait uses a timed Task.Wait.
This appears to leave the outer task produced by AwaitedNextAsync without a future observer if it faults after the synchronous wait times out. The enumerator's _pending scan-page task is a different task; observing only that field would not necessarily address the outer task. This mechanism is inferred from the recorded stacks and matching source, rather than a deterministic scheduler test.
Expected behavior: preserve the caller-visible timeout while ensuring that a later fault of the internal outer task is observed.
Smaller reproduction candidate
The repro/Program.cs and repro/ScanTimeoutRepro.csproj included below use only StackExchange.Redis. They build successfully for .NET 8 with 2.13.17, but this smaller program has not yet been run against Redis.
It proposes CLIENT PAUSE 6000 ALL on a disposable standalone instance, with SyncTimeout=250 and AsyncTimeout=1500, to separate the synchronous timeout from the later asynchronous fault. This is a different fault mechanism and configuration from the confirmed cluster reproduction above; its success must not be assumed. repro/README.md includes commands, controls, cleanup, and result interpretation.
Could you investigate this behavior in StackExchange.Redis 2.13.17 and correct the synchronous timeout path so that a later fault of this internal outer task is observed? We can validate a proposed fix against the original cluster scenario.
Standalone candidate source and instructions
repro/Program.cs
using System.Diagnostics;
using System.Reflection;
using System.Runtime.CompilerServices;
using System.Runtime.InteropServices;
using StackExchange.Redis;
// Candidate reduction: use only a disposable Redis instance on loopback.
// CLIENT PAUSE ALL temporarily affects every client of that instance.
internal static class Program
{
private const int PauseMilliseconds = 6000;
private static int scanEvents;
private static int otherEvents;
private static async Task<int> Main(string[] args)
{
if (args.Length != 3 || !int.TryParse(args[0], out int port) || port is < 1 or > 65535
|| args[1] is not ("sync" or "async" or "healthy") || args[2] != "--disposable-instance")
{
Console.Error.WriteLine("Usage: <port> <sync|async|healthy> --disposable-instance");
return 2;
}
Console.WriteLine($"Runtime: {RuntimeInformation.FrameworkDescription}");
Console.WriteLine($"OS: {RuntimeInformation.OSDescription}");
Console.WriteLine($"Package: {typeof(ConnectionMultiplexer).Assembly.GetCustomAttribute<AssemblyInformationalVersionAttribute>()?.InformationalVersion}");
Console.WriteLine($"Mode: {args[1]}; SyncTimeout=250; AsyncTimeout=1500; pause={PauseMilliseconds}");
TaskScheduler.UnobservedTaskException += OnUnobserved;
RedisKey key = $"scan-timeout-repro:{Guid.NewGuid():N}";
ConnectionMultiplexer? control = null;
ConnectionMultiplexer? subject = null;
Stopwatch? pauseClock = null;
bool callerTimedOut = false;
bool cleanupSucceeded = false;
int exitCode = 2;
try
{
control = await ConnectionMultiplexer.ConnectAsync(Options(port, 5000, 5000));
subject = await ConnectionMultiplexer.ConnectAsync(Options(port, 250, 1500));
var admin = control.GetDatabase();
await admin.HashSetAsync(key, Enumerable.Range(0, 1024)
.Select(i => new HashEntry($"field:{i}", i)).ToArray());
await admin.KeyExpireAsync(key, TimeSpan.FromMinutes(2));
await subject.GetDatabase().PingAsync();
if (args[1] != "healthy")
{
pauseClock = Stopwatch.StartNew();
await admin.ExecuteAsync("CLIENT", "PAUSE", PauseMilliseconds, "ALL");
}
// Both entry points catch their caller-visible exception and dispose the enumerator.
callerTimedOut = args[1] == "async"
? await ScanAsync(subject.GetDatabase(), key)
: ScanSync(subject.GetDatabase(), key);
// Let the asynchronous command timeout occur before making unreachable tasks collectible.
// Forced GC only exposes the event; it is not a production recommendation.
await Task.Delay(PauseMilliseconds + 2000);
for (int i = 0; i < 5; i++)
{
GC.Collect();
GC.WaitForPendingFinalizers();
GC.Collect();
await Task.Delay(200);
}
int count = Volatile.Read(ref scanEvents);
Console.WriteLine($"Observation: callerTimeout={callerTimedOut}; scanEvents={count}; otherEvents={Volatile.Read(ref otherEvents)}");
// 0 means the target symptom was observed, not that the library passed a test.
exitCode = callerTimedOut && count > 0 ? 0 : 1;
}
catch (Exception ex)
{
Console.Error.WriteLine($"Setup/observation error: {ex}");
}
finally
{
// Do not race cleanup against an outstanding server-wide pause.
if (pauseClock is not null)
{
int remaining = PauseMilliseconds + 500 - (int)pauseClock.ElapsedMilliseconds;
if (remaining > 0) await Task.Delay(remaining);
}
try
{
if (control is not null)
{
await control.GetDatabase().KeyDeleteAsync(key);
cleanupSucceeded = !await control.GetDatabase().KeyExistsAsync(key);
}
}
catch (Exception ex)
{
Console.Error.WriteLine($"Cleanup error: {ex.GetType().Name}: {ex.Message}");
}
subject?.Dispose();
control?.Dispose();
TaskScheduler.UnobservedTaskException -= OnUnobserved;
Console.WriteLine($"Cleanup verified: {cleanupSucceeded}");
}
return cleanupSucceeded ? exitCode : 2;
}
private static ConfigurationOptions Options(int port, int syncTimeout, int asyncTimeout) => new()
{
EndPoints = { { "127.0.0.1", port } },
AbortOnConnectFail = true,
ConnectTimeout = 5000,
SyncTimeout = syncTimeout,
AsyncTimeout = asyncTimeout,
HeartbeatInterval = TimeSpan.FromMilliseconds(100),
AllowAdmin = true,
};
[MethodImpl(MethodImplOptions.NoInlining)]
private static bool ScanSync(IDatabase db, RedisKey key)
{
try
{
using var iterator = db.HashScan(key, pattern: "field:*", pageSize: 10).GetEnumerator();
Console.WriteLine($"MoveNext returned: {iterator.MoveNext()}");
return false;
}
catch (RedisTimeoutException ex)
{
Console.WriteLine($"Caller exception: {ex}");
return true;
}
}
private static async Task<bool> ScanAsync(IDatabase db, RedisKey key)
{
try
{
await using var iterator = db.HashScanAsync(key, pattern: "field:*", pageSize: 10).GetAsyncEnumerator();
Console.WriteLine($"MoveNextAsync returned: {await iterator.MoveNextAsync()}");
return false;
}
catch (RedisTimeoutException ex)
{
Console.WriteLine($"Caller exception: {ex}");
return true;
}
}
private static void OnUnobserved(object? sender, UnobservedTaskExceptionEventArgs e)
{
bool scan = e.Exception.Flatten().InnerExceptions.Any(ex =>
ex.StackTrace?.Contains("CursorEnumerable", StringComparison.Ordinal) == true
&& ex.StackTrace.Contains("AwaitedNextAsync", StringComparison.Ordinal));
if (scan) Interlocked.Increment(ref scanEvents);
else Interlocked.Increment(ref otherEvents);
Console.WriteLine($"UNOBSERVED (scan={scan}): {e.Exception}");
// Record first. SetObserved does not prevent the defect or count as its repair.
e.SetObserved();
}
}repro/ScanTimeoutRepro.csproj
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<OutputType>Exe</OutputType>
<TargetFramework>net8.0</TargetFramework>
<ImplicitUsings>enable</ImplicitUsings>
<Nullable>enable</Nullable>
<RedisVersion Condition="'$(RedisVersion)' == ''">2.13.17</RedisVersion>
<IsPackable>false</IsPackable>
<GeneratePackageOnBuild>false</GeneratePackageOnBuild>
</PropertyGroup>
<ItemGroup>
<PackageReference Include="StackExchange.Redis" Version="$(RedisVersion)" />
</ItemGroup>
</Project>repro/NuGet.Config
<?xml version="1.0" encoding="utf-8"?>
<configuration>
<packageSources>
<clear />
<add key="nuget.org" value="https://api.nuget.org/v3/index.json" />
</packageSources>
</configuration>repro/README.md
Scan timeout reproduction candidate
This standalone .NET 8 console project references only StackExchange.Redis. Release build with version 2.13.17 succeeded with zero warnings and errors on 2026-10-05. Redis runtime execution of this reduced program is pending. The confirmed issue evidence comes from a separate cluster harness described above.
Run on a disposable instance
Requirements: .NET 8 SDK or later with the .NET 8 runtime, and Docker or Podman. The example uses Redis 8.4. Use an unused loopback port and a new container name. CLIENT PAUSE ALL suspends command processing for all clients of this Redis instance for six seconds, so use the dedicated container below. It does not require stopping any existing cluster.
docker run --detach --rm --name se-scan-issue-repro -p 127.0.0.1:16389:6379 redis:8.4 redis-server --save "" --appendonly no
docker exec se-scan-issue-repro redis-cli PINGWait for PONG. From this repro directory, build and run each mode in a separate process:
dotnet build -c Release
dotnet run -c Release --no-build -- 16389 sync --disposable-instance
dotnet run -c Release --no-build -- 16389 async --disposable-instance
dotnet run -c Release --no-build -- 16389 healthy --disposable-instanceEach process creates one uniquely named hash with a two-minute TTL, observes events before client disposal, deletes its own hash, and verifies absence. Allow approximately 10–20 seconds per run. The example container has no persistent volume. If execution is interrupted, stop the container to remove all its data:
docker stop se-scan-issue-reproFor Podman, replace docker with podman. These container commands are proposed reproduction instructions and were not executed during preparation of this package.
Read the result
- The target symptom is
callerTimeout=Truefollowed by at least oneUNOBSERVED (scan=True)event whose stack includesCursorEnumerableandAwaitedNextAsync. - Exit 0 means the target symptom was observed and key cleanup succeeded. It means the bug reproduced, not that the library passed a test.
- Exit 1 means the target symptom was not observed in that run. This is not proof of a fix. The healthy and async controls are expected to exit 1; inspect their caller result, all event counts, and cleanup independently.
- Exit 2 means invalid arguments or an inconclusive setup, observation, or cleanup failure. A caught
AggregateExceptioninstead ofRedisTimeoutExceptiondoes not establish the targeted ordering. otherEventsreports unobserved events outside the selected stack filter. Do not count them as this defect or ignore them when interpreting a control.
The program calls SetObserved only after recording each event. This keeps the diagnostic handler from escalating the event; it does not fix the library. Forced GC encourages finalization in this reduced example. In the original cluster run, recorded events preceded the harness's forced GC.
Why this candidate may expose the race
The synchronous scan wait is shorter than the asynchronous command timeout. Pausing Redis beyond both timeouts should allow the public synchronous call to throw first and the internal await to fault later. This hypothesis still requires runtime confirmation. See CLIENT PAUSE documentation for server behavior.
Activity
mgravell commentedon Oct 9, 2026
Looking now
mgravell commentedon Oct 9, 2026
Confirmed and mitigated as above. Note that by default I would not plan to backport here (I note you report on v2, but v3 is current).
Scan: observe the faults of abandoned page tasks after a sync timeout…
Merge main (3.4.0) into v4
Update dependency StackExchange.Redis to 3.4.0 (#9)
Update dependency StackExchange.Redis to 3.4.0 (#9)
mgravell commentedon Oct 23, 2026
Shipped in 3.4.20 (https://github.com/StackExchange/StackExchange.Redis/releases/tag/3.4.20).