Skip to content

Synchronous scan timeout can leave an internal task exception unobserved in 2.13.17 #3260

Closed

Description

@Chao-Shiun

Summary

We observed TaskScheduler.UnobservedTaskException after a synchronous scan had already thrown a timeout that the caller caught. A representative case uses IDatabase.HashScan; the later exception includes CursorEnumerable<T>.Enumerator.AwaitedNextAsync in its stack.

The timeout during a Redis outage is expected. The additional unobserved exception from an internal task is the behavior we are reporting. We have not observed a process crash or measured a memory leak.

Environment

  • StackExchange.Redis 2.13.17, informational version 2.13.17+6e0df970c7.
  • Windows, .NET 8.0.31.
  • Six-node Redis Cluster using Redis 8.4, three primaries and three replicas in local containers.
  • SyncTimeout and AsyncTimeout were both 1000 ms in the original reproduction.

Observed reproduction

  1. Connect to the healthy cluster and create a hash on a known shard.
  2. Stop that shard's primary and replica in the disposable test cluster.
  3. Enumerate IDatabase.HashScan synchronously, catching the caller-visible exception and disposing the enumerator.
  4. Restore the stopped nodes and keep listening for TaskScheduler.UnobservedTaskException.

One representative run produced a caught RedisTimeoutException after approximately 1022 ms. Its library frames were:

CursorEnumerable<T>.Enumerator.ThrowTimeout
CursorEnumerable<T>.Enumerator.Wait
CursorEnumerable<T>.Enumerator.SlowNextSync
CursorEnumerable<T>.Enumerator.MoveNext

Roughly 2.5 seconds after the caught timeout, the event contained an AggregateException wrapping a RedisConnectionException with SocketClosed. This is the finalizer event timing, not a measurement of when the internal task faulted. The inner RedisConnectionException stack included:

StackExchange.Redis.CursorEnumerable`1+Enumerator+<AwaitedNextAsync>d__30.MoveNext

These are selected frames from recorded exceptions, not a complete raw stack trace.

Across seven isolated processes exercising synchronous scan overloads directly through StackExchange.Redis, six recorded this event. The remaining process received an AggregateException at the synchronous call and recorded no later event. Four asynchronous scan processes under the same outage condition and one healthy synchronous HashScan process recorded no such events. These controls do not establish that every asynchronous scan path is unaffected.

The direct-client path did not instantiate our application wrapper, although the original test harness references application assemblies for shared helpers. Every recorded event occurred during restoration, before client disposal and forced GC. We have not isolated whether restoration itself affects event timing.

Suspected mechanism

In the version-matched CursorEnumerable source, SlowNextSync converts the result of SlowNextAsync to a task. Wait throws when TryWait returns false. TryWait uses a timed Task.Wait.

This appears to leave the outer task produced by AwaitedNextAsync without a future observer if it faults after the synchronous wait times out. The enumerator's _pending scan-page task is a different task; observing only that field would not necessarily address the outer task. This mechanism is inferred from the recorded stacks and matching source, rather than a deterministic scheduler test.

Expected behavior: preserve the caller-visible timeout while ensuring that a later fault of the internal outer task is observed.

Smaller reproduction candidate

The repro/Program.cs and repro/ScanTimeoutRepro.csproj included below use only StackExchange.Redis. They build successfully for .NET 8 with 2.13.17, but this smaller program has not yet been run against Redis.

It proposes CLIENT PAUSE 6000 ALL on a disposable standalone instance, with SyncTimeout=250 and AsyncTimeout=1500, to separate the synchronous timeout from the later asynchronous fault. This is a different fault mechanism and configuration from the confirmed cluster reproduction above; its success must not be assumed. repro/README.md includes commands, controls, cleanup, and result interpretation.

Could you investigate this behavior in StackExchange.Redis 2.13.17 and correct the synchronous timeout path so that a later fault of this internal outer task is observed? We can validate a proposed fix against the original cluster scenario.

Standalone candidate source and instructions

repro/Program.cs
using System.Diagnostics;
using System.Reflection;
using System.Runtime.CompilerServices;
using System.Runtime.InteropServices;
using StackExchange.Redis;

// Candidate reduction: use only a disposable Redis instance on loopback.
// CLIENT PAUSE ALL temporarily affects every client of that instance.
internal static class Program
{
    private const int PauseMilliseconds = 6000;
    private static int scanEvents;
    private static int otherEvents;

    private static async Task<int> Main(string[] args)
    {
        if (args.Length != 3 || !int.TryParse(args[0], out int port) || port is < 1 or > 65535
            || args[1] is not ("sync" or "async" or "healthy") || args[2] != "--disposable-instance")
        {
            Console.Error.WriteLine("Usage: <port> <sync|async|healthy> --disposable-instance");
            return 2;
        }

        Console.WriteLine($"Runtime: {RuntimeInformation.FrameworkDescription}");
        Console.WriteLine($"OS: {RuntimeInformation.OSDescription}");
        Console.WriteLine($"Package: {typeof(ConnectionMultiplexer).Assembly.GetCustomAttribute<AssemblyInformationalVersionAttribute>()?.InformationalVersion}");
        Console.WriteLine($"Mode: {args[1]}; SyncTimeout=250; AsyncTimeout=1500; pause={PauseMilliseconds}");
        TaskScheduler.UnobservedTaskException += OnUnobserved;
        RedisKey key = $"scan-timeout-repro:{Guid.NewGuid():N}";
        ConnectionMultiplexer? control = null;
        ConnectionMultiplexer? subject = null;
        Stopwatch? pauseClock = null;
        bool callerTimedOut = false;
        bool cleanupSucceeded = false;
        int exitCode = 2;
        try
        {
            control = await ConnectionMultiplexer.ConnectAsync(Options(port, 5000, 5000));
            subject = await ConnectionMultiplexer.ConnectAsync(Options(port, 250, 1500));
            var admin = control.GetDatabase();
            await admin.HashSetAsync(key, Enumerable.Range(0, 1024)
                .Select(i => new HashEntry($"field:{i}", i)).ToArray());
            await admin.KeyExpireAsync(key, TimeSpan.FromMinutes(2));
            await subject.GetDatabase().PingAsync();

            if (args[1] != "healthy")
            {
                pauseClock = Stopwatch.StartNew();
                await admin.ExecuteAsync("CLIENT", "PAUSE", PauseMilliseconds, "ALL");
            }

            // Both entry points catch their caller-visible exception and dispose the enumerator.
            callerTimedOut = args[1] == "async"
                ? await ScanAsync(subject.GetDatabase(), key)
                : ScanSync(subject.GetDatabase(), key);

            // Let the asynchronous command timeout occur before making unreachable tasks collectible.
            // Forced GC only exposes the event; it is not a production recommendation.
            await Task.Delay(PauseMilliseconds + 2000);
            for (int i = 0; i < 5; i++)
            {
                GC.Collect();
                GC.WaitForPendingFinalizers();
                GC.Collect();
                await Task.Delay(200);
            }

            int count = Volatile.Read(ref scanEvents);
            Console.WriteLine($"Observation: callerTimeout={callerTimedOut}; scanEvents={count}; otherEvents={Volatile.Read(ref otherEvents)}");
            // 0 means the target symptom was observed, not that the library passed a test.
            exitCode = callerTimedOut && count > 0 ? 0 : 1;
        }
        catch (Exception ex)
        {
            Console.Error.WriteLine($"Setup/observation error: {ex}");
        }
        finally
        {
            // Do not race cleanup against an outstanding server-wide pause.
            if (pauseClock is not null)
            {
                int remaining = PauseMilliseconds + 500 - (int)pauseClock.ElapsedMilliseconds;
                if (remaining > 0) await Task.Delay(remaining);
            }
            try
            {
                if (control is not null)
                {
                    await control.GetDatabase().KeyDeleteAsync(key);
                    cleanupSucceeded = !await control.GetDatabase().KeyExistsAsync(key);
                }
            }
            catch (Exception ex)
            {
                Console.Error.WriteLine($"Cleanup error: {ex.GetType().Name}: {ex.Message}");
            }
            subject?.Dispose();
            control?.Dispose();
            TaskScheduler.UnobservedTaskException -= OnUnobserved;
            Console.WriteLine($"Cleanup verified: {cleanupSucceeded}");
        }
        return cleanupSucceeded ? exitCode : 2;
    }

    private static ConfigurationOptions Options(int port, int syncTimeout, int asyncTimeout) => new()
    {
        EndPoints = { { "127.0.0.1", port } },
        AbortOnConnectFail = true,
        ConnectTimeout = 5000,
        SyncTimeout = syncTimeout,
        AsyncTimeout = asyncTimeout,
        HeartbeatInterval = TimeSpan.FromMilliseconds(100),
        AllowAdmin = true,
    };

    [MethodImpl(MethodImplOptions.NoInlining)]
    private static bool ScanSync(IDatabase db, RedisKey key)
    {
        try
        {
            using var iterator = db.HashScan(key, pattern: "field:*", pageSize: 10).GetEnumerator();
            Console.WriteLine($"MoveNext returned: {iterator.MoveNext()}");
            return false;
        }
        catch (RedisTimeoutException ex)
        {
            Console.WriteLine($"Caller exception: {ex}");
            return true;
        }
    }

    private static async Task<bool> ScanAsync(IDatabase db, RedisKey key)
    {
        try
        {
            await using var iterator = db.HashScanAsync(key, pattern: "field:*", pageSize: 10).GetAsyncEnumerator();
            Console.WriteLine($"MoveNextAsync returned: {await iterator.MoveNextAsync()}");
            return false;
        }
        catch (RedisTimeoutException ex)
        {
            Console.WriteLine($"Caller exception: {ex}");
            return true;
        }
    }

    private static void OnUnobserved(object? sender, UnobservedTaskExceptionEventArgs e)
    {
        bool scan = e.Exception.Flatten().InnerExceptions.Any(ex =>
            ex.StackTrace?.Contains("CursorEnumerable", StringComparison.Ordinal) == true
            && ex.StackTrace.Contains("AwaitedNextAsync", StringComparison.Ordinal));
        if (scan) Interlocked.Increment(ref scanEvents);
        else Interlocked.Increment(ref otherEvents);
        Console.WriteLine($"UNOBSERVED (scan={scan}): {e.Exception}");
        // Record first. SetObserved does not prevent the defect or count as its repair.
        e.SetObserved();
    }
}
repro/ScanTimeoutRepro.csproj
<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>net8.0</TargetFramework>
    <ImplicitUsings>enable</ImplicitUsings>
    <Nullable>enable</Nullable>
    <RedisVersion Condition="'$(RedisVersion)' == ''">2.13.17</RedisVersion>
    <IsPackable>false</IsPackable>
    <GeneratePackageOnBuild>false</GeneratePackageOnBuild>
  </PropertyGroup>
  <ItemGroup>
    <PackageReference Include="StackExchange.Redis" Version="$(RedisVersion)" />
  </ItemGroup>
</Project>
repro/NuGet.Config
<?xml version="1.0" encoding="utf-8"?>
<configuration>
  <packageSources>
    <clear />
    <add key="nuget.org" value="https://api.nuget.org/v3/index.json" />
  </packageSources>
</configuration>
repro/README.md

Scan timeout reproduction candidate

This standalone .NET 8 console project references only StackExchange.Redis. Release build with version 2.13.17 succeeded with zero warnings and errors on 2026-10-05. Redis runtime execution of this reduced program is pending. The confirmed issue evidence comes from a separate cluster harness described above.

Run on a disposable instance

Requirements: .NET 8 SDK or later with the .NET 8 runtime, and Docker or Podman. The example uses Redis 8.4. Use an unused loopback port and a new container name. CLIENT PAUSE ALL suspends command processing for all clients of this Redis instance for six seconds, so use the dedicated container below. It does not require stopping any existing cluster.

docker run --detach --rm --name se-scan-issue-repro -p 127.0.0.1:16389:6379 redis:8.4 redis-server --save "" --appendonly no
docker exec se-scan-issue-repro redis-cli PING

Wait for PONG. From this repro directory, build and run each mode in a separate process:

dotnet build -c Release
dotnet run -c Release --no-build -- 16389 sync --disposable-instance
dotnet run -c Release --no-build -- 16389 async --disposable-instance
dotnet run -c Release --no-build -- 16389 healthy --disposable-instance

Each process creates one uniquely named hash with a two-minute TTL, observes events before client disposal, deletes its own hash, and verifies absence. Allow approximately 10–20 seconds per run. The example container has no persistent volume. If execution is interrupted, stop the container to remove all its data:

docker stop se-scan-issue-repro

For Podman, replace docker with podman. These container commands are proposed reproduction instructions and were not executed during preparation of this package.

Read the result

  • The target symptom is callerTimeout=True followed by at least one UNOBSERVED (scan=True) event whose stack includes CursorEnumerable and AwaitedNextAsync.
  • Exit 0 means the target symptom was observed and key cleanup succeeded. It means the bug reproduced, not that the library passed a test.
  • Exit 1 means the target symptom was not observed in that run. This is not proof of a fix. The healthy and async controls are expected to exit 1; inspect their caller result, all event counts, and cleanup independently.
  • Exit 2 means invalid arguments or an inconclusive setup, observation, or cleanup failure. A caught AggregateException instead of RedisTimeoutException does not establish the targeted ordering.
  • otherEvents reports unobserved events outside the selected stack filter. Do not count them as this defect or ignore them when interpreting a control.

The program calls SetObserved only after recording each event. This keeps the diagnostic handler from escalating the event; it does not fix the library. Forced GC encourages finalization in this reduced example. In the original cluster run, recorded events preceded the harness's forced GC.

Why this candidate may expose the race

The synchronous scan wait is shorter than the asynchronous command timeout. Pausing Redis beyond both timeouts should allow the public synchronous call to throw first and the internal await to fault later. This hypothesis still requires runtime confirmation. See CLIENT PAUSE documentation for server behavior.

Activity

  1. mgravell commented on Oct 9, 2026

    @mgravell
    Collaborator

    Looking now

  2. mgravell commented on Oct 9, 2026

    @mgravell
    Collaborator

    Confirmed and mitigated as above. Note that by default I would not plan to backport here (I note you report on v2, but v3 is current).

  3. added a commit that references this issue on Oct 9, 2026
  4. mgravell commented on Oct 23, 2026

    @mgravell
    Collaborator
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions