Sentinel +switch-master announcement missed if master dies while connected

I'm testing Redis, StackExchange.Redis, and Redis-Sentinel in order to setup a appropriate HA environment.

I've noticed that when i stand up a sentinel cluster as described in these docs: https://hub.docker.com/r/bitnami/redis-sentinel/ and i shutdown the master after a multiplexer connection has been successfully established by a client using StackExchange.Redis. The multiplexer fails to realize that the failover has occurred and continues to queue messages for the now non-existent leader.

Funny thing is If i manually trigger a failover using the sentinel-cli with SENTINEL FAILOVER mymaster the multiplexer realizes the change and appropriately updates the leader.

I looked into the code and noticed that the +switch-master subscription hook is never fired when the primary Redis is shutdown even if the sentinel associated is still up and running. In the situation where i manually trigger the failover, the hook properly fires.

StackExchange.Redis/src/StackExchange.Redis/ConnectionMultiplexer.Sentinel.cs

Lines 33 to 62 in f6171a1

    
           if (sub.SubscribedEndpoint(RedisChannel.Literal("+switch-master")) == null) 
        
           { 
        
               sub.Subscribe(RedisChannel.Literal("+switch-master"), (__, message) => 
        
               { 
        
                   string[] messageParts = ((string)message!).Split(new[] { ' ' }, StringSplitOptions.RemoveEmptyEntries); 
        
                   // We don't care about the result of this - we're just trying 
        
                   _ = Format.TryParseEndPoint(string.Format("{0}:{1}", messageParts[1], messageParts[2]), out var switchBlame); 
        
                   lock (sentinelConnectionChildren) 
        
                   { 
        
                       // Switch the primary if we have connections for that service 
        
                       if (sentinelConnectionChildren.ContainsKey(messageParts[0])) 
        
                       { 
        
                           ConnectionMultiplexer child = sentinelConnectionChildren[messageParts[0]]; 
        
                           // Is the connection still valid? 
        
                           if (child.IsDisposed) 
        
                           { 
        
                               child.ConnectionFailed -= OnManagedConnectionFailed; 
        
                               child.ConnectionRestored -= OnManagedConnectionRestored; 
        
                               sentinelConnectionChildren.Remove(messageParts[0]); 
        
                           } 
        
                           else 
        
                           { 
        
                               SwitchPrimary(switchBlame, sentinelConnectionChildren[messageParts[0]]); 
        
                           } 
        
                       } 
        
                   } 
        
               }, CommandFlags.FireAndForget); 
        
           }

I've confirmed that the sentinel is still issuing the +switch-master command in both scenarios.

I've tried multiple ways to setup the multiplexer, here is my last iteration.

var opts = new ConfigurationOptions
            {
                SyncTimeout = 60000,
                AbortOnConnectFail = false,
                Ssl = false,
                Password = config.Password,
                ReconnectRetryPolicy = new LinearRetry(2000),
                ConnectRetry = 30,
                ServiceName = "mymaster",
                ConfigCheckSeconds = 10
            };
            opts.EndPoints.Add("redis-sentinel-1", 26379);
            opts.EndPoints.Add("redis-sentinel-2", 26379);
            opts.EndPoints.Add("redis-sentinel-3", 26379);

            var log = new LoggerWriter(_log, LogLevel.Information);

            //_multiplexer = await ConnectionMultiplexer.ConnectAsync(opts, log);
            _multiplexerSentinel = await ConnectionMultiplexer.SentinelConnectAsync(opts, log);
            _multiplexer = _multiplexerSentinel.GetSentinelMasterConnection(opts, log);


            _multiplexerSentinel.ConfigurationChanged += (s, e) =>
            {
                _log.LogError($"Sentinel configuration changed (Sentinel): {e.EndPoint}");
                _log.LogInformation($"New server status: {_multiplexerSentinel.GetStatus()}");
            };
            _multiplexerSentinel.ConnectionFailed += (s, e) =>
            {
                _log.LogError($"Connection failed (Sentinel): {e.ConnectionType}, {e.FailureType}, {e.EndPoint}, {e.Exception}");
                _log.LogInformation($"New server status: {_multiplexerSentinel.GetStatus()}");
            };
            _multiplexerSentinel.ConnectionRestored += (s, e) =>
            {
                _log.LogError($"Connection restored (Sentinel): {e.ConnectionType}, {e.FailureType}, {e.EndPoint}, {e.Exception}");
                _log.LogInformation($"New server status: {_multiplexerSentinel.GetStatus()}");
            };
            _multiplexerSentinel.ConfigurationChangedBroadcast += (s, e) =>
            {
                _log.LogError($"Connection changed via broadcast (Sentinel): {e.EndPoint}");
                _log.LogInformation($"New server status: {_multiplexerSentinel.GetStatus()}");
            };

            _multiplexer.ConfigurationChanged += (s, e) =>
            {
                _log.LogError($"Configuration changed: {e.EndPoint}");
                _log.LogInformation($"New server status: {_multiplexer.GetStatus()}");
                //var info = _multiplexer.GetServers().Where(x => x.IsConnected).Select(x => x.InfoRaw()).ToArray();
               // _multiplexer.Configure(new LoggerWriter(_log, LogLevel.Information));
                //_log.LogInformation($"Server info: {info}");
                // _multiplexer.PublishReconfigure();
            };
            _multiplexer.ConnectionFailed += (s, e) =>
            {
                _log.LogError($"Connection failed: {e.ConnectionType}, {e.FailureType}, {e.EndPoint}, {e.Exception}");
                _log.LogInformation($"New server status: {_multiplexer.GetStatus()}");
            };
            _multiplexer.ConnectionRestored += (s, e) =>
            {
                _log.LogError($"Connection restored: {e.ConnectionType}, {e.FailureType}, {e.EndPoint}, {e.Exception}");
                _log.LogInformation($"New server status: {_multiplexer.GetStatus()}");
            };
            _multiplexer.ConfigurationChangedBroadcast += (s, e) =>
            {
                _log.LogError($"Connection changed via broadcast: {e.EndPoint}");
                _log.LogInformation($"New server status: {_multiplexer.GetStatus()}");
            };

This workaround works but its more than a bit hacky...

My failover period is 20 seconds so 30 should be enough time to elect a new leader.

            var reconfiguring = false;
            _multiplexer.ConnectionFailed += (s, e) =>
            {
                _log.LogError($"Connection failed: {e.ConnectionType}, {e.FailureType}, {e.EndPoint}, {e.Exception}");

                _ = Task.Run(async () =>
                {
                    try
                    {
                        if (reconfiguring) return;
                        reconfiguring = true;
                        await Task.Delay(TimeSpan.FromSeconds(30));
                        await _multiplexer.ReconfigureAsync("connection_lost");
                        await _multiplexer.PublishReconfigureAsync();
                        await Task.Delay(TimeSpan.FromSeconds(5));
                        _log.LogInformation($"Reconfigured status: {_multiplexer.GetStatus()}");
                    }finally { reconfiguring = false; }
                });
            };

The intended usage for Sentinel today is to connect to the sentinel endpoints with a service name, and the ConnectionMultiplexer you get back will be connected to the current primary. When the primary switches, it'll follow (there are 2 multiplexers under the covers, with 1 listening to Sentinel).

It looks like you may have tried this given the commented out line in code there looking to do this. What was or wasn't happening with that setup and nothing else?

Sentinel: re-resolve the primary when it is demoted on a healthy conn…

Merge main into v4 (19 commits to 2026-10-23), with the v3-only fixes…

Navigation Menu

Repository navigation

Sentinel +switch-master announcement missed if master dies while connected #2487

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

	if (sub.SubscribedEndpoint(RedisChannel.Literal("+switch-master")) == null)
	{
	sub.Subscribe(RedisChannel.Literal("+switch-master"), (__, message) =>
	{
	string[] messageParts = ((string)message!).Split(new[] { ' ' }, StringSplitOptions.RemoveEmptyEntries);
	// We don't care about the result of this - we're just trying
	_ = Format.TryParseEndPoint(string.Format("{0}:{1}", messageParts[1], messageParts[2]), out var switchBlame);

	lock (sentinelConnectionChildren)
	{
	// Switch the primary if we have connections for that service
	if (sentinelConnectionChildren.ContainsKey(messageParts[0]))
	{
	ConnectionMultiplexer child = sentinelConnectionChildren[messageParts[0]];

	// Is the connection still valid?
	if (child.IsDisposed)
	{
	child.ConnectionFailed -= OnManagedConnectionFailed;
	child.ConnectionRestored -= OnManagedConnectionRestored;
	sentinelConnectionChildren.Remove(messageParts[0]);
	}
	else
	{
	SwitchPrimary(switchBlame, sentinelConnectionChildren[messageParts[0]]);
	}
	}
	}
	}, CommandFlags.FireAndForget);
	}

Sentinel +switch-master announcement missed if master dies while connected #2487

Description

Activity

Kilowhisky commented on Jun 22, 2023

NickCraver commented on Jul 2, 2023

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions