fix(instance-manager): promote a replica without waiting for WAL receivers to stop - #11367

tryTakeOver read the lease with the caller's un-deadlined context, so a
Get against an unreachable API server could block indefinitely instead
of retrying at RetryPeriod cadence. That silence starves the lease
watchdog's heartbeat, which can fence a primary that is still correctly
retrying and has not actually stepped down.

Wrap the call in a RetryPeriod-bounded context, mirroring the same
pattern already used for the post-renewal lease check.

Closes cloudnative-pg#11349

Signed-off-by: Leonardo Cecchi <[email protected]>

…ease

Primary self-fencing on network isolation used to live in the liveness
probe: on API-server unreachability it pinged every peer's failsafe
endpoint, and a failed probe caused Kubelet to restart the pod roughly
three probe periods (~30s) after a partition started.

The primary lease now gives a faster, more precise signal for the same
condition. When the primary fails to renew its lease, it runs the same
peer-reachability check itself and, if it should step down, requests a
fast PostgreSQL shutdown directly rather than waiting on Kubelet.

The check runs as soon as renewal fails (not after waiting out the
remaining lease TTL): the RenewDeadline-to-LeaseDuration margin is the
time budget for the check-and-shutdown sequence to complete before a
replica becomes eligible to promote, not idle slack to wait through.

The step-down condition is no longer just peer unreachability: the
/failsafe entrypoint now reports the responding instance's view of the
cluster's target primary (via a new internal/management/failsafe wire
package shared by both sides), so a primary that can still reach every
peer but learns from one of them that someone else is now the target
primary steps down too. This closes a gap the reachability check alone
couldn't see: a peer being reachable doesn't mean it hasn't already
been promoted.

The liveness probe's isolation logic is removed now that the lease
handles it; IsHealthy always reports OK.

Signed-off-by: Leonardo Cecchi <[email protected]>

The liveness probe used to always report healthy, so a Pod could never
be restarted when it truly got stuck. It now fails, so the kubelet can
kill and recreate the Pod, in two cases the primary lease and fencing
logic can't recover from on their own:

- the primary-lease loop stops attempting to renew or take over the
  lease altogether (e.g. a deadlock), as opposed to merely failing to
  renew, which is an expected condition handled separately by the
  lease/failsafe step-down check;

- PostgreSQL was asked to shut down immediately (e.g. because this
  primary had to step down) but didn't honor it within a new
  configurable `.spec.immediateShutdownTimeout` (default 30s).

Signed-off-by: Leonardo Cecchi <[email protected]>

…ivers to stop

The primary lease now guarantees only one instance can act as primary
at a time, so failovers and switchovers no longer need to stall until
every WAL receiver in the cluster reports itself down before a new
primary is chosen and promoted. This removes a source of delay and
stuck-waiting states during failover.

Signed-off-by: Leonardo Cecchi <[email protected]>

The controller used to snapshot replica LSNs and immediately elect a
new primary in the same reconcile pass as marking the failover
pending, even though the old primary could still be alive and
streaming for a few more seconds. That window let the LSN snapshot go
stale before the elected replica actually promoted, leaving a sibling
replica to fork onto a diverged timeline with no automatic repair.

Splits "mark pending" and "elect" into two separate reconcile passes,
deferring election until the primary lease is confirmed released or
expired, so the LSN snapshot used for the election is taken as close
as possible to the actual promotion.

Signed-off-by: Leonardo Cecchi <[email protected]>

…former primary

When an unreachable former primary has no running postmaster, requesting
a shutdown has nothing to act on it and can block forever on the
unbuffered command channel. Skip the request in that case, and export
IsStatusRunning so the controller can check it.

Signed-off-by: Leonardo Cecchi <[email protected]>