You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
A planning-lane unit called gsd_plan_milestone for a milestone and got:
Milestone <ID> is currently leased by auto-<host>-683665-2472600d. Retry after 2026-09-17T21:14:32.951Z.
The named worker had no reachable owning session — the holder is an orphaned/unreachable worker, not a live conflict. This exact failure signature had already recurred twice earlier the same day on the same milestone. Per that prior experience, the unit waited out the reported TTL (via a wall-clock until-loop, not a bare sleep) and retried. The second attempt returned:
Milestone <ID> is currently leased by auto-<host>-683665-2472600d. Retry after 2026-09-17T21:15:32.952Z.
— exactly 60 seconds later than the first, i.e. the reported expiry is relative to each new check, not a fixed point in time that will eventually pass. This was the third occurrence of the identical failure signature on this milestone in one day: the previous session had already exhausted 5 retries across ~13 minutes of real waiting (including one 5-minute wait) with the same non-clearing result.
Ran gsd_doctor scoped to this exact milestone/project as part of investigating a fix:
Zero issues reported — gsd_doctor has no check that would ever catch this class of problem.
Source-level root cause investigation
Traced the lease logic in the installed package (@opengsd/[email protected]):
src/resources/extensions/gsd/tools/workflow-tool-executors.ts, executePlanMilestone (~line 2213 onward): on a held, non-expired lease, it checks holder?.status === "active" (a DB column on the workers table) to decide whether this is a real conflict. If the current call is not itself an active in-process auto session for the same worker id, it returns milestoneLeaseConflictResult immediately — it never corroborates holder.status against the holder's actual OS process liveness.
src/resources/extensions/gsd/db/auto-workers.ts already implements exactly that liveness check — isWorkerProcessAlive() (a real process.kill(pid, 0) probe) and isDeadLocalAutoWorker() — but neither is called anywhere in the executePlanMilestone lease-conflict path.
src/resources/extensions/gsd/doctor-engine-checks.ts, reportOrphanedRunningAttempts() (fixing issue No sanctioned way to reconcile a task's canonical lifecycle after a manual repair, without side effects or irreversible typos #1749) already solves this exact class of problem for task-level Attempts: it looks up the lease-holder's worker row and calls the same process.kill(pid, 0)-style check to decide whether the holder is really dead, then reports orphaned_running_attempt with a documented fix path (gsd_task_settle). There is no equivalent check for orphaned milestone-lease holders — gsd_doctor's empty output against this live incident confirms the gap directly.
src/resources/extensions/gsd/db/milestone-leases.ts: LEASE_TTL_SECONDS = 60. executePlanMilestone sets up a leaseRefreshTimer (setInterval, firing every leaseRefreshMs = (milestoneLeaseTtlSeconds()/2)*1000 = 30000ms) that calls refreshMilestoneLease, extending expires_at to now + 60s on every tick, for as long as the call's async function is alive. The finally block (which calls releaseMilestoneLease/clearInterval) only runs if that async function actually returns or throws.
The observed exact 60-second gap between successive Retry after timestamps, repeated across three independent checks spanning over an hour, is the key diagnostic fact: it is not consistent with a stale DB row for a dead process (which a process.kill(pid, 0) check, like the one already used for #1749, could detect and would show as immediately expired/dead). It is consistent with the original Node process behind the leaseholder worker still being alive and its leaseRefreshTimer still firing every 30s, i.e. a leaked, still-executing process (most likely a hung await handlePlanMilestone(...) inside executePlanMilestone, or an MCP server instance that never received a shutdown signal when its owning Claude Code session died) that keeps renewing its own lease forever. Nothing ties that process's lease-refresh loop to any upper bound on total lease lifetime, and nothing ties it to whether its parent interactive session is still reachable.
Why is this GSD and not a user issue
No user command caused this. The session driving the original gsd_plan_milestone call became unreachable through ordinary auto-mode/session churn — nothing about that is a misuse of GSD's tools. GSD's own leasing subsystem is supposed to guarantee that a milestone is never blocked longer than LEASE_TTL_SECONDS past the last legitimate heartbeat; here a leaked process defeats that guarantee by continuing to legitimately-looking self-renew past the point its holder is reachable or useful. There is no independent circuit breaker on total lease lifetime, no cross-check between "worker row says active" and "OS process is actually alive" in the tool-facing conflict-check path (even though that exact check already exists and is used one call-site away, in doctor-engine-checks.ts), and the one tool explicitly named for catching this kind of problem (gsd_doctor) reports a clean bill of health against the live incident.
What did you expect
One of:
gsd_plan_milestone/gsd_plan_slice/gsd_plan_task's lease-conflict branch corroborates holder.status === "active" against real OS PID liveness (isWorkerProcessAlive, already implemented in db/auto-workers.ts and already consumed by doctor-engine-checks.ts's reportOrphanedRunningAttempts for the equivalent task-Attempt case) before treating a held lease as a real, unresolvable conflict — and reclaims via the already-existing forceReleaseLeasesForWorker() + markWorkerCrashed() primitives when the holder's PID is confirmed dead.
gsd_doctor gains an orphaned_milestone_lease check parallel to orphaned_running_attempt: a held, non-expired lease whose holder is unreachable is reported (ideally --fix-able), instead of the tool reporting a clean bill of health while a milestone sits wedged.
Because a still-alive-but-orphaned holder (not just a dead PID) can also cause this — as the evidence here suggests — the lease-refresh loop should carry an absolute maximum total lifetime independent of TTL renewal (e.g. a worker should not be able to self-renew a single lease past some bounded ceiling, or renewal should require some proof the owning interactive session is still reachable), so an abandoned in-flight call cannot hold a milestone hostage indefinitely regardless of how "alive" its process technically still is.
Steps to reproduce, concrete incident
Not reliably reproducible on demand — the trigger is whatever leaves the process behind a gsd_plan_milestone call still running while its owning session becomes unreachable. Best known repro shape from this incident:
Start /gsd auto (or otherwise dispatch a planning-lane unit) and let it reach a gsd_plan_milestone call for a fresh milestone — this registers a new workers row and claims a lease with a live leaseRefreshTimer.
Have the driving process become unreachable while that call (or its refresh timer) is still in flight, without the tool's finally block ever running (e.g. an abrupt process-tree kill or orchestrator handoff rather than a graceful /gsd stop).
From a new session/unit, call gsd_plan_milestone (or any lease-gated planning tool) for the same milestone.
Observe: Milestone X is currently leased by <old-worker-id>. Retry after <ts>. — and on each subsequent retry (after waiting out the previously reported <ts>), a new<ts> exactly ~60s further in the future, indefinitely.
gsd_doctor run scoped to the milestone against this exact live incident → {"ok": true, "issues": [], "counts": {"error": 0, "warning": 0, "info": 0}}.
gsd_milestone_status throughout → status: "queued", sliceCount: 0 — the milestone had a fully-prepared CONTEXT/RESEARCH pair but zero persisted slice plan, solely because every gsd_plan_milestone call was rejected at the lease gate.
How often does it happen
Three independent recurrences recorded for this milestone in one day, same worker_id every time: two in a preceding session (5 retries total across ~13 minutes, including one 5-minute wait), and a third confirmed here. The lease did not clear on its own across any of these attempts, spanning well over an hour of elapsed real time in total.
Impact on work
Milestone planning was fully blocked for over an hour. A complete, ready-to-persist slice decomposition had been fully worked out during a planning-lane unit and could not be persisted because every gsd_plan_milestone call was rejected before it ever reached validation or the DB write. No automated recovery path exists; the only viable fix is an operator manually finding and terminating the leaked process (or its host's @opengsd/gsd-pi MCP server instance), which is outside what an agent session is equipped or permitted to do (no tool exposes the lease holder's OS PID, and direct gsd.db access is disallowed to avoid WAL corruption).
Workaround used
None successful from within the agent session. Waiting out the reported TTL does not work — proven across 5+ retries over 13+ minutes in one sitting. The only known path forward is operator intervention: locate and terminate the orphaned process behind the lease-holder worker id on the local host (or restart whatever supervises the @opengsd/gsd-pi MCP server instance for the project), after which the lease should expire and clear naturally within LEASE_TTL_SECONDS (60s) once nothing is left renewing it.
Proposed fixes
Corroborate workers.status === 'active' against real OS PID liveness (isWorkerProcessAlive) in executePlanMilestone's (and the equivalent slice/task planning executors') lease-conflict branch before returning a conflict, mirroring the pattern doctor-engine-checks.ts's reportOrphanedRunningAttempts already uses for task Attempts (No sanctioned way to reconcile a task's canonical lifecycle after a manual repair, without side effects or irreversible typos #1749). On a confirmed-dead PID, call markWorkerCrashed() + forceReleaseLeasesForWorker() (both already implemented, currently unused by any planning-tool code path) and proceed to claim fresh instead of failing.
Add an orphaned_milestone_lease check to gsd_doctor, parallel to the existing orphaned_running_attempt check, covering the case this incident actually hit: a held, non-expired lease (because its holder keeps self-renewing) whose owning process can no longer be reached or gracefully stopped. Since the holder here appears to still be technically alive (explaining the rolling 60s renewal), this check likely needs more than a PID-liveness probe — e.g. cross-referencing the worker's registered session/host identity against whatever registry backs live-session discovery, or simply an absolute age ceiling on acquired_at regardless of renewal.
Give executePlanMilestone's lease-acquire-and-hold loop an absolute maximum total lease lifetime independent of TTL renewal, so a single abandoned in-flight call can never hold a milestone hostage indefinitely purely by virtue of its refresh timer still firing.
Expose a user/agent-facing recovery tool (or extend gsd_doctor --fix) that can force-release a specific milestone's lease once a human has confirmed via out-of-band means (e.g. ps) that the holder is abandoned — today there is no supported way to do this short of direct, disallowed gsd.db surgery.
Other context
Pairing: gsd-pi 1.20.0 on Claude Code 2.1.270 — current at filing time.
Related issues observed before
Searched the tracker before filing; none is an exact match, but three are related:
Type: Bug (state desync / availability — planning-lane tools wedged indefinitely)
Environment versions
@opengsd/gsd-pi)@opengsd/gsd-piWhat happened
A planning-lane unit called
gsd_plan_milestonefor a milestone and got:The named worker had no reachable owning session — the holder is an orphaned/unreachable worker, not a live conflict. This exact failure signature had already recurred twice earlier the same day on the same milestone. Per that prior experience, the unit waited out the reported TTL (via a wall-clock until-loop, not a bare
sleep) and retried. The second attempt returned:— exactly 60 seconds later than the first, i.e. the reported expiry is relative to each new check, not a fixed point in time that will eventually pass. This was the third occurrence of the identical failure signature on this milestone in one day: the previous session had already exhausted 5 retries across ~13 minutes of real waiting (including one 5-minute wait) with the same non-clearing result.
Ran
gsd_doctorscoped to this exact milestone/project as part of investigating a fix:{"ok": true, "issues": [], "counts": {"error": 0, "warning": 0, "info": 0}}Zero issues reported —
gsd_doctorhas no check that would ever catch this class of problem.Source-level root cause investigation
Traced the lease logic in the installed package (
@opengsd/[email protected]):src/resources/extensions/gsd/tools/workflow-tool-executors.ts,executePlanMilestone(~line 2213 onward): on a held, non-expired lease, it checksholder?.status === "active"(a DB column on theworkerstable) to decide whether this is a real conflict. If the current call is not itself an active in-process auto session for the same worker id, it returnsmilestoneLeaseConflictResultimmediately — it never corroboratesholder.statusagainst the holder's actual OS process liveness.src/resources/extensions/gsd/db/auto-workers.tsalready implements exactly that liveness check —isWorkerProcessAlive()(a realprocess.kill(pid, 0)probe) andisDeadLocalAutoWorker()— but neither is called anywhere in theexecutePlanMilestonelease-conflict path.src/resources/extensions/gsd/doctor-engine-checks.ts,reportOrphanedRunningAttempts()(fixing issue No sanctioned way to reconcile a task's canonical lifecycle after a manual repair, without side effects or irreversible typos #1749) already solves this exact class of problem for task-level Attempts: it looks up the lease-holder's worker row and calls the sameprocess.kill(pid, 0)-style check to decide whether the holder is really dead, then reportsorphaned_running_attemptwith a documented fix path (gsd_task_settle). There is no equivalent check for orphaned milestone-lease holders —gsd_doctor's empty output against this live incident confirms the gap directly.src/resources/extensions/gsd/db/milestone-leases.ts:LEASE_TTL_SECONDS = 60.executePlanMilestonesets up aleaseRefreshTimer(setInterval, firing everyleaseRefreshMs = (milestoneLeaseTtlSeconds()/2)*1000= 30000ms) that callsrefreshMilestoneLease, extendingexpires_attonow + 60son every tick, for as long as the call's async function is alive. Thefinallyblock (which callsreleaseMilestoneLease/clearInterval) only runs if that async function actually returns or throws.The observed exact 60-second gap between successive
Retry aftertimestamps, repeated across three independent checks spanning over an hour, is the key diagnostic fact: it is not consistent with a stale DB row for a dead process (which aprocess.kill(pid, 0)check, like the one already used for #1749, could detect and would show as immediately expired/dead). It is consistent with the original Node process behind the leaseholder worker still being alive and itsleaseRefreshTimerstill firing every 30s, i.e. a leaked, still-executing process (most likely a hungawait handlePlanMilestone(...)insideexecutePlanMilestone, or an MCP server instance that never received a shutdown signal when its owning Claude Code session died) that keeps renewing its own lease forever. Nothing ties that process's lease-refresh loop to any upper bound on total lease lifetime, and nothing ties it to whether its parent interactive session is still reachable.Why is this GSD and not a user issue
No user command caused this. The session driving the original
gsd_plan_milestonecall became unreachable through ordinary auto-mode/session churn — nothing about that is a misuse of GSD's tools. GSD's own leasing subsystem is supposed to guarantee that a milestone is never blocked longer thanLEASE_TTL_SECONDSpast the last legitimate heartbeat; here a leaked process defeats that guarantee by continuing to legitimately-looking self-renew past the point its holder is reachable or useful. There is no independent circuit breaker on total lease lifetime, no cross-check between "worker row says active" and "OS process is actually alive" in the tool-facing conflict-check path (even though that exact check already exists and is used one call-site away, indoctor-engine-checks.ts), and the one tool explicitly named for catching this kind of problem (gsd_doctor) reports a clean bill of health against the live incident.What did you expect
One of:
gsd_plan_milestone/gsd_plan_slice/gsd_plan_task's lease-conflict branch corroboratesholder.status === "active"against real OS PID liveness (isWorkerProcessAlive, already implemented indb/auto-workers.tsand already consumed bydoctor-engine-checks.ts'sreportOrphanedRunningAttemptsfor the equivalent task-Attempt case) before treating a held lease as a real, unresolvable conflict — and reclaims via the already-existingforceReleaseLeasesForWorker()+markWorkerCrashed()primitives when the holder's PID is confirmed dead.gsd_doctorgains anorphaned_milestone_leasecheck parallel toorphaned_running_attempt: aheld, non-expired lease whose holder is unreachable is reported (ideally--fix-able), instead of the tool reporting a clean bill of health while a milestone sits wedged.Steps to reproduce, concrete incident
Not reliably reproducible on demand — the trigger is whatever leaves the process behind a
gsd_plan_milestonecall still running while its owning session becomes unreachable. Best known repro shape from this incident:/gsd auto(or otherwise dispatch a planning-lane unit) and let it reach agsd_plan_milestonecall for a fresh milestone — this registers a newworkersrow and claims a lease with a liveleaseRefreshTimer.finallyblock ever running (e.g. an abrupt process-tree kill or orchestrator handoff rather than a graceful/gsd stop).gsd_plan_milestone(or any lease-gated planning tool) for the same milestone.Milestone X is currently leased by <old-worker-id>. Retry after <ts>.— and on each subsequent retry (after waiting out the previously reported<ts>), a new<ts>exactly ~60s further in the future, indefinitely.Config and state (if relevant)
LEASE_TTL_SECONDS = 60(db/milestone-leases.ts).leaseRefreshMs = (milestoneLeaseTtlSeconds() / 2) * 1000= 30000ms (executePlanMilestone,workflow-tool-executors.ts).gsd_doctorrun scoped to the milestone against this exact live incident →{"ok": true, "issues": [], "counts": {"error": 0, "warning": 0, "info": 0}}.gsd_milestone_statusthroughout →status: "queued",sliceCount: 0— the milestone had a fully-prepared CONTEXT/RESEARCH pair but zero persisted slice plan, solely because everygsd_plan_milestonecall was rejected at the lease gate.How often does it happen
Three independent recurrences recorded for this milestone in one day, same
worker_idevery time: two in a preceding session (5 retries total across ~13 minutes, including one 5-minute wait), and a third confirmed here. The lease did not clear on its own across any of these attempts, spanning well over an hour of elapsed real time in total.Impact on work
Milestone planning was fully blocked for over an hour. A complete, ready-to-persist slice decomposition had been fully worked out during a planning-lane unit and could not be persisted because every
gsd_plan_milestonecall was rejected before it ever reached validation or the DB write. No automated recovery path exists; the only viable fix is an operator manually finding and terminating the leaked process (or its host's@opengsd/gsd-piMCP server instance), which is outside what an agent session is equipped or permitted to do (no tool exposes the lease holder's OS PID, and directgsd.dbaccess is disallowed to avoid WAL corruption).Workaround used
None successful from within the agent session. Waiting out the reported TTL does not work — proven across 5+ retries over 13+ minutes in one sitting. The only known path forward is operator intervention: locate and terminate the orphaned process behind the lease-holder worker id on the local host (or restart whatever supervises the
@opengsd/gsd-piMCP server instance for the project), after which the lease should expire and clear naturally withinLEASE_TTL_SECONDS(60s) once nothing is left renewing it.Proposed fixes
workers.status === 'active'against real OS PID liveness (isWorkerProcessAlive) inexecutePlanMilestone's (and the equivalent slice/task planning executors') lease-conflict branch before returning a conflict, mirroring the patterndoctor-engine-checks.ts'sreportOrphanedRunningAttemptsalready uses for task Attempts (No sanctioned way to reconcile a task's canonical lifecycle after a manual repair, without side effects or irreversible typos #1749). On a confirmed-dead PID, callmarkWorkerCrashed()+forceReleaseLeasesForWorker()(both already implemented, currently unused by any planning-tool code path) and proceed to claim fresh instead of failing.orphaned_milestone_leasecheck togsd_doctor, parallel to the existingorphaned_running_attemptcheck, covering the case this incident actually hit: aheld, non-expired lease (because its holder keeps self-renewing) whose owning process can no longer be reached or gracefully stopped. Since the holder here appears to still be technically alive (explaining the rolling 60s renewal), this check likely needs more than a PID-liveness probe — e.g. cross-referencing the worker's registered session/host identity against whatever registry backs live-session discovery, or simply an absolute age ceiling onacquired_atregardless of renewal.executePlanMilestone's lease-acquire-and-hold loop an absolute maximum total lease lifetime independent of TTL renewal, so a single abandoned in-flight call can never hold a milestone hostage indefinitely purely by virtue of its refresh timer still firing.gsd_doctor --fix) that can force-release a specific milestone's lease once a human has confirmed via out-of-band means (e.g.ps) that the holder is abandoned — today there is no supported way to do this short of direct, disallowedgsd.dbsurgery.Other context
Pairing: gsd-pi 1.20.0 on Claude Code 2.1.270 — current at filing time.
Related issues observed before
Searched the tracker before filing; none is an exact match, but three are related:
project_root_realpathmismatch across worktrees. Also a self-deadlock variant, not an orphaned-holder case.isWorkerProcessAlive) thatreportOrphanedRunningAttemptsuses — but that check was only wired up for task-level Attempts, never for the milestone-lease conflict-check path this report is about.