Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 8 additions & 5 deletions coderd/database/querier.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

40 changes: 25 additions & 15 deletions coderd/database/queries.sql.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

18 changes: 12 additions & 6 deletions coderd/database/queries/chattoolcallexecutions.sql
Original file line number Diff line number Diff line change
Expand Up @@ -33,11 +33,14 @@ ON CONFLICT (chat_id, assistant_message_id, tool_call_id) DO NOTHING;
-- claimable: its call is resolved in chat, so re-dispatching it
-- could run the command twice. The stale-takeover arm requires
-- stale_epoch to match the exact claim generation the caller
-- verified: evidence gathered against one claim cannot take over
-- a newer one, and a caller without a verified epoch (NULL) can
-- never take over a starting claim. Zero rows means the row
-- exists but is not claimable; the caller reads it to decide how
-- to proceed.
-- verified through the agent: evidence gathered against one claim
-- cannot take over a newer one, and a caller without a verified
-- epoch (NULL) can never take over a starting claim.
-- workspace_agent_id records the dispatch target before the
-- dispatch happens, so recovery can tell whether a token probe
-- reaches the agent the dead claimer actually targeted. Zero rows
-- means the row exists but is not claimable; the caller reads it
-- to decide how to proceed.
INSERT INTO chat_tool_call_executions (
id,
chat_id,
Expand All @@ -48,6 +51,7 @@ INSERT INTO chat_tool_call_executions (
command,
background,
timeout_secs,
workspace_agent_id,
claim_epoch,
claimed_at,
created_at,
Expand All @@ -62,6 +66,7 @@ INSERT INTO chat_tool_call_executions (
@command::text,
@background::boolean,
@timeout_secs::bigint,
sqlc.narg('workspace_agent_id')::uuid,
1,
@now::timestamptz,
@now::timestamptz,
Expand All @@ -72,6 +77,7 @@ ON CONFLICT (chat_id, assistant_message_id, tool_call_id) DO UPDATE SET
command = EXCLUDED.command,
background = EXCLUDED.background,
timeout_secs = EXCLUDED.timeout_secs,
workspace_agent_id = EXCLUDED.workspace_agent_id,
claim_epoch = chat_tool_call_executions.claim_epoch + 1,
claimed_at = EXCLUDED.claimed_at,
updated_at = EXCLUDED.updated_at
Expand Down Expand Up @@ -107,7 +113,7 @@ SET status = CASE WHEN status = 'starting' THEN 'running'::chat_tool_call_execut
-- rows: a late handle write anchored at an older start time
-- must never move the lease backward and reopen a fresh
-- interrupt to early sweep reclaim.
updated_at = GREATEST(updated_at, @started_at::timestamptz)
updated_at = GREATEST(updated_at, @updated_at::timestamptz)
WHERE chat_id = @chat_id::uuid
AND assistant_message_id = @assistant_message_id::bigint
AND tool_call_id = @tool_call_id::text
Expand Down
8 changes: 4 additions & 4 deletions coderd/x/chatd/ARCHITECTURE.md
Original file line number Diff line number Diff line change
Expand Up @@ -890,7 +890,7 @@ The `status` column tracks the external process lifecycle:
| Status | Meaning |
|---|---|
| `reserved` | Intent persisted with the assistant message; nothing dispatched. |
| `starting` | A runner claimed the row (`claim_epoch` advanced, `claimed_at` set); dispatch may be in flight. |
| `starting` | A runner claimed the row (`claim_epoch` advanced, `claimed_at` and the `workspace_agent_id` dispatch target set); dispatch may be in flight. |
| `running` | Process identity recorded (`process_id`, `workspace_agent_id`, `started_at`). |
| `exited` | The tool observed the process exit (fresh wait or re-attach snapshot). |
| `detached` | The chat moved on and the process was deliberately left alive: background start, timed-out foreground, or interrupted background. |
Expand All @@ -913,14 +913,14 @@ Ledger writes that must agree with chat state run inside the same `ChatMachine.U

### Claim and re-attach

Before dispatching, the execute tool claims the row: a compare-and-set that takes a `reserved` intent or a stale `starting` claim (one whose `claimed_at` is older than the staleness window), advancing `claim_epoch`. A row missing entirely (an assistant message that predates the ledger) is claimed by insert. A claim that fails on infrastructure (not the input-hash guard) aborts the call instead of committing an error result: nothing was dispatched, so the task retry re-claims the intent. An abort fails the attempt with a retryable error and never commits a result for the aborted call, but completed sibling results from the same batch are committed first: their side effects already happened, so the retry re-runs only the calls left without a result. The claimer calls `StartProcess` and records the process identity, guarded by `claim_epoch` so a superseded claimer never overwrites the current claim's process. Lifecycle observations (`exited`, `detached`) are recorded only when that identity write landed: a terminal row without a process handle would strand a retry, while a still-`starting` row resolves through claim recovery.
Before dispatching, the execute tool claims the row: a compare-and-set that takes a `reserved` intent or a stale `starting` claim (one whose `claimed_at` is older than the staleness window), advancing `claim_epoch`. A row missing entirely (an assistant message that predates the ledger) is claimed by insert. A claim that fails on infrastructure (not the input-hash guard) aborts the call instead of committing an error result: nothing was dispatched, so the task retry re-claims the intent. An abort fails the attempt with a retryable error and never commits a result for the aborted call, but completed sibling results from the same batch are committed first: their side effects already happened, so the retry re-runs only the calls left without a result. The claim records the dispatch target (`workspace_agent_id`) before the dispatch happens, so recovery can tell whether a later token probe reaches the agent that actually received it. The claimer calls `StartProcess` and records the process identity, guarded by `claim_epoch` so a superseded claimer never overwrites the current claim's process. A `StartProcess` failure with a structured agent response (other than a 409) resolves the row `no_effect` and commits the error: the agent released the token reservation, so nothing is running. A dispatch that reclaimed a stale claim instead aborts uncommitted on such a failure: the superseded claimer may still spawn under the released token, so the row stays `starting` and retries keep probing the token that could observe that late process. The agent answers 409 both for a token parameter mismatch and for an aborted wait on a reservation whose owner may still publish a process, so 409 never commits. A transport failure or token conflict instead aborts the attempt uncommitted with the row still `starting`, so the retried call resolves the dispatch through the agent's token index; recoverable stale-claim failures (a transient probe error, or a found process whose adoption write did not land) abort the same way. Lifecycle observations (`exited`, `detached`) are recorded only when that identity write landed: a terminal row without a process handle would strand a retry, while a still-`starting` row resolves through claim recovery.

An attempt that cannot claim decides from the row's status:

- A fresh `starting` claim is polled until its owner records a process or the claim goes stale; a stale claim yields `unknown`. The unknown write matches only rows still `starting`: the owner's process record can land concurrently with the staleness verdict, and that write wins, with the attempt resuming the recorded process instead.
- A fresh `starting` claim is polled until its owner records a process or the claim goes stale. A stale claim is resolved through the token index of the agent it dispatched to (the execution ID doubles as the agent-side idempotency token): when the turn's connection targets a different agent, recovery dials the recorded dispatch target directly, because only that agent can observe what became of the dispatch. A row with no recorded target, or no dialer to reach it, resolves to `unknown`; a dial failure aborts the attempt uncommitted while the claim is within the trust window (a transient failure proves nothing and a retry can probe once the owner reconnects) and resolves to `unknown` past it, when even a successful probe could no longer trust an absence answer. On the probed agent: a found process is adopted onto the stale claim's epoch, attributed to that agent, and re-attached; a token the agent does not know, answered within the trust window by a token index that predates the claim, proves the dispatch never happened, which on the turn's own agent permits taking over the claim and re-dispatching under the same token, which the agent dedups against any race with the original dispatch (an attached process keeps the original claim time as its start lower bound, so it never gets a fresh timeout), while a trustworthy absence on a prior agent only aborts, since that agent's index cannot fence a dispatch on the current one; anything else (old agents without the probe route, answers past the trust window, a token index younger than the claim after an agent restart, unobservable outcomes) resolves to `unknown`. Recovery never mints a new token for a tool call. The unknown write matches only rows still `starting`: the owner's process record can land concurrently with the staleness verdict, and that write wins, with the attempt resuming the recorded process instead.
- Background rows with a process handle return the started-in-background result directly; output retrieval stays with `process_output`. Foreground `running`, `exited`, and `detached` rows re-attach via an output snapshot. An exited process yields the real result, even past the deadline, and marks `exited`. A running process with time left is block-waited for the remainder. A running process past the deadline yields the graceful timed-out result with `background_process_id` and marks `detached`.
- Only a definite HTTP 404 from the snapshot (the agent was reached and does not know the process) marks `unknown` and produces an `is_error` result stating the command may have run but its outcome is unknown, and only when the connection targets the agent recorded as owning the process: a chat rebound to a different agent asks the wrong agent, so its 404 proves nothing. Transport errors, cancellations, and server errors keep the lifecycle state unchanged and the process retrievable via `process_output`. A 404 from a connection targeting a different agent than the recorded owner instead aborts the attempt: the owning agent was never asked, so committing a result would end re-attachment on no evidence.
- Rows already resolved (`cancel_requested`, `canceled`, `unknown`, `no_effect`) never re-dispatch; the tool returns a stable error result. These rows need no agent access, so the stable result is also returned when the workspace connection cannot be established, instead of coupling an already-resolved call to agent availability. Background rows with a recorded handle likewise resolve without a connection: their durable result is the handle itself. Rows that still need agent access (fresh `starting`, and foreground `running`/`exited`/`detached`) abort the attempt on a connection failure instead of committing an error result that would end re-attachment. A stale `starting` row instead resolves through the same guarded unknown write as the connected path, which needs no agent access: aborting it would wedge the chat for as long as the agent stays unreachable; only a `reserved` row, which proves nothing was dispatched, surfaces the dial error as a normal tool result.
- Rows already resolved (`cancel_requested`, `canceled`, `unknown`, `no_effect`) never re-dispatch; the tool returns a stable error result. These rows need no agent access, so the stable result is also returned when the workspace connection cannot be established, instead of coupling an already-resolved call to agent availability. Background rows with a recorded handle likewise resolve without a connection: their durable result is the handle itself. Rows that still need agent access (fresh `starting`, and foreground `running`/`exited`/`detached`) abort the attempt on a connection failure instead of committing an error result that would end re-attachment. A stale `starting` row still aborts while its claim is within `TokenTrustWindow`, because the token probe can adopt a live process once the agent is reachable again; past the window (when absence answers would prove nothing anyway) it resolves through the same guarded unknown write as the connected path, which needs no agent access, so an agent that never returns cannot wedge the chat forever; only a `reserved` row, which proves nothing was dispatched, surfaces the dial error as a normal tool result.

The execute timeout is clamped to 4 hours at claim time and the clamped value is stored on the row so re-attaching attempts agree on the deadline.

Expand Down
Loading
Loading