Skip to content

fix(coderd/x/chatd): keep task attempts alive while execute tools make progress - #27103

Closed
ibetitsmike wants to merge 10 commits into
mike/codagt-757-agent-idempotency-tokenfrom
mike/codagt-757-attempt-watchdog
Closed

ibetitsmike wants to merge 10 commits into
mike/codagt-757-agent-idempotency-tokenfrom
mike/codagt-757-attempt-watchdog

Conversation

@ibetitsmike

@ibetitsmike ibetitsmike commented Jul 8, 2026

Copy link
Copy Markdown
Collaborator

This PR was authored by Mux, an AI coding agent, working on behalf of Mike.

Problem

Chatd task retries respawn execute tool processes, leaving orphaned duplicates in workspaces (CODAGT-757). The earlier PRs in this stack make retries re-attach to the original process instead of starting a new one, but every task attempt still dies at a fixed 15 minute timeout. A legitimately long-running execute (the clamp allows up to 4 hours) can never finish within one attempt: the attempt times out mid-wait, the task retries, re-attaches, and times out again, burning retries on a command that is making progress.

What this PR changes

  • Redefines the 15 minute task attempt timeout in coderd/x/chatd/tasks.go as a resettable idle window, and adds a non-resettable 24 hour absolute cap (maxTaskAttemptDuration). Both cancel the attempt with the existing retryable errTaskTimeout.
  • Adds chattool.WithAttemptKeepalive / chattool.KickAttemptKeepalive (new coderd/x/chatd/chattool/keepalive.go) using an unexported typed context key. Kicking is a safe no-op when the keepalive is absent or the attempt already ended. taskAttemptContext installs the kick into the attempt context.
  • Kicks the watchdog in chattool only, after each successful agent round-trip: StartProcess returns (foreground and background), every successful ProcessOutput round inside waitForProcess (including rounds where the process is still running), the successful re-attach snapshot, and the process_output tool. Other workspace tools finish well within the idle window and deliberately do not extend it.
  • Documents the attempt watchdog and the keepalive scoping in coderd/x/chatd/ARCHITECTURE.md.
  • Clamps wait_agent's model-supplied timeout_seconds to 15 minutes per call (maxSubagentWaitTimeout), so a sibling tool cannot ride a progressing execute's keepalives for hours on one blocking call; a wait_agent timeout stays a resumable non-failure. A single computer wait action is likewise clamped to 30 seconds.
  • Extracts the settle-once watchdog timer that chatloop's streamSilenceGuard already used into coderd/x/chatd/internal/watchdog and backs both the stream-silence guard and the new idle window with it, so the concurrency-sensitive timer lifecycle exists once. Quartz timer tags are unchanged; the disarm-versus-fire race tests move to the shared package.

What this PR does not change

  • Streaming code never kicks the watchdog. The 10 minute stream-silence guard and the effective 15 minute window for non-tool work are unchanged, so silent provider streams keep failing through chat-specific retry handling first.
  • No database or API changes.

Test plan

  • go test ./coderd/x/chatd/... (full chatd tree, passes)
  • TestTaskAttemptContext_* tests with quartz.NewMock: kicks extend the attempt past the idle window, the 24h cap fires regardless of kicks, a no-kick attempt still dies at 15m, kicking after cancel or timeout is a no-op.
  • Chattool tests: keepalive kicked once per successful start, poll round, and re-attach snapshot; process_output tool kicks on success.
  • internal/watchdog tests: fires with its cause, reset restarts the window, the disarm-versus-fire race cancels at most once, disarm preserves the context cause.
  • make fmt, make lint, make gen: clean, no dirty diff.

Stack

PR 3 of 3 for CODAGT-757: builds on #27102 (agent idempotency token), which builds on #27100 (execution ledger).

@linear-code

linear-code Bot commented Jul 8, 2026

Copy link
Copy Markdown

CODAGT-757

@ibetitsmike
ibetitsmike force-pushed the mike/codagt-757-agent-idempotency-token branch from 8cf446d to 26369ad Compare July 8, 2026 19:00
@ibetitsmike
ibetitsmike force-pushed the mike/codagt-757-attempt-watchdog branch from cfe86d1 to 7f46733 Compare July 8, 2026 19:40
@ibetitsmike

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Hooray!

Reviewed commit: 7f4673359b

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@ibetitsmike
ibetitsmike force-pushed the mike/codagt-757-attempt-watchdog branch from 7f46733 to 3135ef7 Compare July 12, 2026 13:39
@ibetitsmike
ibetitsmike force-pushed the mike/codagt-757-agent-idempotency-token branch from ceb245f to 0bc4075 Compare July 12, 2026 13:39
@ibetitsmike
ibetitsmike force-pushed the mike/codagt-757-attempt-watchdog branch from 3135ef7 to b84c305 Compare July 12, 2026 13:57
@ibetitsmike
ibetitsmike force-pushed the mike/codagt-757-agent-idempotency-token branch from 0bc4075 to 6a29678 Compare July 12, 2026 13:57
@ibetitsmike
ibetitsmike force-pushed the mike/codagt-757-attempt-watchdog branch 2 times, most recently from d2ee076 to a272bf7 Compare July 13, 2026 18:45
@ibetitsmike
ibetitsmike force-pushed the mike/codagt-757-agent-idempotency-token branch 2 times, most recently from a8ec642 to 437fcd2 Compare July 13, 2026 20:19
@ibetitsmike
ibetitsmike force-pushed the mike/codagt-757-attempt-watchdog branch from a272bf7 to 02a5363 Compare July 13, 2026 20:19
@ibetitsmike
ibetitsmike force-pushed the mike/codagt-757-agent-idempotency-token branch from 437fcd2 to 8e72ddd Compare July 15, 2026 21:25
@ibetitsmike
ibetitsmike force-pushed the mike/codagt-757-attempt-watchdog branch from 02a5363 to 57ecf81 Compare July 15, 2026 21:25
@ibetitsmike

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Keep them coming!

Reviewed commit: 57ecf819c3

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@ibetitsmike
ibetitsmike force-pushed the mike/codagt-757-attempt-watchdog branch from 57ecf81 to d1d4dfa Compare July 15, 2026 21:43
@ibetitsmike

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. 🎉

Reviewed commit: d1d4dfa833

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@ibetitsmike
ibetitsmike force-pushed the mike/codagt-757-attempt-watchdog branch from d1d4dfa to 1aad417 Compare July 15, 2026 22:11
@ibetitsmike
ibetitsmike force-pushed the mike/codagt-757-agent-idempotency-token branch from 2916ac1 to 0dcbd7e Compare July 15, 2026 22:11
@ibetitsmike

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Breezy!

Reviewed commit: 1aad417d86

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@ibetitsmike
ibetitsmike force-pushed the mike/codagt-757-attempt-watchdog branch from 1aad417 to e912b0b Compare July 15, 2026 22:32
@ibetitsmike
ibetitsmike force-pushed the mike/codagt-757-agent-idempotency-token branch from 0dcbd7e to 62ca73b Compare July 15, 2026 22:32
@ibetitsmike
ibetitsmike force-pushed the mike/codagt-757-attempt-watchdog branch from d331b9a to e06273d Compare July 16, 2026 15:15
@ibetitsmike
ibetitsmike force-pushed the mike/codagt-757-agent-idempotency-token branch from 3bed2f2 to 6e7c6f8 Compare July 16, 2026 16:20
@ibetitsmike
ibetitsmike force-pushed the mike/codagt-757-attempt-watchdog branch from e06273d to 6b59101 Compare July 16, 2026 16:20
@ibetitsmike
ibetitsmike force-pushed the mike/codagt-757-agent-idempotency-token branch from 6e7c6f8 to 9d45193 Compare July 16, 2026 17:03
@ibetitsmike
ibetitsmike force-pushed the mike/codagt-757-attempt-watchdog branch from 6b59101 to be5e13b Compare July 16, 2026 17:03
@ibetitsmike
ibetitsmike force-pushed the mike/codagt-757-agent-idempotency-token branch from 9d45193 to b4bb971 Compare July 17, 2026 11:20
@ibetitsmike
ibetitsmike force-pushed the mike/codagt-757-attempt-watchdog branch 2 times, most recently from 115b230 to 0c310f5 Compare July 17, 2026 11:59
@ibetitsmike
ibetitsmike force-pushed the mike/codagt-757-agent-idempotency-token branch from b4bb971 to 81f3737 Compare July 17, 2026 11:59
…e progress

Task attempts previously died at a fixed 15 minute timeout even while
an execute tool was actively polling a long-running process. The
timeout is now an idle window that workspace tools reset after each
successful agent round-trip, with a separate 24 hour absolute cap
that is never reset. Streaming code never resets the watchdog, so
the stream-silence guard semantics are unchanged.

Part of CODAGT-757.
attemptIdleTimer duplicated chatloop's streamSilenceGuard: the same
mutex, quartz timer, cancel cause, and settle-once lifecycle copied
into a second concurrency-sensitive type. Extract the primitive into
internal/watchdog and use it for both the stream-silence guard and
the task attempt idle window, so timer race and shutdown fixes land
in one place. The disarm-versus-fire race tests move to the shared
package; quartz timer tags are unchanged.
…output

The idle watchdog reset was documented as a general workspace-tool
behavior, but only the execute and process_output tools kick it,
after process-API round-trips. Narrow the architecture doc and the
keepalive comment to that reality and record the scoping decision:
quick tools finish well within the idle window and deliberately do
not extend it.
@ibetitsmike

Copy link
Copy Markdown
Collaborator Author

Superseded by the recomposed CODAGT-757 stack: #27317, #27318, #27319, #27320, #27321, #27322, #27323, #27324, #27325. The combined final tree of the new stack is byte-identical to the reviewed heads of this stack (minus an accidental workflow orchestration file); the recomposition splits the work into reviewable pieces.

This comment was posted by Mux, an AI coding agent, on Mike's behalf.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant