Skip to content

feat: show what a chat lifecycle hook changed - #27655

Merged
ibetitsmike merged 16 commits into
mainfrom
mike/chat-hooks-uat/hook-output-fidelity
Aug 3, 2026
Merged

feat: show what a chat lifecycle hook changed#27655
ibetitsmike merged 16 commits into
mainfrom
mike/chat-hooks-uat/hook-output-fidelity

Conversation

@ibetitsmike

@ibetitsmike ibetitsmike commented Jul 29, 2026

Copy link
Copy Markdown
Collaborator

Stack Context

Follow-up fixes from live UAT of the merged chat lifecycle hooks stack (#27430). Two PRs:

  1. This PR: make hook effects visible and correctly attributed in the transcript.
  2. [mike/chat-hooks-uat/dispatch-capacity]: reserve dispatch capacity so an admission burst can't fail running turns.

Why?

UAT found three ways the transcript misrepresented what a lifecycle hook did. All three are user-visible and share the same surface (chathooks/effects.go, codersdk.ChatMessagePart, the conversation timeline), so they're reviewed together.

A prompt input_override silently discarded attachments. ComposeUserPromptContent replaced the entire submitted part list with one text part, dropping file and file-reference parts along with their chat_file_links. The user saw their attachments vanish with no explanation. The override now replaces submitted text parts only and preserves non-text parts in order. A consumer that wants to block attachments uses deny, which is the documented mechanism for refusing a submission.

Every user-visible system row was labelled "Lifecycle hook". The timeline keyed the notice off role === "system". That was correct only by accident, because the hook user_message was the sole client-visible system row. The backend now emits the notice as a typed hook-notice part and the timeline renders on that, so a future system row can't be mislabelled as a policy notice.

Nothing marked a tool call the hook had rewritten. A consumer could replace tool input via input_override and the transcript showed the rewritten input as if the model had produced it. ChatMessagePart gains hook_rewritten, set from preflight.Overrides on the same path that already carries ToolCallCreatedAt, and the tool row renders a "Modified by policy" badge.

ToolCall.PolicyProvider renders the badge itself, at four wrap sites: the Tool dispatch wrapper, the ReadFilesTool aggregate and its per-file rows, and ReadFileTimelineBlock (grouped and single read_file rows bypass Tool). Renderer props do not include the flag; descendants consume it through the provider context.

The badge is emitted by the provider rather than by the shared header because several renderer branches return early without one, including the auth-required execute card, a completed ask_user_question, and an empty question payload. Those branches would drop the attribution with no type or runtime error, and the gap is not greppable: every renderer file contains a header somewhere, only individual branches do not. Emitting at the provider removes the possibility instead of enumerating the cases.

A rewritten call is wrapped in a group labelled by its badge, so one rewritten file inside a merged read is attributed on its own rather than inheriting the group's badge. HeaderButton still appends the policy wording to an explicit ariaLabel, since an explicit aria-label replaces the name computed from descendants.

Provider-executed calls are excluded from attribution. Hooks never see them, and duplicate tool-call ID rejection deliberately skips them, so a reused ID would otherwise mark a provider-executed call as policy-rewritten.

Testing

Go: coderd/x/chatd/..., coderd/x/agenthooks/..., codersdk/..., and coderd -run 'Hook|Chat'. Frontend: tsc plus every AgentsPage story; the only failures are MCP Tool Completed and Scroll To Bottom Button Works With Inverse Scroll, both of which fail on trunk.

A registry-wide story asserts every registered renderer shows the badge, verified against three inverted toggles: removing the badge, hiding it with display:none, and skipping the provider for one renderer (which names that renderer). Storybook also covers the rewritten subagent spawn, a completed empty question payload, a non-hook system message, and a failed read_file guarding the accessible name.

Mux opened this PR on Mike's behalf.

@ibetitsmike
ibetitsmike requested review from a team and DanielleMaywood as code owners July 29, 2026 18:58
@github-actions

github-actions Bot commented Jul 29, 2026

Copy link
Copy Markdown
Contributor

Docs preview

Check off each page once it's been reviewed. If a page changes in a later push, its checkbox clears automatically so it gets a fresh look. Pages not yet wired into the docs navigation aren't listed here.

@ibetitsmike

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 0017be0fed

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread site/src/pages/AgentsPage/components/ChatElements/tools/ToolCall.tsx Outdated
@ibetitsmike
ibetitsmike force-pushed the mike/chat-hooks-uat/hook-output-fidelity branch from 0017be0 to 9e7825f Compare July 29, 2026 19:13
@ibetitsmike

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 9e7825fd3d

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread site/src/pages/AgentsPage/components/ChatConversation/ConversationTimeline.tsx Outdated
@ibetitsmike ibetitsmike changed the title fix(coderd/x/chatd): show what a chat lifecycle hook changed feat: show what a chat lifecycle hook changed Jul 29, 2026
@ibetitsmike ibetitsmike changed the title feat: show what a chat lifecycle hook changed feat: show what a chat lifecycle hook changed Jul 29, 2026
@ibetitsmike ibetitsmike changed the title feat: show what a chat lifecycle hook changed feat: show what a chat lifecycle hook changed Jul 29, 2026
@ibetitsmike
ibetitsmike force-pushed the mike/chat-hooks-uat/hook-output-fidelity branch from 9e7825f to 83e6f0c Compare July 29, 2026 19:29
@ibetitsmike

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 83e6f0c73d

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread site/src/pages/AgentsPage/components/ChatElements/tools/ToolCall.tsx Outdated
Comment thread site/src/pages/AgentsPage/components/ChatConversation/messageHelpers.ts Outdated
@ibetitsmike
ibetitsmike force-pushed the mike/chat-hooks-uat/hook-output-fidelity branch from 83e6f0c to 278af8e Compare July 29, 2026 19:39
@ibetitsmike

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 278af8ec04

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@ibetitsmike

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 278af8ec04

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread site/src/pages/AgentsPage/components/ChatElements/tools/Tool.tsx
@ibetitsmike
ibetitsmike force-pushed the mike/chat-hooks-uat/hook-output-fidelity branch from 278af8e to 3904bbb Compare July 29, 2026 20:20
@ibetitsmike

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 3904bbb5f2

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@ibetitsmike
ibetitsmike force-pushed the mike/chat-hooks-uat/hook-output-fidelity branch from 3904bbb to 92de546 Compare July 29, 2026 20:29
@ibetitsmike

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 92de546247

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread site/src/pages/AgentsPage/components/ChatConversation/ConversationTimeline.tsx Outdated
@ibetitsmike
ibetitsmike force-pushed the mike/chat-hooks-uat/hook-output-fidelity branch from 92de546 to 5fb46b8 Compare July 29, 2026 20:37
@ibetitsmike

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 5fb46b8c09

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread site/src/pages/AgentsPage/components/ChatElements/tools/Tool.tsx
@ibetitsmike
ibetitsmike force-pushed the mike/chat-hooks-uat/hook-output-fidelity branch from 5fb46b8 to 28381f5 Compare July 29, 2026 20:46
Tool headers now read `hookRewritten` from the context `ToolCall.Root`
already creates, instead of a separate exported `PolicyProvider` that
every caller had to remember to wrap. Renderers forward the flag to
their root the same way they already forward `status` and `isError`.

The two rows that render a bare `ToolCall.PolicyBadge` (the execute
auth-required card and the answered ask_user_question row) now sit
inside a root, so a badge rendered outside one throws instead of
silently rendering nothing.
The registry-wide story counted rendered cases, so the extra subagent and
unknown-tool fixtures could offset a registered renderer that produced
nothing and the count assertion still passed. It now records which
registry names actually rendered and requires all of them.

Folds the negative auth-required badge assertion into the story it
duplicated, and makes the internal renderer prop required now that
`Tool` normalizes the public one.
@ibetitsmike
ibetitsmike force-pushed the mike/chat-hooks-uat/hook-output-fidelity branch from c0740e0 to d51a378 Compare July 30, 2026 16:44
@ibetitsmike

Copy link
Copy Markdown
Collaborator Author

Correction to my previous comment, and a rebase note.

Main had drifted into a conflict with this branch, which meant GitHub could not
build the merge ref and no pull_request CI ran at all on my last two pushes.
Only contrib (a pull_request_target workflow) was running, so the PR looked
quiet rather than broken. Rebased onto main; ci is running again now.

The rebase invalidated part of what I told you. Two of the files I described were
deleted on main as dead code while I was working:

So one of the two "badge outside a root" sites no longer exists. I described
converting the execute auth-required card into a ToolCall.Root; that component is
gone, and I took main's deletion rather than reviving it. The remaining conversion
is real and still in the diff: AskUserQuestionTool's answered row.

Corrected numbers for the threading cost, since I quoted the pre-rebase ones:

Before Now
Registry renderers 19 18
Renderers forwarding the flag 21 19
Leaf components taking the prop 19 (17 files) 19 (17 files)

Everything else in my previous comment stands: the flag lives on the context
ToolCall.Root already owns, the exported PolicyProvider is deleted, and
PolicyBadge throws outside a root instead of silently rendering nothing. I
re-ran the red toggle after the rebase and the registry story still fails with
expected [ 'list_templates' ] to strictly equal [] when one forwarding hop is
dropped.

Post-rebase validation: make gen produced zero drift, tsc --noEmit clean,
make pre-commit green, and 13 of 14 affected story files pass. The one failure,
MCP Tool Completed, is inherited; my diff touches no MCP code.

Mux investigated and implemented this on Mike's behalf.

@ibetitsmike

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: d51a378ea2

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread site/src/pages/AgentsPage/components/ChatElements/tools/Tool.stories.tsx Outdated
FE10 forbids `querySelector` in tests. The registry-wide badge story
scanned a private `data-policy-case` attribute and cast each node, so a
markup change could silently stop the assertions from finding anything.
It now walks the fixture table and looks each case up with
`getByTestId`, which is the sanctioned query for a wrapper that has no
semantic role.
@ibetitsmike

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: df3bf63067

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread site/src/pages/AgentsPage/components/ChatElements/tools/AskUserQuestionTool.tsx Outdated
Comment thread site/src/pages/AgentsPage/components/ChatElements/tools/Tool.stories.tsx Outdated
Comment thread site/src/pages/AgentsPage/components/ChatElements/tools/Tool.stories.tsx Outdated
Comment thread coderd/x/chatd/message_conversion.go
…alls

Hooks never see provider-executed tool calls, so a provider-executed call
that reused an admitted call's ID inherited hook_rewritten and the API
attributed it to policy. Duplicate-ID rejection deliberately skips
provider-executed calls, so the collision reaches conversion.
Threading `hookRewritten` from `Tool` into each renderer and on into every
`ToolCall.Root` touched 21 files to express one cross-cutting display flag.
`ToolCall.PolicyProvider` returns, and the provider now renders the badge
itself instead of relying on the shared header.

Emitting the badge at the provider closes the gap the prop threading was
chasing: a renderer branch that returns early without a `ToolCall.Header`,
such as the auth-required execute card or an empty `ask_user_question`
payload, keeps the attribution without participating at all. Renderers no
longer see the flag, and 15 of them leave the diff entirely.

A rewritten call is wrapped in a group labelled by its badge, so a nested
row such as one file inside a merged read carries its own attribution.
`ReadFileTool` names its header row, which had been relying on the badge
sitting inside the button to describe the row.
The coverage loop used a text query, which matches a badge the card's own
layout hides. Verified by hiding the badge with display:none, which the
previous assertion accepted.
@ibetitsmike
ibetitsmike force-pushed the mike/chat-hooks-uat/hook-output-fidelity branch from 2b18c61 to c8259c6 Compare July 30, 2026 17:56
The fallback that renders merged tools without a parsed block omitted
hookRewritten, so a rewritten call reaching that path lost the badge.
@ibetitsmike

Copy link
Copy Markdown
Collaborator Author

Reverted the prop-piping refactor from the previous round.

Mike pushed back: threading hookRewritten through every renderer was over-plumbing for one cross-cutting display flag. ToolCall.PolicyProvider is back, with one structural change: the provider renders the badge itself instead of delegating to the shared header.

That closes the defect class you found three times (ask_user_question completed, ExecuteAuthRequired, empty-payload) by construction rather than by enumeration. A renderer branch that returns early without a ToolCall.Header now keeps the attribution because no renderer participates at all. 15 renderer files left the diff entirely.

Also in this round:

  • !part.ProviderExecuted guard plus a test, per your P2.
  • Production query-key helpers imported instead of retyped arrays, per your P1.
  • The coverage story queries role="group" by accessible name; no test IDs remain.
  • Self-found: the coverage loop used queryByText, which passes on a badge that is present but hidden. Now asserts checkVisibility().
  • Self-found: the remaining-tools fallback in ConversationTimeline never forwarded the flag.

One pre-existing issue I am deliberately not fixing here: ApplyAdmittedToolCalls looks up Overrides[toolCall.ToolCallID] without a ProviderExecuted check, so the same ID collision would replace a provider-executed call's input, not just its badge. Byte-identical on origin/main.

@codex review

Mux is working on behalf of Mike.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: aa411153eb

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread site/src/pages/AgentsPage/components/ChatElements/tools/ReadFileTool.tsx Outdated
The previous commit gave ReadFileTool an explicit ariaLabel so a grouped
row could be told apart by name. An explicit aria-label replaces the name
computed from descendants, dropping the failure text ToolCall.Status
contributes, so a failed read announced only the filename.

The grouped story reads attribution from the policy group instead, and
ReadFileFailed guards the failure name.
@ibetitsmike

Copy link
Copy Markdown
Collaborator Author

Fixed the read-file accessible name regression from the last round (61b94b96b5d) and re-requested review below.

That one was mine: I had added an explicit ariaLabel purely so a story could tell two grouped rows apart, and it replaced the name computed from descendants, dropping the failure text. The story now reads attribution from the policy group instead, and ReadFileFailed guards the failure name with verified red-green.

@codex review

Mux is working on behalf of Mike.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 61b94b96b5

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread site/src/pages/AgentsPage/components/ChatElements/tools/Tool.stories.tsx Outdated
The story seeded workspaceByIdKey with an untyped partial, so a change to
the Workspace or WorkspaceBuild shape could not surface here. It now
composes MockWorkspace and MockWorkspaceBuild under an explicit
TypesGen.Workspace annotation, which rejects the previous partial.
@ibetitsmike

Copy link
Copy Markdown
Collaborator Author

Fixed the fixture finding in e18c880d58c and re-requested review below.

The policy story now composes MockWorkspace / MockWorkspaceBuild under an explicit TypesGen.Workspace annotation. Verified the annotation actually rejects the old partial (TS2740, 17 missing properties), so this is enforced rather than cosmetic.

That change also surfaced a latent inconsistency in my own fixture: latest_build.id and the seeded build-logs key were unrelated strings, even though WorkspaceBuildLogSection falls back to latest_build.id when a tool result has no build_id. Both now share one constant.

@codex review

Mux is working on behalf of Mike.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Swish!

Reviewed commit: e18c880d58

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread site/src/pages/AgentsPage/components/ChatElements/tools/ToolCall.tsx Outdated
@ibetitsmike

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Breezy!

Reviewed commit: f0fa131fcb

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@ibetitsmike
ibetitsmike merged commit df1c0f9 into main Aug 3, 2026
32 checks passed
@ibetitsmike
ibetitsmike deleted the mike/chat-hooks-uat/hook-output-fidelity branch August 3, 2026 16:27
@github-actions github-actions Bot locked and limited conversation to collaborators Aug 3, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants