Skip to content

perf: materialize AI Bridge sessions for list query - #27996

Closed
evgeniy-scherbina wants to merge 8 commits into
mainfrom
yevhenii/optimize-sessions-query
Closed

perf: materialize AI Bridge sessions for list query#27996
evgeniy-scherbina wants to merge 8 commits into
mainfrom
yevhenii/optimize-sessions-query

Conversation

@evgeniy-scherbina

@evgeniy-scherbina evgeniy-scherbina commented Aug 10, 2026

Copy link
Copy Markdown
Contributor

Related issue: https://linear.app/codercom/issue/AIGOV-580/ai-gateway-sessions-page-takes-5-10-seconds-to-load

Follow-up PR: https://github.com/coder/coder/pull/28083/changes


Metric Baseline 6765731ea9 Branch 57b4f907f7
aibridge_interceptions 2,000,000 2,000,000
Distinct sessions 321,653 (318,747 with a completed interception) same
aibridge_user_prompts 3,999,559 3,999,559
On-disk size, aibridge tables 10 GB 10 GB
Users / providers / models 50 / 5 / 12 50 / 5 / 12
Migration 000569, full up n/a ~15.1 s (backfill 15,057 ms for 318,747 rows)
Wall clock, warm, page 1 of 25 5,003 - 5,585 ms 100 - 115 ms (~50x)

Output parity

Scenario Rows Result
Page 1, limit 25 25 identical, all columns
Page 1, limit 100 100 identical
Offset 5000, limit 25 25 identical
Provider filter openai 25 identical
Client filter codex 25 identical
Initiator filter 50 identical
started_after 2026-08-05 50 identical
Model filter gpt-5, limit 25 25 same set, 3 rows differ per side
Model filter gpt-5, limit 500 500 same set, 60 rows differ per side
started_before 2026-08-01 50 same set, 28 rows differ per side

Matching-session universes, unpaginated

Filter Baseline Branch
model = gpt-5 74,365 74,365
started_after = 2026-08-05 61,023 61,023
All sessions 318,747 318,747

@evgeniy-scherbina
evgeniy-scherbina force-pushed the yevhenii/optimize-sessions-query branch from e16d462 to 5325657 Compare August 10, 2026 18:46
@evgeniy-scherbina
evgeniy-scherbina marked this pull request as ready for review August 10, 2026 20:29
@evgeniy-scherbina

Copy link
Copy Markdown
Contributor Author

/coder-agents-review

@coder-agents-review

coder-agents-review Bot commented Aug 10, 2026

Copy link
Copy Markdown
Contributor

Chat: Review posted | View chat
Requested: 2026-08-10 20:29 UTC by @evgeniy-scherbina

Review history
  • R1 (2026-08-10), 2 Note, 1 P0, 3 P1, 1 P2, 2 P3, COMMENT. Review

deep-review v0.9.0 | Round 1 | 1c993c7..a90109e

Last posted: Round 1, 9 findings (1 P0, 3 P1, 1 P2, 2 P3, 2 Note), COMMENT. Review

Finding inventory

Finding inventory: PR #27996

Findings

# Sev Status Location Summary Round Reviewer Posted
CRF-1 P0 Open coderd/database/migrations/000562_aibridge_sessions.up.sql:1 Migration number 000562 collides with main; merged tree cannot migrate (sqlc-vet CI failure) R1 Netero Yes
CRF-2 P1 Open coderd/database/migrations/000562_aibridge_sessions.up.sql:133 Prompts recorded before interception ends never advance last_active_at; sort key wrong for normal production flow R1 Netero Yes
CRF-3 P1 Open coderd/database/queries/aibridge.sql:470 Retention purge orphans aibridge_sessions rows; list query then fails scanning NULL aggregates into non-nullable fields R1 Netero Yes
CRF-4 P1 Open coderd/database/queries/aibridge.sql:301 CountAIBridgeSessions not migrated; list and count disagree under filters, count keeps full-scan cost R1 Netero Yes
CRF-5 P2 Open coderd/database/migrations/000562_aibridge_sessions.up.sql:1 165 lines of trigger/upsert/backfill plus rewritten list semantics with zero new tests R1 Netero Yes
CRF-6 P3 Open coderd/database/migrations/000562_aibridge_sessions.up.sql:114 Trigger, backfill, and display query disagree on which client a session has R1 Netero Yes
CRF-7 P3 Open scripts/seed/main.go:1 834 LOC of debug tooling in diff; Law mandatory-split: remove scripts/ from this PR R1 Netero P3, Law (mandatory split) Yes
CRF-8 Note Open coderd/database/queries/aibridge.sql:382 Time-window filter semantics changed from per-interception start to session span overlap; user-visible, undocumented in PR description R1 Netero Yes
CRF-9 Note Open coderd/database/migrations/000562_aibridge_sessions.up.sql:146 Backfill runs full GROUP BY over aibridge_interceptions inside migration transaction; upgrade cost scales with row count R1 Netero Yes

Law analysis

  • Effective LOC: +1063 -66 (78% is scripts/seed + scripts/sqldebug, +834 LOC)
  • Head SHA: a90109e
  • Verdict: Split
  • Enforcement: Mandatory
  • Proposal: remove scripts/seed/ and scripts/sqldebug/ from this PR; the remaining ~230-LOC materialization change (migration + query rewrite + codegen) is one vertical slice and should not be split further.

Round log

Round 1

Netero + Law (mandatory split). No panel. 1 P0, 3 P1, 1 P2, 2 P3, 2 Note. Reviewed against 1c993c7..a90109e.

About deep-review

CRF = Coder Review Finding (P0-P4, Nit, Note)

Reviewer Focus
Bisky tests
Chopper ops/errors
Churn-guard change verification
Ging language modernization
Gon naming
Hisoka edge cases
Killua perf
Kite change integrity
Knov contracts
Knuckle SQL
Komugi flake/determinism
Kurapika security
Law decomposition
Leorio docs
Luffy product
Mafu-san process
Mafuuu contracts
Melody dispatch/pairing
Meruem structural
Nami frontend
Netero mechanical checks
Pariston premise testing
Pen-botter product gaps
Razor verification
Robin duplication
Ryosuke Go arch
Takumi concurrency
Zoro shape

🤖 Managed by Coder Agents.

@coder-agents-review coder-agents-review Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a well-motivated perf change: materializing sessions into a trigger-maintained table with GIN indexes is the right shape for the list query's cost problem, the migration comments are unusually thorough, and the backfill correctly avoids the prompt fan-out distortion the old lateral existed to prevent.

This is a first-pass review only: mechanical findings from Netero plus a PR-decomposition analysis from Law. The full review panel has not yet reviewed this PR and will review after these findings are addressed.

Severity count: 1 P0, 3 P1, 1 P2, 2 P3, 2 Notes.

Split required (Law, mandatory). scripts/seed/ and scripts/sqldebug/ are +834 LOC, 78% of the effective diff, independent of the feature, and you have already stated they will not merge. Remove them from this branch now rather than "later": reviews, CI, and the merge decision run against what is in the branch, and the remaining ~230-LOC change (migration + query rewrite + codegen) deserves an undiluted diff. As Law put it: "'Will be removed later before merging' is a promise, not a state of the diff."

The P0 explains the sqlc-vet CI failure: migration 000562 collides with 000562_oauth2_public_client_tokens already on main. The three P1s are semantic: the prompt trigger silently drops timestamps for the normal production flow (prompt recorded before the interception ends), retention purge orphans session rows and then breaks the list endpoint, and CountAIBridgeSessions was left on the old table so list and count disagree under filters.


coderd/database/queries/aibridge.sql:301

P1 [CRF-4] CountAIBridgeSessions was not migrated to aibridge_sessions; list and count disagree whenever filters are set, and count keeps the full-scan cost this PR exists to remove. (Netero)

Reproduced: one session with interceptions (anthropic, claude) and (openai, gpt-4); filters provider=anthropic, model=gpt-4 yield list=1, count=0. The same divergence exists for the time-window filters (span overlap vs per-interception started_at) and the client filter (stored scalar vs any interception).

The API returns both Sessions and Count from the same request, so pagination totals will contradict the visible rows. Count also still does COUNT(DISTINCT ...) over every interception on every request. Fix: count from aibridge_sessions with the identical filter block.

🤖

🤖 This review was automatically generated with Coder Agents.

Comment thread coderd/database/migrations/000562_aibridge_sessions.up.sql Outdated
SET last_active_at = GREATEST(s.last_active_at, NEW.created_at)
FROM aibridge_interceptions ai
WHERE ai.id = NEW.interception_id
AND s.session_id = ai.session_id

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 [CRF-2] Prompts recorded before their interception ends never advance last_active_at, so the sort key is wrong for the normal production flow. (Netero)

Production order is RecordPromptUsage (inserts the prompt) during the interception, then RecordInterceptionEnded (sets ended_at): coderd/aibridgedserver/aibridgedserver.go:481 and :311. For a session's first interception, the prompt trigger's UPDATE matches no aibridge_sessions row (the row is only created when an interception completes), so the prompt timestamp is silently dropped. The interception trigger then seeds last_active_at from started_at; its GREATEST only ever compares started_at values and never sees the prompt's created_at.

Reproduced empirically by Netero: interception in-flight, prompt at 00:05, interception ended at 00:06 yields last_active_at = 00:00 (started_at), where the old query returned 00:05. The trigger comment claiming the interception trigger's GREATEST "keeps whichever timestamp is later" is false. Every single-interception session, and the first interception of every session, sorts by start time instead of prompt time, diverging from the column's own documented semantics and from pre-PR behavior.

🤖

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch, confirmed.

Rather than patch it, I'd drop aibridge_user_prompts from the sync path and define last_active_at as MAX(ended_at) or MAX(started_at).

That removes the prompt trigger and the backfill's LEFT JOIN, and makes this bug structurally impossible: it exists only because prompts arrive before the session row does. Ordering barely changes.

What do you think?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I changed my mind, fixed here: 7e5b180

@@ -488,13 +470,21 @@ FROM
JOIN

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 [CRF-3] Retention purge orphans aibridge_sessions rows, and the list query then fails to scan NULL aggregates into non-nullable Go fields. (Netero)

DeleteOldAIBridgeRecords deletes interceptions but nothing deletes the corresponding aibridge_sessions row: no DELETE trigger exists and the purge CTE does not touch the table. Reproduced: after deleting a session's interceptions, the session row survives and the page lateral's MIN(ai.started_at) / MAX(ai.ended_at) return NULL.

ListAIBridgeSessionsRow.StartedAt/EndedAt are non-nullable time.Time, so scanning NULL fails and the entire ListAIBridgeSessions call errors; the sessions list endpoint breaks once retention purges any session still on a page. Fix: delete session rows whose interceptions are purged (a DELETE trigger or an extra CTE in DeleteOldAIBridgeRecords).

🤖

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed here: e320c5c

@@ -0,0 +1,165 @@
-- Materializes AI Bridge sessions so the sessions list can order and filter

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 [CRF-5] 165 lines of trigger, upsert, and backfill logic plus rewritten list-query semantics ship with zero new tests (diff test density 0.0%). (Netero)

The two proven bugs above (prompt ordering, purge orphans) are exactly the paths no test exercises: TestAIBridgeListSessions inserts data in an order that avoids the prompt race, and no test covers purge interaction or the backfill.

At minimum: a test inserting a prompt before UpdateAIBridgeInterceptionEnded, a test running DeleteOldAIBridgeRecords then listing, and a backfill assertion in the migration fixtures.

🤖

-- or, for interceptions recorded after the fact, on insert. Restricting the
-- update case to ended_at keeps later updates such as credential_hint from
-- re-running the upsert.
CREATE TRIGGER aibridge_interceptions_track_session

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3 [CRF-6] The trigger, the backfill, and the display query disagree on which client a session has. (Netero)

The trigger keeps the first non-NULL client (COALESCE(existing, EXCLUDED)); the backfill and the outer display lateral keep the first client ordered by started_at, id, which can be NULL even when later interceptions have one.

A session whose first interception has NULL client filters as client=X (trigger-maintained) but displays as empty, and the same session backfilled stores NULL and does not match the filter. The old filter matched any interception's client. Pick one rule and use it in all three places.

🤖

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We have an invariant that one session = one client. I don’t think it’s explicitly enforced, but it shouldn’t break. And if it does, I’m fine with minor issues—the important thing is that we shouldn’t return a 500 or panic.

Comment thread scripts/seed/main.go Outdated
Comment thread coderd/database/queries/aibridge.sql Outdated
Comment thread coderd/database/migrations/000562_aibridge_sessions.up.sql Outdated
@evgeniy-scherbina
evgeniy-scherbina force-pushed the yevhenii/optimize-sessions-query branch 3 times, most recently from 5d8ffa4 to edbdfb1 Compare August 12, 2026 00:26
@evgeniy-scherbina
evgeniy-scherbina force-pushed the yevhenii/optimize-sessions-query branch from edbdfb1 to af7b44a Compare August 12, 2026 13:18
@evgeniy-scherbina

Copy link
Copy Markdown
Contributor Author

Closed to avoid a potentially expensive migration. It takes ~15 seconds on simulated data and should take around 7–10 seconds on Dogfood, but may run longer for very large customers.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant