chore: refresh AI model price book - #28228
Closed
cdrci wants to merge 30 commits into
Closed
Conversation
The price book is regenerated by hand with `make gen/aibridge-prices`, so newly launched models stay unpriced and changed prices stay stale until someone remembers to run it. Add a weekly workflow that regenerates both artifacts from live models.dev data and opens a pull request when the output changes. The refresh is never merged automatically: prices are customer-visible cost numbers, so every change goes through human review. A generator failure, which happens by design when upstream drops a pinned or curated model, is announced in Slack. The pull request body carries a generated summary of models added, models removed, and prices changed, rendered by a new deterministic scripts/aibridgepricesdiff tool. Add CODEOWNERS entries for the price book, the frontend catalog, and both generators so the right reviewer is requested on every refresh.
The pull request diff already shows exact prices for every added, removed, and changed row, so repeating them in the body duplicated what a reviewer can read directly. Keep the counts and the model lists, which answer what moved without restating the diff.
… fields The per-field count did not match the model list below it and needed a qualifier to reconcile. Counting models keeps every figure on the same unit.
… fields The summary lists models per category, so tracking which individual price field moved produced detail nothing rendered. Compare rows as a unit and carry model keys through, which removes the per-field change type, the field accessor table, and the two name-mapping helpers. Every category now holds at most one entry per model, so a single key comparator gives a total order and the stable-sort requirement disappears. Verified byte-identical output against the previous implementation on a live upstream snapshot.
…ir own lines Cases with more than one model packed both rows onto a single line, which made the difference between the old and new snapshot hard to spot when reviewing.
…ixtures Struct literals packed two fields onto a line, and cases mixed an inline old snapshot with a multiline new one, so the two sides did not line up when read side by side. Inline both only when each snapshot holds a single row.
…ror messages Substring assertions left section order, spacing, and stray content unverified: reordering the Removed and Changed sections passed the suite. Compare the whole rendered summary for both the empty and populated cases. Assert on error contents rather than mere failure, and cover a missing input file, which the workflow hits if the snapshot step is ever skipped.
The identity was a plausible-looking address that is not verified on any GitHub account, so refresh commits would show an unlinked author. Use the bot identity that backport.yaml and cherry-pick.yaml already use, whose numeric noreply address associates the commit with a real profile.
The style guide's word-choice table lists AI Bridge as a form to avoid. Identifiers stay as they are, since they name the paths and make targets this workflow drives: coderd/aibridge/prices, make gen/aibridge-prices, and scripts/aibridgepricesgen.
The refresh PR touches exactly these two files, so directory-level entries on the generators never fired on it. The knownModels directory also holds hand-written TypeScript, which would have requested review on unrelated frontend work.
New artifacts follow the aigateway naming; a file named aibridge would be invisible to anyone grepping the current name. The branch and concurrency group move with it, since neither exists yet. Paths, make targets, and the scripts keep their aibridge names, which match what they refer to.
…RICES_SLACK_WEBHOOK Matches the workflow name and the <SCOPE>_SLACK_WEBHOOK convention. The secret does not exist yet, so renaming now costs nothing.
DO NOT MERGE. Drop this commit after verifying the run.
The alert built its text in double quotes, so bash kept \n as two characters and jq --arg escaped the backslash. Slack printed a literal \n mid-sentence instead of breaking the line; build the text with printf.
The step only runs after the refresh has already failed, so exiting 0 hid the more serious problem: nobody was told. Mark the step red and name the secret, now that it exists and an empty value means a misconfiguration.
gh pr list/create/edit go through GraphQL, whose queries resolve reviewer and team fields that require the read:org scope. cdrci's token carries only repo and workflow, so the calls succeed while no pull request exists and fail once CODEOWNERS attaches a reviewer: the first refresh would open a PR and every run after it would fail. Observed on this branch: run 32058047624 created #28224, then 32058781222 and 32059086614 both failed on gh pr list with a read:org scope error. The equivalent REST endpoints need only repo.
cdrci
requested review from
DanielleMaywood and
jdomeracki-coder
as code owners
August 17, 2026 20:20
…acts Spreads review of the weekly refresh beyond one person, so a stale price book is not blocked on a single reviewer's availability.
cdrci
force-pushed
the
bot/aigateway-prices-refresh
branch
from
August 17, 2026 20:27
872f3b5 to
c5394c0
Compare
Monday leaves only one business day before Tuesday releases and competes with release preparation. Thursday leaves Thursday, Friday, and Monday for review without changing the weekly freshness bound.
cdrci
force-pushed
the
bot/aigateway-prices-refresh
branch
from
August 18, 2026 20:27
c5394c0 to
208d6a3
Compare
Complements the value-to-null case and verifies that an existing model whose price becomes available is reported as changed, not added.
cdrci
force-pushed
the
bot/aigateway-prices-refresh
branch
from
August 18, 2026 21:15
208d6a3 to
8031478
Compare
Complements zero-to-null and verifies that an explicit zero is treated as a populated price in both directions.
cdrci
force-pushed
the
bot/aigateway-prices-refresh
branch
from
August 18, 2026 21:20
8031478 to
be3c424
Compare
Large upstream refreshes can add or reprice dozens of models, which buries the review notes under a long body. Keep the counts visible and put each model category in a native GitHub details block so reviewers can expand only the sections they need.
cdrci
force-pushed
the
bot/aigateway-prices-refresh
branch
from
August 19, 2026 13:42
be3c424 to
27bd1f4
Compare
GitHub wraps rendered prose to the viewer width, so hard-wrapping the source at roughly 70 characters only made the raw Markdown and copied text look cramped.
cdrci
force-pushed
the
bot/aigateway-prices-refresh
branch
from
August 19, 2026 13:53
27bd1f4 to
65bb8c3
Compare
When generation is unchanged there is no refresh PR, so a successful no-op and a scheduled workflow that never ran are otherwise indistinguishable. Post a short heartbeat only for the no-change path; a refresh PR remains the success signal when changes exist.
cdrci
force-pushed
the
bot/aigateway-prices-refresh
branch
from
August 19, 2026 14:13
65bb8c3 to
e89de0e
Compare
Select the failure alert or no-change heartbeat in one final step. This removes duplicated webhook validation, JSON encoding, and curl setup, and prevents a failed heartbeat delivery from triggering a second, contradictory failure notification.
cdrci
force-pushed
the
bot/aigateway-prices-refresh
branch
from
August 19, 2026 14:38
e89de0e to
9d7018b
Compare
…ontext JOB_STATUS makes it clear that the value comes directly from job.status and is not a custom refresh result.
cdrci
force-pushed
the
bot/aigateway-prices-refresh
branch
from
August 19, 2026 14:46
9d7018b to
2de1651
Compare
evgeniy-scherbina
added a commit
that referenced
this pull request
Aug 19, 2026
Example of generated PR: #28228 ## Summary The price book is regenerated by hand with `make gen/aibridge-prices`, so keeping it current depends on someone remembering to run it. Newly launched models stay unpriced and changed prices stay stale until the next manual run. This adds a weekly workflow that regenerates both artifacts from live [models.dev](https://models.dev) data and opens a pull request when the output changes. For scale, a run against `main` today produced 32 models added, 11 removed, and 43 repriced. ## What it does - Runs every Thursday at 09:00 UTC, plus `workflow_dispatch`. Exits without opening anything when regeneration produces no diff. - Reuses one branch and one pull request, force-pushing each week, so at most one refresh PR is open and it always carries the newest snapshot. - Ships `prices.json` and `knownModelsGenerated.json` together, since both come from a single upstream snapshot. - Never merges automatically. Prices are customer-visible cost numbers, so every change goes through human review. - Announces failures in Slack. The generator fails by design when upstream drops a model pinned in `overrides.jq` or curated in `curation.json`, and a silently red weekly cron would defeat the point. `scripts/aibridgepricesdiff` renders the PR body summary: counts, then the models added, removed, and repriced. Exact figures stay in the diff rather than being restated in the body. It is deterministic and unit tested rather than AI-generated, so the model lists cannot drift from the file. `CODEOWNERS` covers the two generated artifacts, which are exactly the files a refresh PR touches. ## Verification Ran on this branch via a temporary `push` trigger, since `schedule` and `workflow_dispatch` only fire from the default branch. Confirmed end to end: mise and pnpm setup, generation from live models.dev, change detection, summary and body assembly, commit under the bot identity, branch creation, force-push on the second run, and Slack delivery. `gh pr create` and `gh pr edit` remain unverified: both runs hit `HTTP 503` from `api.github.com/graphql` during a GitHub major outage. The push path itself succeeded, so the credential and permissions are proven. Also verified locally: `go test ./scripts/aibridgepricesdiff/`, `make lint/actions/actionlint`, `make lint/actions/zizmor`, and `make pre-commit`. ## Notes - The PR is opened with `secrets.CDRCI_GITHUB_TOKEN`. PRs opened with the default `GITHUB_TOKEN` do not trigger workflow runs, which would leave the refresh without CI signal on a file that feeds cost calculation. - Requires the `AIGATEWAY_PRICES_SLACK_WEBHOOK` repository secret. The notification step fails loudly if it is unset, since by then the refresh has already failed and nobody would be told. - The workflow installs pnpm dependencies because catalog generation formats its output with biome; without `site/node_modules` the formatting step silently no-ops and an unformatted file would be committed. - A refresh that lands after a release branch is cut can reach that release through the existing cherry-pick label, so no separate release-cut trigger is included. The label is not applied automatically; that stays a reviewer decision. - Identifiers keep the `aibridge` spelling where they name real paths and make targets (`coderd/aibridge/prices`, `make gen/aibridge-prices`, `scripts/aibridgepricesgen`). The workflow itself is named `aigateway-prices-refresh`, and prose says AI Gateway per the style guide. - The issue also asked for code owners on the generator. Only the generated artifacts are owned here, since those are the files a refresh PR touches. Ownership of `curation.json` and `overrides.jq` can be added separately if human edits to them should request review. <details> <summary>Implementation plan and decision log</summary> ### Goal Refresh `make gen/aibridge-prices` output on a schedule and open a human-reviewed PR when it changes, so releases never ship a stale price book. ### Schedule Weekly, Thursdays 09:00 UTC, leaving Thursday, Friday, and Monday for review before Tuesday releases. A monthly run was considered first. Release ships Tuesday morning and the freeze is the last Tuesday before EOM, so a monthly run on the 19th or 20th collapses to 1-2 usable business days in roughly 10 of 132 months: February, when the release Tuesday is the 22nd and the run day falls on a weekend. Measured across 2026-2036: | Cron day | Worst case usable business days | Months with 2 or fewer | Median | |---|---|---|---| | 20th | 1 | 10 | 6 | | 19th | 1 | 10 | 6 | | 16th | 3 | 0 | 6 | | Weekly Thursday | n/a, always a full week | 0 | n/a | The median is 6 days for every candidate, so moving to weekly costs nothing and removes the bad tail. The shipped book is never more than 7 days stale. ### Branch strategy One fixed branch, force-pushed, rather than a new branch and PR per run. `prices.json` is a full regeneration from the current upstream snapshot, not an incremental patch, so two open refresh PRs are the same file at two points in time and the older one is strictly wrong. A single PR makes merging a stale snapshot impossible, keeps reviewer load at one PR, and avoids a full CI run per week per stale PR. ### Summary generation Deterministic Go tool, not an AI summary. An LLM summarizing a JSON diff can drop or invent a row, and would be non-reproducible run to run. A jq implementation was considered and rejected as effectively untestable. The body lists which models moved; the diff in the Files tab remains the source of truth for figures. An earlier version rendered full price tables with percentage deltas. That duplicated the diff, so it was reduced to model lists. ### Token `secrets.CDRCI_GITHUB_TOKEN`, the cdrci machine user already used for bot-authored PRs in `release.yaml` and cited for exactly this reason in the commented-out `update-flake` job in `ci.yaml`. A dedicated GitHub App scoped to this repo would be tighter, but needs org admin to create. ### Commit identity `github-actions[bot]` with its numeric noreply address, matching `backport.yaml` and `cherry-pick.yaml`. An invented address would leave refresh commits with an unlinked author. ### Alerting A dedicated webhook secret rather than reusing `CI_FAILURE_SLACK_WEBHOOK`. The repo already splits alerting per domain (docs, security, dependabot each have their own webhook), and the CI-failure channel carries a user mention plus `vars.BLINK_CI_FAILURE_PROMPT` that feeds an automated triage flow this payload does not belong in. GitHub's built-in email notification was considered and rejected as the primary channel: for scheduled workflows it goes to a single user, the one who last modified the cron syntax, and cannot be routed to a team. ### Out of scope Triggering a refresh when a release branch is cut. The existing cherry-pick label already gets a late refresh into a release, so a separate mechanism is unnecessary. </details> --- Closes [AIGOV-578](https://linear.app/codercom/issue/AIGOV-578/automate-updates-to-the-shipped-ai-model-price-book). Authored by Coder Agents on behalf of @evgeniy-scherbina.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Price book changes
59 models added, 11 models removed, 51 models changed.
Added
Removed
Changed
Review notes
Regenerated by
make gen/aibridge-pricesfrom the live models.dev catalog. Both artifacts come from one snapshot, so they ship together:coderd/aibridge/prices/data/prices.jsonsite/src/pages/AgentsPage/components/ChatModelAdminPanel/knownModels/knownModelsGenerated.jsonThese are customer-visible cost numbers taken from upstream data, so this PR is never merged automatically. The summary above lists what moved; check the diff for exact figures before approving.
Opened automatically by the aigateway-prices-refresh workflow.