Skip to content

chore: refresh AI model price book - #28228

Closed
cdrci wants to merge 30 commits into
mainfrom
bot/aigateway-prices-refresh
Closed

chore: refresh AI model price book#28228
cdrci wants to merge 30 commits into
mainfrom
bot/aigateway-prices-refresh

Conversation

@cdrci

@cdrci cdrci commented Aug 17, 2026

Copy link
Copy Markdown
Collaborator

Price book changes

59 models added, 11 models removed, 51 models changed.

Added
  • azure/claude-mythos-5
  • azure/claude-opus-4-7
  • azure/kimi-k2.7-code
  • bedrock/global.openai.gpt-5.6-luna
  • bedrock/global.openai.gpt-5.6-sol
  • bedrock/global.openai.gpt-5.6-terra
  • copilot/gemini-3.7-flash
  • copilot/grok-4.6
  • copilot/kimi-k3
  • copilot/mai-code-1.1-flash
  • google/gemini-3.7-flash
  • openrouter/bytedance-seed/seed-2-1-turbo
  • openrouter/bytedance-seed/seed-2.0-code
  • openrouter/deepseek/deepseek-v4-pro-0813
  • openrouter/dots-studio/dots-3-note-preview:free
  • openrouter/google/gemini-3.7-flash
  • openrouter/inclusionai/ling-3.0-flash
  • openrouter/liquid/lfm-2.5-2.6b:free
  • openrouter/meta/muse-glimmer-30b
  • openrouter/meta/muse-spark-1.2
  • openrouter/nvidia/nemotron-3.5-lightning
  • openrouter/nvidia/nemotron-3.5-lightning:free
  • openrouter/qwen/qwen3.8-2.4t-a95b
  • openrouter/qwen/qwen3.8-27b
  • openrouter/sakana/sakana-namazu
  • openrouter/upstage/solar-pro4
  • openrouter/x-ai/grok-4.6
  • openrouter/z-ai/glm-5.2:free
  • openrouter/z-ai/glm-5.3
  • vercel/alibaba/qwen3.8-2.4t-a95b
  • vercel/alibaba/qwen3.8-27b
  • vercel/deepseek/deepseek-v4-pro-0813
  • vercel/google/gemini-3.7-flash
  • vercel/inclusionai/ling-3.0-flash
  • vercel/meta/muse-glimmer-30b
  • vercel/meta/muse-spark-1.2
  • vercel/meta/muse-spark-1.2-contributor
  • vercel/nvidia/nemotron-3.5-lightning
  • vercel/openai/gpt-4.1-fast
  • vercel/openai/gpt-4.1-mini-fast
  • vercel/openai/gpt-4.1-nano-fast
  • vercel/openai/gpt-4o-fast
  • vercel/openai/gpt-4o-mini-fast
  • vercel/openai/gpt-5-fast
  • vercel/openai/gpt-5-mini-fast
  • vercel/openai/gpt-5.1-thinking-fast
  • vercel/openai/gpt-5.2-fast
  • vercel/openai/gpt-5.3-codex-fast
  • vercel/openai/gpt-5.4-fast
  • vercel/openai/gpt-5.4-mini-fast
  • vercel/openai/gpt-5.5-fast
  • vercel/openai/gpt-5.6-luna-fast
  • vercel/openai/gpt-5.6-sol-fast
  • vercel/openai/gpt-5.6-terra-fast
  • vercel/openai/o3-fast
  • vercel/openai/o4-mini-fast
  • vercel/sakana/namazu
  • vercel/xai/grok-4.6
  • vercel/zai/glm-5.3
Removed
  • anthropic/claude-opus-4-1
  • anthropic/claude-opus-4-1-20250805
  • google/gemini-2.0-flash
  • google/gemini-2.0-flash-lite
  • google/gemini-3-pro-preview
  • openrouter/inclusionai/ling-3.0-flash:free
  • openrouter/openai/gpt-5.3-chat
  • vercel/anthropic/claude-opus-4.1
  • vercel/inclusionai/ling-3.0-flash-free
  • vercel/openai/gpt-5.1-instant
  • vercel/openai/gpt-5.3-chat
Changed
  • azure/gpt-5.6-luna
  • azure/gpt-5.6-sol
  • azure/gpt-5.6-terra
  • bedrock/openai.gpt-5.6-sol
  • openrouter/deepseek/deepseek-chat-v3-0324
  • openrouter/deepseek/deepseek-v3.1-terminus
  • openrouter/deepseek/deepseek-v4-flash
  • openrouter/deepseek/deepseek-v4-flash-0731
  • openrouter/deepseek/deepseek-v4-pro
  • openrouter/google/gemini-3.6-flash
  • openrouter/google/gemma-4-31b-it
  • openrouter/gryphe/mythomax-l2-13b
  • openrouter/minimax/minimax-m2.5
  • openrouter/minimax/minimax-m2.7
  • openrouter/mistralai/mistral-small-3.2-24b-instruct
  • openrouter/moonshotai/kimi-k2.5
  • openrouter/moonshotai/kimi-k2.6
  • openrouter/moonshotai/kimi-k2.7-code
  • openrouter/openai/gpt-5.6-luna
  • openrouter/openai/gpt-5.6-luna-pro
  • openrouter/openai/gpt-5.6-sol
  • openrouter/openai/gpt-5.6-sol-pro
  • openrouter/openai/gpt-5.6-terra
  • openrouter/openai/gpt-5.6-terra-pro
  • openrouter/openai/gpt-oss-120b
  • openrouter/qwen/qwen-plus-2025-07-28:thinking
  • openrouter/qwen/qwen2.5-vl-72b-instruct
  • openrouter/qwen/qwen3-14b
  • openrouter/qwen/qwen3-235b-a22b-2507
  • openrouter/qwen/qwen3-30b-a3b
  • openrouter/qwen/qwen3-coder-30b-a3b-instruct
  • openrouter/qwen/qwen3-vl-235b-a22b-thinking
  • openrouter/qwen/qwen3-vl-30b-a3b-instruct
  • openrouter/qwen/qwen3.5-35b-a3b
  • openrouter/qwen/qwen3.6-27b
  • openrouter/qwen/qwen3.6-35b-a3b
  • openrouter/tencent/hy3-preview
  • openrouter/thinkingmachines/inkling
  • openrouter/thinkingmachines/inkling-small
  • openrouter/z-ai/glm-5
  • openrouter/z-ai/glm-5.2
  • openrouter/~deepseek/deepseek-v4-flash-latest
  • openrouter/~google/gemini-flash-latest
  • openrouter/~moonshotai/kimi-latest
  • openrouter/~openai/gpt-latest
  • openrouter/~x-ai/grok-latest
  • vercel/deepseek/deepseek-v4-flash
  • vercel/deepseek/deepseek-v4-pro
  • vercel/google/gemini-3.6-flash
  • vercel/openai/gpt-5.6-sol
  • vercel/zai/glm-5.2

Review notes

Regenerated by make gen/aibridge-prices from the live models.dev catalog. Both artifacts come from one snapshot, so they ship together:

  • coderd/aibridge/prices/data/prices.json
  • site/src/pages/AgentsPage/components/ChatModelAdminPanel/knownModels/knownModelsGenerated.json

These are customer-visible cost numbers taken from upstream data, so this PR is never merged automatically. The summary above lists what moved; check the diff for exact figures before approving.

Opened automatically by the aigateway-prices-refresh workflow.

The price book is regenerated by hand with `make gen/aibridge-prices`, so
newly launched models stay unpriced and changed prices stay stale until
someone remembers to run it.

Add a weekly workflow that regenerates both artifacts from live models.dev
data and opens a pull request when the output changes. The refresh is never
merged automatically: prices are customer-visible cost numbers, so every
change goes through human review. A generator failure, which happens by
design when upstream drops a pinned or curated model, is announced in Slack.

The pull request body carries a generated summary of models added, models
removed, and prices changed, rendered by a new deterministic
scripts/aibridgepricesdiff tool.

Add CODEOWNERS entries for the price book, the frontend catalog, and both
generators so the right reviewer is requested on every refresh.
The pull request diff already shows exact prices for every added, removed,
and changed row, so repeating them in the body duplicated what a reviewer
can read directly. Keep the counts and the model lists, which answer what
moved without restating the diff.
… fields

The per-field count did not match the model list below it and needed a
qualifier to reconcile. Counting models keeps every figure on the same
unit.
… fields

The summary lists models per category, so tracking which individual price
field moved produced detail nothing rendered. Compare rows as a unit and
carry model keys through, which removes the per-field change type, the
field accessor table, and the two name-mapping helpers.

Every category now holds at most one entry per model, so a single key
comparator gives a total order and the stable-sort requirement disappears.

Verified byte-identical output against the previous implementation on a
live upstream snapshot.
…ir own lines

Cases with more than one model packed both rows onto a single line, which
made the difference between the old and new snapshot hard to spot when
reviewing.
…ixtures

Struct literals packed two fields onto a line, and cases mixed an inline
old snapshot with a multiline new one, so the two sides did not line up
when read side by side. Inline both only when each snapshot holds a single
row.
…ror messages

Substring assertions left section order, spacing, and stray content
unverified: reordering the Removed and Changed sections passed the suite.
Compare the whole rendered summary for both the empty and populated cases.

Assert on error contents rather than mere failure, and cover a missing
input file, which the workflow hits if the snapshot step is ever skipped.
The identity was a plausible-looking address that is not verified on any
GitHub account, so refresh commits would show an unlinked author. Use the
bot identity that backport.yaml and cherry-pick.yaml already use, whose
numeric noreply address associates the commit with a real profile.
The style guide's word-choice table lists AI Bridge as a form to avoid.
Identifiers stay as they are, since they name the paths and make targets
this workflow drives: coderd/aibridge/prices, make gen/aibridge-prices,
and scripts/aibridgepricesgen.
The refresh PR touches exactly these two files, so directory-level entries
on the generators never fired on it. The knownModels directory also holds
hand-written TypeScript, which would have requested review on unrelated
frontend work.
New artifacts follow the aigateway naming; a file named aibridge would be
invisible to anyone grepping the current name. The branch and concurrency
group move with it, since neither exists yet. Paths, make targets, and the
scripts keep their aibridge names, which match what they refer to.
…RICES_SLACK_WEBHOOK

Matches the workflow name and the <SCOPE>_SLACK_WEBHOOK convention. The
secret does not exist yet, so renaming now costs nothing.
DO NOT MERGE. Drop this commit after verifying the run.
The alert built its text in double quotes, so bash kept \n as two
characters and jq --arg escaped the backslash. Slack printed a literal \n
mid-sentence instead of breaking the line; build the text with printf.
The step only runs after the refresh has already failed, so exiting 0 hid
the more serious problem: nobody was told. Mark the step red and name the
secret, now that it exists and an empty value means a misconfiguration.
gh pr list/create/edit go through GraphQL, whose queries resolve reviewer
and team fields that require the read:org scope. cdrci's token carries only
repo and workflow, so the calls succeed while no pull request exists and
fail once CODEOWNERS attaches a reviewer: the first refresh would open a PR
and every run after it would fail.

Observed on this branch: run 32058047624 created #28224, then 32058781222
and 32059086614 both failed on gh pr list with a read:org scope error. The
equivalent REST endpoints need only repo.
…acts

Spreads review of the weekly refresh beyond one person, so a stale price
book is not blocked on a single reviewer's availability.
Monday leaves only one business day before Tuesday releases and competes
with release preparation. Thursday leaves Thursday, Friday, and Monday for
review without changing the weekly freshness bound.
@cdrci
cdrci force-pushed the bot/aigateway-prices-refresh branch from c5394c0 to 208d6a3 Compare August 18, 2026 20:27
Complements the value-to-null case and verifies that an existing model whose
price becomes available is reported as changed, not added.
@cdrci
cdrci force-pushed the bot/aigateway-prices-refresh branch from 208d6a3 to 8031478 Compare August 18, 2026 21:15
Complements zero-to-null and verifies that an explicit zero is treated as a
populated price in both directions.
@cdrci
cdrci force-pushed the bot/aigateway-prices-refresh branch from 8031478 to be3c424 Compare August 18, 2026 21:20
Large upstream refreshes can add or reprice dozens of models, which buries
the review notes under a long body. Keep the counts visible and put each
model category in a native GitHub details block so reviewers can expand only
the sections they need.
@cdrci
cdrci force-pushed the bot/aigateway-prices-refresh branch from be3c424 to 27bd1f4 Compare August 19, 2026 13:42
GitHub wraps rendered prose to the viewer width, so hard-wrapping the source
at roughly 70 characters only made the raw Markdown and copied text look
cramped.
@cdrci
cdrci force-pushed the bot/aigateway-prices-refresh branch from 27bd1f4 to 65bb8c3 Compare August 19, 2026 13:53
When generation is unchanged there is no refresh PR, so a successful no-op
and a scheduled workflow that never ran are otherwise indistinguishable.
Post a short heartbeat only for the no-change path; a refresh PR remains the
success signal when changes exist.
@cdrci
cdrci force-pushed the bot/aigateway-prices-refresh branch from 65bb8c3 to e89de0e Compare August 19, 2026 14:13
Select the failure alert or no-change heartbeat in one final step. This
removes duplicated webhook validation, JSON encoding, and curl setup, and
prevents a failed heartbeat delivery from triggering a second, contradictory
failure notification.
@cdrci
cdrci force-pushed the bot/aigateway-prices-refresh branch from e89de0e to 9d7018b Compare August 19, 2026 14:38
evgeniy-scherbina and others added 2 commits August 19, 2026 14:44
…ontext

JOB_STATUS makes it clear that the value comes directly from job.status and
is not a custom refresh result.
@cdrci
cdrci force-pushed the bot/aigateway-prices-refresh branch from 9d7018b to 2de1651 Compare August 19, 2026 14:46
evgeniy-scherbina added a commit that referenced this pull request Aug 19, 2026
Example of generated PR: #28228

## Summary

The price book is regenerated by hand with `make gen/aibridge-prices`,
so keeping it current depends on someone remembering to run it. Newly
launched models stay unpriced and changed prices stay stale until the
next manual run.

This adds a weekly workflow that regenerates both artifacts from live
[models.dev](https://models.dev) data and opens a pull request when the
output changes.

For scale, a run against `main` today produced 32 models added, 11
removed, and 43 repriced.

## What it does

- Runs every Thursday at 09:00 UTC, plus `workflow_dispatch`. Exits
without opening anything when regeneration produces no diff.
- Reuses one branch and one pull request, force-pushing each week, so at
most one refresh PR is open and it always carries the newest snapshot.
- Ships `prices.json` and `knownModelsGenerated.json` together, since
both come from a single upstream snapshot.
- Never merges automatically. Prices are customer-visible cost numbers,
so every change goes through human review.
- Announces failures in Slack. The generator fails by design when
upstream drops a model pinned in `overrides.jq` or curated in
`curation.json`, and a silently red weekly cron would defeat the point.

`scripts/aibridgepricesdiff` renders the PR body summary: counts, then
the models added, removed, and repriced. Exact figures stay in the diff
rather than being restated in the body. It is deterministic and unit
tested rather than AI-generated, so the model lists cannot drift from
the file.

`CODEOWNERS` covers the two generated artifacts, which are exactly the
files a refresh PR touches.

## Verification

Ran on this branch via a temporary `push` trigger, since `schedule` and
`workflow_dispatch` only fire from the default branch. Confirmed end to
end: mise and pnpm setup, generation from live models.dev, change
detection, summary and body assembly, commit under the bot identity,
branch creation, force-push on the second run, and Slack delivery.

`gh pr create` and `gh pr edit` remain unverified: both runs hit `HTTP
503` from `api.github.com/graphql` during a GitHub major outage. The
push path itself succeeded, so the credential and permissions are
proven.

Also verified locally: `go test ./scripts/aibridgepricesdiff/`, `make
lint/actions/actionlint`, `make lint/actions/zizmor`, and `make
pre-commit`.

## Notes

- The PR is opened with `secrets.CDRCI_GITHUB_TOKEN`. PRs opened with
the default `GITHUB_TOKEN` do not trigger workflow runs, which would
leave the refresh without CI signal on a file that feeds cost
calculation.
- Requires the `AIGATEWAY_PRICES_SLACK_WEBHOOK` repository secret. The
notification step fails loudly if it is unset, since by then the refresh
has already failed and nobody would be told.
- The workflow installs pnpm dependencies because catalog generation
formats its output with biome; without `site/node_modules` the
formatting step silently no-ops and an unformatted file would be
committed.
- A refresh that lands after a release branch is cut can reach that
release through the existing cherry-pick label, so no separate
release-cut trigger is included. The label is not applied automatically;
that stays a reviewer decision.
- Identifiers keep the `aibridge` spelling where they name real paths
and make targets (`coderd/aibridge/prices`, `make gen/aibridge-prices`,
`scripts/aibridgepricesgen`). The workflow itself is named
`aigateway-prices-refresh`, and prose says AI Gateway per the style
guide.
- The issue also asked for code owners on the generator. Only the
generated artifacts are owned here, since those are the files a refresh
PR touches. Ownership of `curation.json` and `overrides.jq` can be added
separately if human edits to them should request review.

<details>
<summary>Implementation plan and decision log</summary>

### Goal

Refresh `make gen/aibridge-prices` output on a schedule and open a
human-reviewed PR when it changes, so releases never ship a stale price
book.

### Schedule

Weekly, Thursdays 09:00 UTC, leaving Thursday, Friday, and Monday for
review before Tuesday releases. A monthly run was considered first.
Release ships Tuesday morning and the freeze is the last Tuesday before
EOM, so a monthly run on the 19th or 20th collapses to 1-2 usable
business days in roughly 10 of 132 months: February, when the release
Tuesday is the 22nd and the run day falls on a weekend. Measured across
2026-2036:

| Cron day | Worst case usable business days | Months with 2 or fewer |
Median |
|---|---|---|---|
| 20th | 1 | 10 | 6 |
| 19th | 1 | 10 | 6 |
| 16th | 3 | 0 | 6 |
| Weekly Thursday | n/a, always a full week | 0 | n/a |

The median is 6 days for every candidate, so moving to weekly costs
nothing and removes the bad tail. The shipped book is never more than 7
days stale.

### Branch strategy

One fixed branch, force-pushed, rather than a new branch and PR per run.
`prices.json` is a full regeneration from the current upstream snapshot,
not an incremental patch, so two open refresh PRs are the same file at
two points in time and the older one is strictly wrong. A single PR
makes merging a stale snapshot impossible, keeps reviewer load at one
PR, and avoids a full CI run per week per stale PR.

### Summary generation

Deterministic Go tool, not an AI summary. An LLM summarizing a JSON diff
can drop or invent a row, and would be non-reproducible run to run. A jq
implementation was considered and rejected as effectively untestable.
The body lists which models moved; the diff in the Files tab remains the
source of truth for figures.

An earlier version rendered full price tables with percentage deltas.
That duplicated the diff, so it was reduced to model lists.

### Token

`secrets.CDRCI_GITHUB_TOKEN`, the cdrci machine user already used for
bot-authored PRs in `release.yaml` and cited for exactly this reason in
the commented-out `update-flake` job in `ci.yaml`. A dedicated GitHub
App scoped to this repo would be tighter, but needs org admin to create.

### Commit identity

`github-actions[bot]` with its numeric noreply address, matching
`backport.yaml` and `cherry-pick.yaml`. An invented address would leave
refresh commits with an unlinked author.

### Alerting

A dedicated webhook secret rather than reusing
`CI_FAILURE_SLACK_WEBHOOK`. The repo already splits alerting per domain
(docs, security, dependabot each have their own webhook), and the
CI-failure channel carries a user mention plus
`vars.BLINK_CI_FAILURE_PROMPT` that feeds an automated triage flow this
payload does not belong in.

GitHub's built-in email notification was considered and rejected as the
primary channel: for scheduled workflows it goes to a single user, the
one who last modified the cron syntax, and cannot be routed to a team.

### Out of scope

Triggering a refresh when a release branch is cut. The existing
cherry-pick label already gets a late refresh into a release, so a
separate mechanism is unnecessary.

</details>

---

Closes
[AIGOV-578](https://linear.app/codercom/issue/AIGOV-578/automate-updates-to-the-shipped-ai-model-price-book).

Authored by Coder Agents on behalf of @evgeniy-scherbina.
@github-actions github-actions Bot locked and limited conversation to collaborators Aug 19, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants