fix(pricing): take models.dev rates from the authoring catalog - #1541
Conversation
Deploying with
|
| Status | Name | Latest Commit | Preview URL | Updated (UTC) |
|---|---|---|---|---|
| ✅ Deployment successful! View logs |
ccusage-guide | 99c5a4a | Commit Preview URL Branch Preview URL |
Aug 15 2026, 10:38 AM |
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughThe models.dev generator now applies catalog-derived trust, modality, pricing, and exact-match rules. It emits catalog metadata and expanded pricing snapshots. The Rust runtime embeds compressed snapshots and applies deterministic catalog selection and lookup behavior. ChangesModels.dev pricing selection and runtime integration
Estimated code review effort: 5 (Critical) | ~100 minutes Sequence Diagram(s)sequenceDiagram
participant ModelsDev
participant Generator
participant Snapshots
participant RustBuild
participant PricingRuntime
ModelsDev->>Generator: catalog models, modalities, and pricing
Generator->>Snapshots: pricing and catalog-rules JSON
Snapshots->>RustBuild: committed snapshots
RustBuild->>PricingRuntime: compressed embedded data
PricingRuntime->>PricingRuntime: rank catalogs and enforce lookup rules
Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
@coderabbitai @cubic-dev-ai please review. Two areas worth extra attention:
|
@ryoppippi I have started the AI code review. It will take a few minutes to complete. |
|
Tip For best results, initiate chat on the files or code changes. 🧩 Analysis chain🏁 Script executed: #!/bin/bash
set -euo pipefail
echo '== PR changed files =='
git diff --name-status HEAD^ HEAD
echo
echo '== Generator outline =='
ast-grep outline nix/tools/models-dev-gen/compact.ts --items all 2>/dev/null || true
echo
echo '== Focused generator diff =='
git diff --unified=80 HEAD^ HEAD -- nix/tools/models-dev-gen/compact.ts
echo
echo '== Candidate gate / tiering / lookup references =='
rg -n -C 5 \
-e 'isEmbeddableModelsDevCandidate\s*\(' \
-e 'authoring|authored|trust|tier|zhipuai|zai' \
-e 'models\.dev|modelsDev' \
--glob '!nix/tools/models-dev-gen/compact.ts' \
.
echo
echo '== Likely adapter/model-definition files =='
fd -t f . | rg -i '(adapter|provider|model|pricing|cost|catalog)' | head -250Length of output: 174213 🏁 Script executed: #!/bin/bash
set -euo pipefail
echo '== compact.ts =='
cat -n nix/tools/models-dev-gen/compact.ts
echo
echo '== Generator candidate selection =='
sed -n '55,135p' nix/tools/models-dev-gen/gen.ts | cat -n
echo
echo '== Runtime PR diff =='
git diff --unified=100 HEAD^ HEAD -- rust/crates/ccusage-core/src/pricing.rs
echo
echo '== Pricing lookup and aliases =='
ast-grep outline rust/crates/ccusage-core/src/pricing.rs --items all 2>/dev/null || true
sed -n '360,535p' rust/crates/ccusage-core/src/pricing.rs | cat -n
sed -n '1,260p' rust/crates/ccusage-core/src/model_aliases.rs | cat -n
echo
echo '== Adapter model extraction / model-key construction =='
rg -n -C 4 --glob 'rust/adapters/**/src/*.rs' \
-e '\bmodel\b.*(?:String|str|Cow|CompactString)' \
-e '(?:model|provider).*(?:id|name)' \
-e 'format!\(.*(?:model|provider)' \
-e 'join\(.*(?:model|provider)' \
-e 'PricingMap|find(?:_entry|_exact)?\(' \
rust/adapters
echo
echo '== Snapshot keys with reseller-style namespaces =='
python3 - <<'PY'
import json
p = 'rust/crates/ccusage-core/src/models-dev-pricing.json'
with open(p) as f:
keys = list(json.load(f))
for k in keys:
if '/' in k or '.' in k:
print(k)
PYLength of output: 161175 🏁 Script executed: #!/bin/bash
set -euo pipefail
echo '== compact.ts, focused =='
sed -n '1,270p' nix/tools/models-dev-gen/compact.ts | cat -n
echo
echo '== Compact tests covering trust and gate =='
sed -n '1,225p' nix/tools/models-dev-gen/compact.test.ts | cat -n
echo
echo '== Exact pricing lookup fallback =='
rg -n -A85 -B10 'fn find_entry_or_alias|fn find_exact|fn find\(' rust/crates/ccusage-core/src/pricing.rs
echo
echo '== OpenCode and Droid candidate construction =='
sed -n '195,250p' rust/adapters/opencode/src/parser.rs | cat -n
sed -n '175,207p' rust/adapters/droid/src/parser.rs | cat -n
echo
echo '== Provider-qualified snapshot keys changed by this PR =='
git diff --unified=0 HEAD^ HEAD -- rust/crates/ccusage-core/src/models-dev-pricing.json \
| rg '^[+-]"[^"]+' || true
echo
echo '== Embedded snapshot keys likely queried by adapter provider prefixes =='
python3 - <<'PY'
import json
from collections import Counter
with open('rust/crates/ccusage-core/src/models-dev-pricing.json') as f:
keys = json.load(f).keys()
prefixes = ('anthropic/', 'openrouter/', 'google/', 'xai/', 'openai/', 'github_copilot/')
for k in keys:
if k.startswith(prefixes):
print(k)
PYLength of output: 29672 🏁 Script executed: #!/bin/bash
set -euo pipefail
echo '== Branch and merge-base =='
git status --short --branch
git remote -v
git merge-base HEAD origin/main 2>/dev/null || true
base="$(git merge-base HEAD origin/main 2>/dev/null || git rev-list --max-parents=0 HEAD | tail -1)"
echo "BASE=$base"
git diff --stat "$base"...HEAD
git diff --name-status "$base"...HEAD
echo
echo '== Complete PR changes to generator and snapshots =='
git diff --unified=35 "$base"...HEAD -- \
nix/tools/models-dev-gen/compact.ts \
nix/tools/models-dev-gen/gen.ts \
nix/tools/models-dev-gen/compact.test.ts \
rust/crates/ccusage-core/src/models-dev-pricing.json \
rust/crates/ccusage-core/src/models-dev-provider-trust.json
echo
echo '== Available models.dev source paths =='
fd -H -t d -d 5 '^models$|models\.dev$' . 2>/dev/null || true
fd -H -t f -d 8 -e toml . 2>/dev/null | rg '/models/' | head -80 || true
echo
echo '== Static catalog-id usage and types =='
rg -n -C 4 'catalogId|catalog_id|catalog' nix/tools/models-dev-gen
echo
echo '== Tests and code that call isEmbeddableModelsDevCandidate =='
rg -n -C 5 'isEmbeddableModelsDevCandidate\s*\(' nix/tools/models-dev-genLength of output: 152626
I did not run the test suite in this environment. |
![pullfrog[bot]](https://avatars.githubusercontent.com/in/1768019?s=60&v=4){kind=small}
There was a problem hiding this comment.
✅ No new issues found.
Reviewed changes — replaces the hardcoded model-name regex and provider-id priority list for models.dev pricing selection with a catalog-index-based trust tier system, and brings the same ranking into the Rust runtime loader.
- Trust tier selection in the generator — the
/claude|anthropic|kimi|moonshot/iregex and hardcodedcandidateProviderPriority()are replaced bymodelsDevProviderTrust(), which derives authorship from the canonical catalog directory scan.isEmbeddableModelsDevCandidate()restricts reseller entries to models the authored catalog still knows, andisPriceableModelsDevCost()plusisTextOutputModel()reject flat-fee catalogs and non-text-output models. - New committed artifact —
models-dev-provider-trust.jsoncarries the sorted owner and platform provider-id lists so the Rust runtime can rank liveapi.jsonresponses without needing the catalog itself. - Runtime catalog ranking —
load_models_dev_json_missingsorts providers by trust rank before loading, so authoring catalogs always load first regardless of hash iteration order. The runtime zero-cost guard mirrors the generator's. - Build-time minification — the committed indented
models-dev-pricing.jsonis minified viaserde_json::to_string()at build time, keeping the tree copy reviewable without shipping indentation in the binary. - Wiring and tests — the new artifact is plumbed through
default.nix,justfile,cargo-artifacts.nix,package.nix, andupdate-models-dev-lock.nu. 14 new TypeScript tests cover the trust tiers, replace/embed decisions, and filtering; 4 new Rust tests covergrok-build-0.1offline pricing, author-price regression forkimi-k2.7-code, live catalog ranking, and live flat-fee rejection.
@v0 or keep the SHA fresh with Dependabot ｜ View workflow run ｜ Using DeepSeek Pro (free via Pullfrog for OSS) ｜ 𝕏
ccusage
@ccusage/ccusage-darwin-arm64
@ccusage/ccusage-darwin-x64
@ccusage/ccusage-linux-arm64
@ccusage/ccusage-linux-x64
@ccusage/ccusage-win32-x64
commit: |
ccusage performance comparisonPR SHA: This compares the PR package against the configured base package on the same CI runner. Package runtime diagnosticsCompares the PR package wrapper, the installed native optional dependency binary, and the workspace release binary on the same large fixture. This identifies whether slow package results come from JavaScript wrapper overhead, the published native binary build, or the Rust core itself. Fixtures: Claude
Committed fixture performanceCommitted small fixtures for stable PR-to-PR feedback and explicit Claude/Codex command coverage. Fixtures: Claude
Large real-world-shaped fixture performanceGenerated fixtures shaped from aggregate local log statistics: thousands of JSONL files, many small sessions, and a long tail of larger sessions. No real prompts, paths, or outputs are stored in the fixtures. Fixtures: Claude
Artifact size
Lower medians and smaller artifacts are better. CI runner noise still applies; use same-run ratios as directional PR feedback, not release guarantees. |
ccusage performance comparisonPR SHA: This compares the Rust PR release binary against the configured base package on the same CI runner. Package runtime diagnosticsCompares the PR package wrapper, the installed native optional dependency binary, and the workspace release binary on the same large fixture. This identifies whether slow package results come from JavaScript wrapper overhead, the published native binary build, or the Rust core itself. Fixtures: Claude
Committed fixture performanceCommitted small fixtures for stable PR-to-PR feedback and explicit Claude/Codex command coverage. Fixtures: Claude
Large real-world-shaped fixture performanceGenerated fixtures shaped from aggregate local log statistics: thousands of JSONL files, many small sessions, and a long tail of larger sessions. No real prompts, paths, or outputs are stored in the fixtures. Fixtures: Claude
Artifact size
Lower medians and smaller artifacts are better. CI runner noise still applies; use same-run ratios as directional PR feedback, not release guarantees. |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@rust/crates/ccusage-core/src/models-dev-pricing.json`:
- Around line 4045-4052: Update the pricing snapshot generator’s eligibility
predicate to exclude asset-priced models such as whisper-large-v3 from token
pricing, then regenerate models-dev-pricing.json without that entry. Add a
regression test covering exclusion of asset-priced models while preserving
token-priced model inclusion.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: fc321687-1f22-40e0-b1db-eb54a0af7975
📒 Files selected for processing (13)
.github/scripts/update-models-dev-lock.nujustfilenix/cargo-artifacts.nixnix/tools/models-dev-gen/compact.test.tsnix/tools/models-dev-gen/compact.tsnix/tools/models-dev-gen/default.nixnix/tools/models-dev-gen/gen.tspackage.nixrust/crates/ccusage-core/README.mdrust/crates/ccusage-core/build.rsrust/crates/ccusage-core/src/models-dev-pricing.jsonrust/crates/ccusage-core/src/models-dev-provider-trust.jsonrust/crates/ccusage-core/src/pricing.rs
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
There was a problem hiding this comment.
All reported issues were addressed across 13 files
Not reviewed (too large): rust/crates/ccusage-core/src/models-dev-pricing.json (~3,844 lines) - if these are generated or fixture files, add them to ignored paths to exclude them from future reviews.
Tip: cubic can generate docs of your entire codebase and keep them up to date. Try it here.
Re-trigger cubic
|
@coderabbitai review Pushed 5995ba1, which fixes the asset-priced pricing CodeRabbit found and a second instance of the same root cause ( |
@ryoppippi I have started the AI code review. It will take a few minutes to complete. |
🧩 Analysis chain🏁 Script executed: #!/bin/bash
set -euo pipefail
echo '== Commit and changed files =='
git show --no-ext-diff --format=fuller --name-status 5995ba1
echo
echo '== Focused diff =='
git diff --no-ext-diff --unified=80 5995ba1^ 5995ba1 -- \
nix/tools/models-dev-gen/compact.ts \
nix/tools/models-dev-gen/gen.ts \
nix/tools/models-dev-gen/compact.test.ts \
rust/crates/ccusage-core/src/models-dev-pricing.json \
rust/crates/ccusage-core/src/models-dev-provider-trust.json
echo
echo '== Generator symbol map =='
ast-grep outline nix/tools/models-dev-gen/compact.ts --items all
ast-grep outline nix/tools/models-dev-gen/gen.ts --items all
ast-grep outline nix/tools/models-dev-gen/compact.test.ts --items all
echo
echo '== Eligibility, modality, and catalog-index references =='
rg -n -C 8 \
-e '\bisTextOutputModel\s*\(' \
-e '\bisEmbeddableModelsDevCandidate\s*\(' \
-e '\bbuildModelsDevCatalogIndex\s*\(' \
-e '\bmodalities\b' \
-e 'authoredModel' \
nix/tools/models-dev-gen
echo
echo '== Snapshot entries mentioned by user =='
python3 - <<'PY'
import json
path = 'rust/crates/ccusage-core/src/models-dev-pricing.json'
with open(path, encoding='utf-8') as f:
pricing = json.load(f)
for needle in ('gemini-2.5-flash-image', 'gemini-3-pro-image-preview'):
matches = {k: v for k, v in pricing.items() if needle in k}
print(f'{needle}: {len(matches)} matches')
for key, value in matches.items():
print(key, json.dumps(value, sort_keys=True))
PYLength of output: 69629 🏁 Script executed: #!/bin/bash
set -euo pipefail
echo '== Locate authored model catalog sources =='
fd -H -t f -e toml . | rg '(^|/)models/[^/]+/[^/]+\.toml$' | head -20
echo
echo '== Bare authored-ID collisions (if catalog sources are present) =='
python3 - <<'PY'
from collections import defaultdict
from pathlib import Path
paths = [p for p in Path('.').rglob('*.toml') if len(p.parts) >= 3 and p.parts[-3] == 'models']
by_id = defaultdict(list)
for path in paths:
by_id[path.stem].append(path.as_posix())
collisions = {model_id: sources for model_id, sources in by_id.items() if len(sources) > 1}
print(f'catalog TOMLs: {len(paths)}')
print(f'bare-ID collisions: {len(collisions)}')
for model_id, sources in sorted(collisions.items())[:100]:
print(model_id)
for source in sources:
print(f' {source}')
PY
echo
echo '== Rust snapshot-regression test and pricing-test map =='
ast-grep outline rust/crates/ccusage-core/src/pricing.rs --items all
rg -n -C 12 \
-e 'gemini-2\.5-flash-image' \
-e 'gemini-3-pro-image-preview' \
-e 'whisper-large-v3' \
-e 'qwen3-asr-flash' \
rust/crates/ccusage-core/src/pricing.rs
echo
echo '== Generator source provenance and catalog model-key assumptions =='
rg -n -C 5 \
-e 'generateCatalog' \
-e 'models:' \
-e 'Object\.entries\(models\)' \
-e 'authoredModalities' \
nix/tools/models-dev-genLength of output: 1706 🏁 Script executed: #!/bin/bash
set -u
echo '== Available catalog-related paths (non-failing) =='
fd -H -t d -d 8 . . 2>/dev/null | rg -i 'models.?dev|catalog|models$' || true
fd -H -t f -d 10 . . 2>/dev/null | rg -i '(models.?dev|catalog).*\.(toml|json|nix|ts)$' || true
echo
echo '== Rust snapshot-regression tests =='
ast-grep outline rust/crates/ccusage-core/src/pricing.rs --items all || true
rg -n -C 14 \
-e 'gemini-2\.5-flash-image' \
-e 'gemini-3-pro-image-preview' \
-e 'whisper-large-v3' \
-e 'qwen3-asr-flash' \
rust/crates/ccusage-core/src/pricing.rs || true
echo
echo '== Nix input / source provenance for models.dev =='
rg -n -C 5 -i \
-e 'models-dev' \
-e 'models\.dev' \
-e 'modelsDev' \
flake.nix flake.lock nix package.nix justfile .github 2>/dev/null || true
echo
echo '== Call sites and authored modality-map implementation =='
rg -n -C 8 \
-e '\bauthoredModalities\b' \
-e '\bisTokenPricedModel\s*\(' \
-e '\bbuildModelsDevCatalogIndex\s*\(' \
nix/tools/models-dev-gen || true
echo
echo '== Commit-specific Rust test diff =='
git diff --no-ext-diff --unified=100 5995ba1^ 5995ba1 -- rust/crates/ccusage-core/src/pricing.rsLength of output: 88327 🏁 Script executed: #!/bin/bash
set -euo pipefail
repo='anomalyco/models.dev'
rev='52d5045ee7c89127a3f8966b48599b9dec8afed6'
tree_file="$(mktemp)"
gh api "repos/${repo}/git/trees/${rev}?recursive=1" > "$tree_file"
python3 - "$tree_file" <<'PY'
import json
import sys
from collections import defaultdict
from pathlib import PurePosixPath
with open(sys.argv[1], encoding='utf-8') as f:
tree = json.load(f)
print(f"tree_truncated: {tree.get('truncated')}")
by_id = defaultdict(list)
for entry in tree.get('tree', []):
path = entry.get('path', '')
parts = PurePosixPath(path).parts
if len(parts) == 3 and parts[0] == 'models' and path.endswith('.toml'):
by_id[PurePosixPath(path).stem].append(path)
collisions = {model_id: paths for model_id, paths in by_id.items() if len(paths) > 1}
print(f"model_tomls: {sum(map(len, by_id.values()))}")
print(f"bare_id_collisions: {len(collisions)}")
for model_id in sorted(collisions):
print(f"{model_id}\t" + "\t".join(sorted(collisions[model_id])))
PY
rm -f "$tree_file"Length of output: 1221
The new eligibility rule correctly takes modalities from the authored catalog before considering reseller metadata, so a reseller’s text-only description can no longer admit an asset-priced model. The generated snapshot excludes the reported Gemini image models and I also checked the exact pinned models.dev revision: its 279 authored model TOMLs have no duplicate bare model IDs, so the bare-ID I did not run the test suite in this environment. ✅ Action performedReview finished.
|
There was a problem hiding this comment.
Reviewed changes — the isTextOutputModel eligibility check is replaced by isTokenPricedModel, which reads modalities from the authored catalog to correctly reject duration-priced models (whisper-large-v3) and image-output models (gemini-2.5-flash-image, gemini-3-pro-image-preview) regardless of how reseller catalogs describe them.
isTokenPricedModelconsults the authored catalog —ModelsDevCatalogIndexgainsauthoredModalities,buildModelsDevCatalogIndexextracts modalities from the canonical models, andisTokenPricedModeluses authored modalities as the authoritative signal. The serving catalog's modalities are a fallback for models the authored catalog omits.- Asset-priced models are removed from the snapshot —
whisper-large-v3(audio input, billed per second),gemini-2.5-flash-image(image output, billed per image), andgemini-3-pro-image-previewno longer appear in the embedded JSON. - New tests fence the filtering — six new TypeScript tests cover text-token models with audio/video inputs, duration-priced rejection, image-output rejection via authored modalities, and fallback behavior. One new Rust test asserts the three asset-priced models are absent while text models with audio/video input survive.
✅ No new issues found.
@v0 or keep the SHA fresh with Dependabot ｜ View workflow run ｜ Using DeepSeek Pro (free via Pullfrog for OSS) ｜ 𝕏
ccusage performance comparisonPR SHA: This compares the PR package against the configured base package on the same CI runner. Package runtime diagnosticsCompares the PR package wrapper, the installed native optional dependency binary, and the workspace release binary on the same large fixture. This identifies whether slow package results come from JavaScript wrapper overhead, the published native binary build, or the Rust core itself. Fixtures: Claude
Committed fixture performanceCommitted small fixtures for stable PR-to-PR feedback and explicit Claude/Codex command coverage. Fixtures: Claude
Large real-world-shaped fixture performanceGenerated fixtures shaped from aggregate local log statistics: thousands of JSONL files, many small sessions, and a long tail of larger sessions. No real prompts, paths, or outputs are stored in the fixtures. Fixtures: Claude
Artifact size
Lower medians and smaller artifacts are better. CI runner noise still applies; use same-run ratios as directional PR feedback, not release guarantees. |
ccusage performance comparisonPR SHA: This compares the Rust PR release binary against the configured base package on the same CI runner. Package runtime diagnosticsCompares the PR package wrapper, the installed native optional dependency binary, and the workspace release binary on the same large fixture. This identifies whether slow package results come from JavaScript wrapper overhead, the published native binary build, or the Rust core itself. Fixtures: Claude
Committed fixture performanceCommitted small fixtures for stable PR-to-PR feedback and explicit Claude/Codex command coverage. Fixtures: Claude
Large real-world-shaped fixture performanceGenerated fixtures shaped from aggregate local log statistics: thousands of JSONL files, many small sessions, and a long tail of larger sessions. No real prompts, paths, or outputs are stored in the fixtures. Fixtures: Claude
Artifact size
Lower medians and smaller artifacts are better. CI runner noise still applies; use same-run ratios as directional PR feedback, not release guarantees. |
There was a problem hiding this comment.
All reported issues were addressed across 13 files
Not reviewed (too large): rust/crates/ccusage-core/src/models-dev-pricing.json (~3,817 lines) - if these are generated or fixture files, add them to ignored paths to exclude them from future reviews.
Tip: cubic can generate docs of your entire codebase and keep them up to date. Try it here.
Re-trigger cubic
… too The asset-priced guard and the same-tier tie-break existed only in the snapshot generator, so an online refresh could still price a model the embedded snapshot deliberately excludes, and could pick a different catalog than the snapshot for the same model. Offline and default runs disagreed. The live `api.json` records neither who authored a model nor the authored modalities, so both have to be carried in from generation time. The generated artifact therefore grows an `assetPricedModelIds` list, and is renamed from `models-dev-provider-trust.json` to `models-dev-catalog-rules.json` now that it carries more than provider trust. The loader applies it the same way the generator does: the authored verdict decides where it exists, so a reseller describing an image model as text-only cannot reintroduce a per-image rate, and entries the authored catalog has never seen fall back to their own modalities, which is enough to reject an audio-only transcriber from any catalog. Same-tier ties now compare how much pricing detail an entry carries - explicit cache-read, cache-write, context limit - before falling back to the provider id, matching `shouldReplaceModelsDevPricingCandidate`. That needed the loader to track which catalog claimed each id during a pass, so a better candidate can replace a weaker one; ids absent from that record still belong to another pricing source and are left alone. Replacements do not increment the loaded count, which stays a count of resolved models. Reported by cubic on #1541.
|
@coderabbitai review Pushed 42e9edc and 351c668, addressing both cubic findings:
Also fixed a
|
🧩 Analysis chain🏁 Script executed: #!/bin/bash
set -euo pipefail
echo '== Commit metadata and changed files =='
git show --no-ext-diff --format=fuller --name-status 42e9edc
git show --no-ext-diff --format=fuller --name-status 351c668
echo
echo '== Focused combined diff =='
git diff --no-ext-diff --unified=100 42e9edc^ 351c668 -- \
nix/tools/models-dev-gen/compact.ts \
nix/tools/models-dev-gen/gen.ts \
nix/tools/models-dev-gen/compact.test.ts \
nix/tools/models-dev-gen/default.nix \
rust/crates/ccusage-core/build.rs \
rust/crates/ccusage-core/src/pricing.rs \
rust/crates/ccusage-core/src/models-dev-catalog-rules.json \
rust/crates/ccusage-core/src/models-dev-provider-trust.json \
nix/cargo-artifacts.nix \
package.nix \
justfile \
.github/scripts/update-models-dev-lock.nu
echo
echo '== Relevant source outlines =='
ast-grep outline nix/tools/models-dev-gen/compact.ts --items all
ast-grep outline nix/tools/models-dev-gen/gen.ts --items all
ast-grep outline rust/crates/ccusage-core/src/pricing.rs --items all
echo
echo '== Artifact/schema and live-loader references =='
rg -n -C 8 \
-e 'models-dev-(catalog-rules|provider-trust)' \
-e 'assetPricedModelIds' \
-e 'ModelsDevCatalogRules|ModelsDevProviderTrust' \
-e 'models_dev_.*trust|models_dev_.*rules' \
-e 'sort_by|sort_unstable_by|cmp\(' \
-e 'compareModelsDevPricingCandidates|compare.*Candidate' \
nix rust package.nix justfile .github
echo
echo '== Generator and runtime tests around the new behavior =='
rg -n -C 12 \
-e 'asset[- ]priced|assetPriced|whisper-large-v3|gemini-2\.5-flash-image|gemini-3-pro-image-preview' \
-e 'same[- ]tier|detail-first|cache_read|cache_write|context' \
nix/tools/models-dev-gen/compact.test.ts \
rust/crates/ccusage-core/src/pricing.rsLength of output: 202285 🏁 Script executed: #!/bin/bash
set -euo pipefail
echo '== TypeScript candidate comparator, authored modality classification, and artifact builder =='
rg -n -A150 -B25 \
-e 'function compareModelsDevPricingCandidates|const compareModelsDevPricingCandidates|export function compareModelsDevPricingCandidates' \
-e 'function isTokenPricedModel|export function isTokenPricedModel' \
-e 'function modelsDevCatalogRulesArtifact|export function modelsDevCatalogRulesArtifact' \
nix/tools/models-dev-gen/compact.ts
echo
echo '== Generator: catalog rules emission and asset-priced model collection =='
rg -n -A100 -B20 \
-e 'modelsDevCatalogRulesArtifact' \
-e 'assetPricedModelIds' \
-e 'CATALOG_RULES|RULES_OUTFILE|PROVIDER_TRUST_OUTFILE' \
-e 'isTokenPricedModel' \
nix/tools/models-dev-gen/gen.ts nix/tools/models-dev-gen/default.nix
echo
echo '== Runtime: rules rank, provider ordering, asset filtering, and claim replacement =='
sed -n '130,205p' rust/crates/ccusage-core/src/pricing.rs | cat -n
sed -n '415,540p' rust/crates/ccusage-core/src/pricing.rs | cat -n
echo
echo '== Complete artifact rename/reference audit =='
rg -n -C2 'models-dev-provider-trust|models-dev-catalog-rules|PROVIDER_TRUST_OUTFILE|CATALOG_RULES_OUTFILE' \
. \
-g '!flake.lock'
echo
echo '== Commit-level test changes only =='
git diff --no-ext-diff --unified=35 42e9edc^ 351c668 -- \
nix/tools/models-dev-gen/compact.test.ts \
rust/crates/ccusage-core/src/pricing.rsLength of output: 69915
Consequently, a reseller entry with The asset-priced catalog-rules artifact and its rename otherwise appear correctly wired through generation and runtime loading. ✅ Action performedReview finished.
|
@ryoppippi Looks good. I've verified both fixes against the code. P1 — asset-priced guard for live responses. The P2 — same-tier tie-breaking. The The Tip: get faster answers by chatting with cubic’s review copilot in the review UI |
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (1)
rust/crates/ccusage-core/src/pricing.rs (1)
2167-2241: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winUse fixtures for the new models.dev loader payloads.
These are parser/loader tests with full provider payloads; move the JSON into
fs_fixturefiles to keep cases reusable and readable. As per coding guidelines, “Prefer fixture-backed parser and loader tests for Rust code.”🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@rust/crates/ccusage-core/src/pricing.rs` around lines 2167 - 2241, Move the inline JSON payloads from the tests live_models_dev_pricing_skips_models_priced_per_asset and live_models_dev_pricing_prefers_the_more_detailed_entry_within_a_tier into reusable fs_fixture files, then update each test to load its corresponding fixture through the existing fixture helper while preserving the current assertions and behavior.Source: Coding guidelines
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@rust/crates/ccusage-core/src/pricing.rs`:
- Line 441: Update the pricing claim-selection logic around claims and the
replacement path near lines 494–500 and 526–528 to rank candidates
lexicographically by cache-read, cache-write, then context, matching the
generator rather than reducing them to a count. When replacing a winning entry,
replace the context limit atomically, including clearing the prior limit when
the winner has no context limit.
---
Nitpick comments:
In `@rust/crates/ccusage-core/src/pricing.rs`:
- Around line 2167-2241: Move the inline JSON payloads from the tests
live_models_dev_pricing_skips_models_priced_per_asset and
live_models_dev_pricing_prefers_the_more_detailed_entry_within_a_tier into
reusable fs_fixture files, then update each test to load its corresponding
fixture through the existing fixture helper while preserving the current
assertions and behavior.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: ef7304e1-dd77-4764-a464-90411b1e048f
📒 Files selected for processing (11)
.github/scripts/update-models-dev-lock.nujustfilenix/cargo-artifacts.nixnix/tools/models-dev-gen/compact.test.tsnix/tools/models-dev-gen/compact.tsnix/tools/models-dev-gen/default.nixnix/tools/models-dev-gen/gen.tspackage.nixrust/crates/ccusage-core/README.mdrust/crates/ccusage-core/src/models-dev-catalog-rules.jsonrust/crates/ccusage-core/src/pricing.rs
🚧 Files skipped from review as they are similar to previous changes (7)
- nix/cargo-artifacts.nix
- package.nix
- .github/scripts/update-models-dev-lock.nu
- rust/crates/ccusage-core/README.md
- nix/tools/models-dev-gen/gen.ts
- nix/tools/models-dev-gen/compact.test.ts
- nix/tools/models-dev-gen/compact.ts
There was a problem hiding this comment.
✅ No new issues found.
Reviewed changes — the five-line isTokenPricedModel call in the generator is replicated in the Rust runtime so the online refresh path makes the same filtering and tiebreaking decisions as the embedded snapshot.
- Renamed the artifact for clarity —
models-dev-provider-trust.jsonbecomesmodels-dev-catalog-rules.json, and it now carriesassetPricedModelIdsin addition to the owner and platform lists. The TypeScript export, Rust struct, Nix wiring, and docs all follow the rename. - Runtime
is_token_pricedmirrors the generator's filter —ModelsDevCatalogRuleschecks the precomputedassetPricedModelIdsblacklist first (covering models the authored catalog says price per asset), then falls through to the serving catalog's modalities for models the authored catalog does not list. TheModelsDevJson::Modelspath now runs through the same rules. - Same-tier detail tiebreaking —
load_models_dev_modelstracks per-model(trust, detail)claims so a reseller entry with cache-read and context-limit data can replace a weaker reseller entry for the same model, matching howshouldReplaceModelsDevPricingCandidateworks in the generator. - New tests fence the online parity —
live_models_dev_pricing_skips_models_priced_per_asset,live_models_dev_pricing_prefers_the_more_detailed_entry_within_a_tier, plus the existing catalog-ranking and flat-fee tests.
@v0 or keep the SHA fresh with Dependabot ｜ View workflow run ｜ Using DeepSeek Pro (free via Pullfrog for OSS) ｜ 𝕏
|
@cubic-dev-ai review Pushed 263c22f: cache-write explicitness is now tracked rather than inferred (the round-3 heuristic is gone), and the Grok troubleshooting note matches the Pricing bullet. Full workspace tests, Node tests and clippy green. |
@ryoppippi I have started the AI code review. It will take a few minutes to complete. |
There was a problem hiding this comment.
No issues found across 20 files
Not reviewed (too large): rust/crates/ccusage-core/src/models-dev-pricing.json (~23,508 lines), rust/crates/ccusage-core/src/pricing.rs (~1,811 lines) - if these are generated or fixture files, add them to ignored paths to exclude them from future reviews.
You've manually re-run cubic several times on this PR. Each manual re-review checks the full PR again and counts toward your usage quota. To preserve your usage limits, we recommend letting cubic automatically review new commits.
Tip: cubic can generate docs of your entire codebase and keep them up to date. Try it here.
Re-trigger cubic
There was a problem hiding this comment.
ℹ️ No new issues found in the commits since
860a587; the existing provider tie-break finding is unchanged.
Reviewed changes — Reviewed the cache-write explicitness change and its propagation through LiteLLM/models.dev parsing, overrides, the GLM fallback patch, and regression coverage. The focused ccusage-core pricing suite passed with 82 tests, and Rust formatting passed.
- Tracked cache-write provenance —
Pricingnow records whether cache-write rates were explicitly published in LiteLLM and models.dev data, rather than inferring provenance from the numeric value. - Restricted GLM fallback patching —
put_builtin_glmnow fills cache fields only when the loaded entry did not publish them, preserving an explicit rate even when it equals the derived default. - Covered the regression — added a test proving an explicitly published cache-write rate survives the GLM provider patch and updated affected constructors and Grok pricing documentation.
@v0 or keep the SHA fresh with Dependabot ｜ Fix it ➔ ｜ View workflow run ｜ Using GPT Luna (free via Pullfrog for OSS) ｜ 𝕏
This comment has been minimized.
This comment has been minimized.
Two catalogs declaring the same provider id are one provider to generation, which breaks exact-strength ties by `(sourceProviderId, sourceModelId)` - and with the ids colliding, by the source model key. The live loader kept whichever catalog arrived first instead, so a same-strength duplicate could resolve to a different rate online than in the snapshot. The claim slot now records the declared id and source key it was claimed under, and an exactly equal claim yields to a smaller source key from the same declared id. Differing declared ids were already settled by arrival order, which matches generation's ascending walk. Reported by pullfrog on #1541. Also narrows the new `cache_create_explicit` field to `pub(crate)`: nothing outside the crate reads it, and the hawk gate fails the build for exactly that.
|
@cubic-dev-ai review Pushed 5795951: exact claim ties within a declared provider id now break by source key (generation's ordering), and the new |
@ryoppippi I have started the AI code review. It will take a few minutes to complete. |
There was a problem hiding this comment.
No issues found across 20 files
Not reviewed (too large): rust/crates/ccusage-core/src/models-dev-pricing.json (~23,508 lines), rust/crates/ccusage-core/src/pricing.rs (~1,915 lines) - if these are generated or fixture files, add them to ignored paths to exclude them from future reviews.
You've manually re-run cubic several times on this PR. Each manual re-review checks the full PR again and counts toward your usage quota. To preserve your usage limits, we recommend letting cubic automatically review new commits.
Tip: cubic can generate docs of your entire codebase and keep them up to date. Try it here.
Re-trigger cubic
There was a problem hiding this comment.
✅ No new issues found.
Reviewed changes — Reviewed the delta in 5795951 against the prior Pullfrog review, including the exact-claim tie fix, the Rust visibility adjustment, and the associated regression coverage.
- Aligned equal-claim ordering — Added the declared provider ID and source key to live loader claim slots so ties within one declared provider resolve by source key, matching the generator's ordering even when provider map keys differ.
- Applied the hawk visibility fix — Narrowed
Pricing::cache_create_explicittopub(crate)without changing its internal provider-patch behavior. - Added regression coverage — Covered the case where the lexicographically smaller source key appears in the catalog with the larger map key.
Validation: focused ccusage-core pricing tests, Rust formatting, and all 22 models.dev generator tests passed. The direct hawk command was unavailable in this environment, but the visibility change matches the reported CI gate requirement and the Rust workspace compiled successfully.
@v0 or keep the SHA fresh with Dependabot ｜ View workflow run ｜ Using GPT Luna (free via Pullfrog for OSS) ｜ 𝕏
There was a problem hiding this comment.
✅ No new issues found.
Reviewed changes — reviewed the final incremental pricing commit after the prior Pullfrog review, focusing on deterministic live models.dev claim selection for duplicate declared provider IDs.
- Recorded complete claim identity —
ModelsDevClaimSlotnow retains the declared provider ID and source model key, allowing equal-strength claims from the same declared provider to use the generator's smaller-source-key tie-break. - Added cross-catalog regression coverage — the new test places the winning source key behind the later provider map key and verifies that the live loader selects the same rate as generation.
@v0 or keep the SHA fresh with Dependabot ｜ View workflow run ｜ Using GPT Luna (free via Pullfrog for OSS) ｜ 𝕏
ccusage performance comparisonPR SHA: Performance comparison skipped. Base package URL was not ready before 300.000s. Fixture performance comparison requires a base package when --base-dir is not provided. Base package: |
ccusage performance comparisonPR SHA: Performance comparison skipped. Base package URL was not ready before 300.000s. Fixture performance comparison requires a base package when --base-dir is not provided. Base package: |
hawk: pub(crate) was still wider than its uses, which never leave the pricing module.
|
@BlackSmith review it |
|
Your payment method failed for a Blacksmith invoice. [code]smith is paused until it is resolved. Update your payment method on the billing page. |
ccusage performance comparisonPR SHA: Performance comparison skipped. Base package URL was not ready before 300.000s. Fixture performance comparison requires a base package when --base-dir is not provided. Base package: |
ccusage performance comparisonPR SHA: Performance comparison skipped. Base package URL was not ready before 300.000s. Fixture performance comparison requires a base package when --base-dir is not provided. Base package: |
|
@codesmith-bot review it |
|
Your payment method failed for a Blacksmith invoice. [code]smith is paused until it is resolved. Update your payment method on the billing page. |
There was a problem hiding this comment.
✅ No new issues found.
Reviewed changes — reviewed the merge from main since the prior Pullfrog review, including the regenerated models.dev catalog rules and pricing snapshot plus the related dependency pins.
- Regenerated catalog rules — Added the newly authored model ids from the updated models.dev input while preserving the generator/runtime trust and eligibility contract.
- Regenerated pricing data — Added provider-specific entries and updated rates, context limits, and long-context tiers from the new pinned catalog.
- Updated dependency pins — Refreshed the models.dev and LiteLLM inputs and the workspace package-manager pin used to produce the snapshots.
Validation: the focused ccusage-core pricing suite passed all 83 tests, including snapshot parsing, asset-priced exclusion, and Grok long-context regressions.
@v0 or keep the SHA fresh with Dependabot ｜ View workflow run ｜ Using GPT Luna (free via Pullfrog for OSS) ｜ 𝕏
ccusage performance comparisonPR SHA: Performance comparison skipped. Base package URL was not ready before 300.000s. Fixture performance comparison requires a base package when --base-dir is not provided. Base package: |
ccusage performance comparisonPR SHA: Performance comparison skipped. Base package URL was not ready before 300.000s. Fixture performance comparison requires a base package when --base-dir is not provided. Base package: |
Summary
models.dev publishes one catalog per provider, so every model appears once per provider that serves it, each with that provider's own rates — and some of those rates are two-stage: Grok, GPT-5.x and Gemini bill requests above a context boundary at higher prices. This PR fixes which catalog a rate comes from, and prices those long-context tiers from data instead of a hardcoded table.
What changed
Selection is by catalog trust. The authoring catalog wins, cloud platforms reselling at list price plus a published regional premium come next, resellers are last. Authorship is derived from the
models/<author>/<id>.tomldirectory scan, so new authors need no maintenance. Every published id is embedded; separately priced tiers and unversioned aliases are marked exact-only so the fuzzy lookup cannot resolve a base model onto them.Long-context tiers come from models.dev
cost.tiers. The generator emits each model's band in the upstream shape, the Rust loader parses the snapshot and a liveapi.jsonwith the same code, and models whose base rates come from LiteLLM (gpt-5.x) get the band filled in from the snapshot. The hardcodedbuiltin_long_context_ratestable is deleted — which also fixed two prices it had frozen after OpenAI's gpt-5.6 price cut (luna was over-reported at 5x). Built-in entries are now a last resort, never an overwrite.The tier is selected by the whole context. Adapters normalize usage so
input_tokensis the uncached remainder, but vendors choose the tier by the request's whole context: a Grok turn re-reading 8M cached tokens with 10K fresh ones is a long-context request. The comparison now sums uncached input, cache reads and cache writes.Dotted and dashed spellings merge onto one entry.
grok-4.5(xAI, tiered) andgrok-4-5(a reseller's flat copy) tied in the fuzzy lookup, sogrok-4.5-buildcould bill flat. Spellings contend for one slot keyed by the normalized id in the generator and the live loader alike, and within a trust tier a catalog publishing a long-context band outranks one with more cache detail.The runtime loader applies the same rules as generation — trust ranking, token/asset verdicts from the authored catalog, zero-cost and modality guards, detail-ordered tie-breaks — carried in via the generated
models-dev-catalog-rules.json, so--offlineand a live refresh cannot disagree.Eligibility reads the authored catalog. Asset-priced models (
whisper-large-v3per second,gemini-*-imageper image) stay out even when a reseller describes them as text-only. The committed snapshots are also excluded from thetyposformatter, which had rewrittengemini-2.5-flash-nothinkinto a nonexistent id, and are deflated at embed time.Grok resolves pricing candidates exactly before fuzzily, so an override naming the request's model precisely beats a fuzzy hit on a shorter spelling.
Real-log verification
169-day Claude + Codex log: byte-identical to main in online and offline modes — the models it contains were priced correctly before and stay untouched.
85-session Grok Build log (grok-4.5/4.6-build; 47 of 86 turn×model rows above 200K context):
auto/display(recorded ticks)calculateWhole-catalogue probe (every snapshot id × {100K, 300K} input): nothing loses pricing; every changed cost is an intended category (tier application, regional premiums, spelling-merge corrections).
Testing
cargo test --workspace,just test,nix flake check— all green. New regression fences: resold-model list rates, asset-priced exclusion both ways, tier data through both grok spellings, whole-context tier selection with cache-heavy usage, spelling-merge including entry removal, tier-aware and detail-ordered claim comparison, exact-before-fuzzy candidate resolution, the gpt-5.6 price-cut rates, and the snapshot's byte-for-byte reproducibility from the pinned input.Remaining follow-ups (out of scope)
experimental.modes.fast) still bill via the fast multiplier; a logged-fastid bills at the only catalogue rate published for it.