fix(pricing): stop asserting which models upstream has not shipped yet - #1523
Conversation
`fuzzy_match_allows_date_like_suffixes_for_known_numeric_model_versions` loads the embedded catalogue and then asserted that `claude-opus-4-9` and `claude-opus-5` resolve to nothing. Both litellm and models.dev now ship `claude-opus-5`, so the assertion fails the moment either input is refreshed: every hourly update-pricing run has failed its Rust test leg since, and the snapshot could no longer land on main. The remaining assertion is the one the test is named after - a date-like suffix on a version the catalogue really has. Which versions must not fuzzy match is already covered by fuzzy_match_requires_model_key_boundaries and fuzzy_match_does_not_fall_back_across_numeric_model_versions, which build their own maps and therefore cannot rot when upstream adds a version. Verified by updating both inputs, regenerating the snapshots and running the suite against data that does contain claude-opus-5.
|
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (1)
💤 Files with no reviewable changes (1)
📝 WalkthroughWalkthroughThe fuzzy-matching pricing test now verifies that ChangesPricing fuzzy matching
Estimated code review effort: 1 (Trivial) | ~2 minutes Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Deploying with
|
| Status | Name | Latest Commit | Preview URL | Updated (UTC) |
|---|---|---|---|---|
| ✅ Deployment successful! View logs |
ccusage-guide | 37d4ebe | Commit Preview URL Branch Preview URL |
Jul 28 2026, 12:08 PM |
ccusage performance comparisonPR SHA: This compares the Rust PR release binary against the configured base package on the same CI runner. Package runtime diagnosticsCompares the PR package wrapper, the installed native optional dependency binary, and the workspace release binary on the same large fixture. This identifies whether slow package results come from JavaScript wrapper overhead, the published native binary build, or the Rust core itself. Fixtures: Claude
Committed fixture performanceCommitted small fixtures for stable PR-to-PR feedback and explicit Claude/Codex command coverage. Fixtures: Claude
Large real-world-shaped fixture performanceGenerated fixtures shaped from aggregate local log statistics: thousands of JSONL files, many small sessions, and a long tail of larger sessions. No real prompts, paths, or outputs are stored in the fixtures. Fixtures: Claude
Artifact size
Lower medians and smaller artifacts are better. CI runner noise still applies; use same-run ratios as directional PR feedback, not release guarantees. |
ccusage performance comparisonPR SHA: This compares the PR package against the configured base package on the same CI runner. Package runtime diagnosticsCompares the PR package wrapper, the installed native optional dependency binary, and the workspace release binary on the same large fixture. This identifies whether slow package results come from JavaScript wrapper overhead, the published native binary build, or the Rust core itself. Fixtures: Claude
Committed fixture performanceCommitted small fixtures for stable PR-to-PR feedback and explicit Claude/Codex command coverage. Fixtures: Claude
Large real-world-shaped fixture performanceGenerated fixtures shaped from aggregate local log statistics: thousands of JSONL files, many small sessions, and a long tail of larger sessions. No real prompts, paths, or outputs are stored in the fixtures. Fixtures: Claude
Artifact size
Lower medians and smaller artifacts are better. CI runner noise still applies; use same-run ratios as directional PR feedback, not release guarantees. |
Summary
Every hourly
update-pricingrun has been failing its Rust test leg, so the refreshed pricing snapshot cannot reachmain. The workflow is not at fault — the validation added in #1509 is doing its job and refusing to push a tree whose tests fail:The test loads the embedded catalogue and then asserted that
claude-opus-4-9andclaude-opus-5resolve to nothing. Both litellm and models.dev now shipclaude-opus-5, so the assertion fails the moment either input is refreshed.What Changed
Drops the two absence assertions. What remains is the case the test is named after: a date-like suffix on a version the catalogue really has (
claude-opus-4-8-20270898).Which versions must not fuzzy match is already covered by
fuzzy_match_requires_model_key_boundariesandfuzzy_match_does_not_fall_back_across_numeric_model_versions— both build their ownPricingMap, including the exactclaude-opus-4-9andclaude-opus-5cases, so they cannot rot when upstream adds a version.Testing
Reproduced and verified against the data that breaks it:
nix flake update litellm models-devplusjust gen-models-dev-pricing, after which the snapshot does containclaude-opus-5. Then:The lock and snapshots were reverted afterwards;
update-pricingwill land those itself once this is onmain.Need help on this PR? Tag
@codesmith-botwith what you need. Autofix is enabled.Summary by cubic
Fixes the failing pricing test by removing brittle absence checks for
claude-opus-4-9andclaude-opus-5. This unblocks hourlyupdate-pricingsnapshots when upstream ships new versions.fuzzy_match_allows_date_like_suffixes_for_known_numeric_model_versions; kept the date-suffix check for a known version.fuzzy_match_requires_model_key_boundariesandfuzzy_match_does_not_fall_back_across_numeric_model_versions.Written for commit 37d4ebe. Summary will update on new commits.
Summary by CodeRabbit