Skip to content

fix(schema): classify off_peak_pricing as a structured object in the model prices schema generator - #41847

Merged
kerry-berri merged 5 commits into
mainfrom
litellm_lit_8128_off_peak_pricing_schema
Sep 19, 2026
Merged

kerry-berri merged 5 commits into
mainfrom
litellm_lit_8128_off_peak_pricing_schema

Conversation

@devin-ai-integration

@devin-ai-integration devin-ai-integration Bot commented Sep 18, 2026 •

Copy link
Copy Markdown
Contributor

TLDR

Problem this solves:

  • The schema generator rejects off_peak_pricing as an unclassified key
  • The provider price sync could not publish the openrouter catalog because of it

How it solves it:

  • Adds off_peak_pricing to OBJECT_KEYS as a structured object schema
  • Shape mirrors the runtime OffPeakPricing TypedDict: hours_utc, windows, weekday_timezone, five override rates
  • Windows must be HH:MM-HH:MM, weekday names must be ones the runtime recognizes, and a block needs hours_utc or windows
  • Regression tests accept a valid block and reject malformed ones

User Flow

Before: the price sync bot cannot publish an openrouter catalog update that carries an off-peak rate block

  1. The sync bot fetches openrouter prices and finds a model with a time-of-day discount
  2. It tries to open a catalog PR against BerriAI/litellm and reports in Slack: "The catalog uses fields its schema does not define and the sync cannot classify: off_peak_pricing"
  3. Running python ci_cd/generate_model_prices_schema.py on that catalog prints "Unclassified keys in model_prices_and_context_window.json: off_peak_pricing" and no PR is opened

After: the same catalog validates and the sync can publish

  1. The sync bot fetches openrouter prices and finds a model with a time-of-day discount
  2. python ci_cd/generate_model_prices_schema.py renders off_peak_pricing as an object with hours_utc, windows, weekday_timezone and the override rates, and the catalog validates against the result
  3. The sync opens its catalog PR as usual

Relevant issues

Companion change in BerriAI/litellm-providers keeps the sync side's schema classification identical to this generator

Affected release

Linear ticket

Resolves LIT-8128

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

  • I have added meaningful tests
  • The handful of test files covering my change pass locally, e.g. uv run pytest tests/test_litellm/<your_test_file>.py -v. Leave the suites (make test-unit-*, make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more
  • My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.)
  • My PR's scope is as isolated as possible; it only solves 1 specific problem
  • I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

This change only touches the schema generator that CI and the price sync run, so the proof is the generator itself, run in a worktree at each commit against the current catalog plus one entry carrying an off-peak block:

"openrouter/deepseek/deepseek-chat-off-peak-demo": {
    "litellm_provider": "openrouter",
    "mode": "chat",
    "input_cost_per_token": 2e-06,
    "output_cost_per_token": 8e-06,
    "off_peak_pricing": {"hours_utc": "16:30-00:30", "input_cost_per_token": 1e-06, "output_cost_per_token": 4e-06}
}

Before (db756b9)

  1. python ci_cd/generate_model_prices_schema.py --check
  2. Output: Unclassified keys in model_prices_and_context_window.json: off_peak_pricing. Add them to the key tables in generate_model_prices_schema.py and rerun it.

After (4b215e2)

  1. python ci_cd/generate_model_prices_schema.py --check
  2. Output: model_prices_and_context_window.schema.json is out of sync with model_prices_and_context_window.json. Run python ci_cd/generate_model_prices_schema.py and commit the result. (the key is classified; the committed schema simply does not yet carry it because no entry on main uses it)
  3. python ci_cd/generate_model_prices_schema.py then --check again: model_prices_and_context_window.schema.json is in sync and model_prices_and_context_window.json validates against it

On the unmodified main catalog --check still passes at the PR tip, so the committed schema is unchanged by this PR

Type

🐛 Bug Fix

Caveats (if any)

Low

  • The committed schema file does not change until a catalog entry actually uses off_peak_pricing
  • The generator has preexisting ruff findings (UP045, T201) that this PR does not touch

Final Attestation

  • The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

Link to Devin session: https://app.devin.ai/sessions/b3650eb76464401f9bd6e0e7264bf62b
Open in Devin Desktop: https://app.devin.ai/desktop/session/b3650eb76464401f9bd6e0e7264bf62b?variant=devin


Note

Low Risk
CI/schema-generator and test-only changes; no runtime billing logic is modified, only validation rules for catalog JSON.

Overview
Teaches the model-prices schema generator to recognize off_peak_pricing instead of failing catalog validation when OpenRouter (or other) entries include time-of-day rate overrides.

The new object schema matches the runtime shape: optional hours_utc (single or list of HH:MM-HH:MM UTC windows, including midnight wrap) or windows with per-window weekdays (ISO 1–7 or English day names) and optional weekday_timezone, plus nonnegative overrides for input/output/reasoning/cache token costs. Blocks must supply hours_utc or windows, and unknown keys are rejected.

Tests assert a representative off-peak entry validates and that common malformed shapes (bad times, wrong types, missing windows, invalid weekdays, extra fields) do not.

Reviewed by Cursor Bugbot for commit 7208e31. Bugbot is set up for automated code reviews on this repo. Configure here.

…model prices schema generator

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
@devin-ai-integration
devin-ai-integration Bot requested a review from a team September 18, 2026 18:12
@devin-ai-integration

Copy link
Copy Markdown
Contributor Author

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

  • Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
  • Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

  • Disable automatic comment, CI, and merge conflict monitoring

@CLAassistant

Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

@greptile-apps

greptile-apps Bot commented Sep 18, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerConfidence Score: 5/5

The PR appears safe to merge, with the previous findings fixed or manually resolved and no new actionable issues

Summary

This PR extends the model-prices schema generator to recognize and validate structured off-peak pricing metadata. It validates time windows, weekdays, override rates, and the requirement for at least one schedule source. Regression tests cover valid metadata and malformed blocks

Reviews (3) · Last reviewed commit: "fix(schema): annotate new off_peak_prici..."

Comment thread ci_cd/generate_model_prices_schema.py Outdated
Comment thread ci_cd/generate_model_prices_schema.py
…ricing

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
@devin-ai-integration

Copy link
Copy Markdown
Contributor Author

Pushed 4b215e2 fixing both Greptile findings: window strings and weekday names are now pattern checked, and a block must carry a schedule

@devin-ai-integration

Copy link
Copy Markdown
Contributor Author

osv-scan failure is unrelated to this PR: a new anyio 4.13.0 advisory (GHSA-82r6-8w77-94w6) fails osv-scan on every open PR, including #41838 and #41814.

@codecov

codecov Bot commented Sep 18, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@kerry-berri

Copy link
Copy Markdown
Contributor

@greptileai

@kerry-berri

Copy link
Copy Markdown
Contributor

bugbot run

Comment thread ci_cd/generate_model_prices_schema.py Outdated

@cursor cursor Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
@kerry-berri

Copy link
Copy Markdown
Contributor

@greptileai

@kerry-berri

Copy link
Copy Markdown
Contributor

bugbot run

@cursor cursor Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit 7208e31. Configure here.

@kerry-berri
kerry-berri merged commit f71129f into main Sep 19, 2026
85 checks passed
@kerry-berri
kerry-berri deleted the litellm_lit_8128_off_peak_pricing_schema branch September 19, 2026 00:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants