build(perf): migrate benchmark harness to Babashka - #1432
Conversation
Replace the large Nushell PR benchmark script with a Babashka implementation split by data, system, benchmark, report, and orchestration responsibilities. The new process boundary keeps argv, environment, and working-directory data explicit while preserving hyperfine, package installation, memory, size, and Markdown behavior. Move the CI caller and profiling guidance to the executable Babashka entry point. Add focused tests behind their own Nix shebang so contributors can run the harness suite without adding Babashka to the full development shell.
Route small command-oriented automation to Nushell and data-heavy, testable automation to Babashka. Keep production binaries in Rust and npm-integrated APIs in TypeScript so future tooling changes follow the same criteria used by the benchmark migration.
|
@coderabbitai review |
@ryoppippi I have started the AI code review. It will take a few minutes to complete. |
|
✅ Action performedReview finished.
|
📝 WalkthroughWalkthroughThe PR replaces the Nushell performance comparison script with a Babashka implementation split into CLI/data, system, benchmarking, reporting, and orchestration modules. It updates CI and documentation references, adds a just recipe, and introduces focused Babashka tests. ChangesPerformance comparison harness
Estimated code review effort: 4 (Complex) | ~60 minutes Sequence Diagram(s)sequenceDiagram
participant CLI
participant compare_pr_performance_main
participant benchmark
participant report
participant Output
CLI->>compare_pr_performance_main: -main arguments
compare_pr_performance_main->>benchmark: run-benchmark-suites
benchmark-->>compare_pr_performance_main: measurements and sections
compare_pr_performance_main->>report: render-report
report-->>compare_pr_performance_main: Markdown output
compare_pr_performance_main->>Output: write-output!
Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Deploying with
|
| Status | Name | Latest Commit | Preview URL | Updated (UTC) |
|---|---|---|---|---|
| ✅ Deployment successful! View logs |
ccusage-guide | 6c90fc2 | Commit Preview URL Branch Preview URL |
Jul 12 2026, 12:24 PM |
Execute the self-contained benchmark harness test entry point in the CI test job so changes to CLI parsing, normalization, fallback decisions, and report rendering cannot bypass pull request validation.
|
@coderabbitai @cubic-dev-ai Added CI execution for the Babashka harness tests in cf46b61. Local validation: 10 tests, 24 assertions passed. Please review the latest commit. |
@ryoppippi I have started the AI code review. It will take a few minutes to complete. |
|
✅ Action performedReview finished.
|
ccusage
@ccusage/ccusage-darwin-arm64
@ccusage/ccusage-darwin-x64
@ccusage/ccusage-linux-arm64
@ccusage/ccusage-linux-x64
@ccusage/ccusage-win32-x64
commit: |
ccusage performance comparisonPR SHA: This compares the Rust PR release binary against the configured base package on the same CI runner. Package runtime diagnosticsCompares the PR package wrapper, the installed native optional dependency binary, and the workspace release binary on the same large fixture. This identifies whether slow package results come from JavaScript wrapper overhead, the published native binary build, or the Rust core itself. Fixtures: Claude
Committed fixture performanceCommitted small fixtures for stable PR-to-PR feedback and explicit Claude/Codex command coverage. Fixtures: Claude
Large real-world-shaped fixture performanceGenerated fixtures shaped from aggregate local log statistics: thousands of JSONL files, many small sessions, and a long tail of larger sessions. No real prompts, paths, or outputs are stored in the fixtures. Fixtures: Claude
Artifact size
Lower medians and smaller artifacts are better. CI runner noise still applies; use same-run ratios as directional PR feedback, not release guarantees. |
ccusage performance comparisonPR SHA: This compares the PR package against the configured base package on the same CI runner. Package runtime diagnosticsCompares the PR package wrapper, the installed native optional dependency binary, and the workspace release binary on the same large fixture. This identifies whether slow package results come from JavaScript wrapper overhead, the published native binary build, or the Rust core itself. Fixtures: Claude
Committed fixture performanceCommitted small fixtures for stable PR-to-PR feedback and explicit Claude/Codex command coverage. Fixtures: Claude
Large real-world-shaped fixture performanceGenerated fixtures shaped from aggregate local log statistics: thousands of JSONL files, many small sessions, and a long tail of larger sessions. No real prompts, paths, or outputs are stored in the fixtures. Fixtures: Claude
Artifact size
Lower medians and smaller artifacts are better. CI runner noise still applies; use same-run ratios as directional PR feedback, not release guarantees. |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 4
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@apps/ccusage/scripts/compare_pr_performance/benchmark.clj`:
- Around line 21-38: Update the unsupported-platform branch in measure-memory so
it writes a warning with write-progress and returns nil instead of calling
fail!. Warn once before or outside the per-sample keep iteration when
timed-command cannot produce an argv, then skip memory measurement gracefully
while preserving the existing non-zero-exit and parse-error handling.
In `@apps/ccusage/scripts/compare_pr_performance/system.clj`:
- Around line 1-9: Update run-process to accept and enforce a cancellable
timeout, terminating the underlying process and returning a timeout result when
it expires so all callers remain bounded by --package-runner-timeout-ms. Pass
that timeout through the package URL, install, packing, and git revision flows,
and configure the curl probe with explicit connection and read limits.
- Around line 19-20: Update platform-name to recognize os.name values beginning
with “Windows” (including version-qualified values such as Windows 10 and
Windows 11) and return “win32”; preserve the existing Mac OS X and Linux
mappings and lowercase fallback for other platforms.
- Around line 93-110: The packed tarball filename from `packed-tarball-size` is
relative to the temporary destination, but it is currently passed directly to
`optional-file-size`. Join `destination` with `filename` before sizing the file,
preserving the existing missing-file handling and cleanup behavior.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 0f1cf3a0-401b-44ed-9af7-6039dd3c3471
📒 Files selected for processing (13)
.agents/skills/profile/references/rust.md.github/workflows/ci.yamlAGENTS.mdapps/ccusage/AGENTS.mdapps/ccusage/justfileapps/ccusage/scripts/compare-pr-performance.bbapps/ccusage/scripts/compare-pr-performance.nuapps/ccusage/scripts/compare-pr-performance_test.cljapps/ccusage/scripts/compare_pr_performance/benchmark.cljapps/ccusage/scripts/compare_pr_performance/data.cljapps/ccusage/scripts/compare_pr_performance/main.cljapps/ccusage/scripts/compare_pr_performance/report.cljapps/ccusage/scripts/compare_pr_performance/system.clj
💤 Files with no reviewable changes (1)
- apps/ccusage/scripts/compare-pr-performance.nu
ccusage performance comparisonPR SHA: This compares the Rust PR release binary against the configured base package on the same CI runner. Package runtime diagnosticsCompares the PR package wrapper, the installed native optional dependency binary, and the workspace release binary on the same large fixture. This identifies whether slow package results come from JavaScript wrapper overhead, the published native binary build, or the Rust core itself. Fixtures: Claude
Committed fixture performanceCommitted small fixtures for stable PR-to-PR feedback and explicit Claude/Codex command coverage. Fixtures: Claude
Large real-world-shaped fixture performanceGenerated fixtures shaped from aggregate local log statistics: thousands of JSONL files, many small sessions, and a long tail of larger sessions. No real prompts, paths, or outputs are stored in the fixtures. Fixtures: Claude
Artifact size
Lower medians and smaller artifacts are better. CI runner noise still applies; use same-run ratios as directional PR feedback, not release guarantees. |
ccusage performance comparisonPR SHA: This compares the PR package against the configured base package on the same CI runner. Package runtime diagnosticsCompares the PR package wrapper, the installed native optional dependency binary, and the workspace release binary on the same large fixture. This identifies whether slow package results come from JavaScript wrapper overhead, the published native binary build, or the Rust core itself. Fixtures: Claude
Committed fixture performanceCommitted small fixtures for stable PR-to-PR feedback and explicit Claude/Codex command coverage. Fixtures: Claude
Large real-world-shaped fixture performanceGenerated fixtures shaped from aggregate local log statistics: thousands of JSONL files, many small sessions, and a long tail of larger sessions. No real prompts, paths, or outputs are stored in the fixtures. Fixtures: Claude
Artifact size
Lower medians and smaller artifacts are better. CI runner noise still applies; use same-run ratios as directional PR feedback, not release guarantees. |
Normalize version-qualified Windows os.name values to win32 so native executable and package paths use the expected suffixes. Resolve relative pnpm pack filenames against the temporary destination while preserving the absolute paths emitted by current pnpm versions. Add regression coverage for both platform normalization and relative or absolute tarball filenames.
|
@coderabbitai @cubic-dev-ai Addressed the actionable review findings in b8dbb41 and replied to each CodeRabbit thread. Validation: 12 tests, 30 assertions, and local packed-tarball smoke passed. Please review the latest commit. |
@ryoppippi I have started the AI code review. It will take a few minutes to complete. |
|
✅ Action performedReview finished.
|
|
✅ Action performedReview finished.
|
|
✅ Action performedReview finished.
|
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
There was a problem hiding this comment.
All reported issues were addressed across 13 files
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
ccusage performance comparisonPR SHA: This compares the Rust PR release binary against the configured base package on the same CI runner. Package runtime diagnosticsCompares the PR package wrapper, the installed native optional dependency binary, and the workspace release binary on the same large fixture. This identifies whether slow package results come from JavaScript wrapper overhead, the published native binary build, or the Rust core itself. Fixtures: Claude
Committed fixture performanceCommitted small fixtures for stable PR-to-PR feedback and explicit Claude/Codex command coverage. Fixtures: Claude
Large real-world-shaped fixture performanceGenerated fixtures shaped from aggregate local log statistics: thousands of JSONL files, many small sessions, and a long tail of larger sessions. No real prompts, paths, or outputs are stored in the fixtures. Fixtures: Claude
Artifact size
Lower medians and smaller artifacts are better. CI runner noise still applies; use same-run ratios as directional PR feedback, not release guarantees. |
ccusage performance comparisonPR SHA: This compares the PR package against the configured base package on the same CI runner. Package runtime diagnosticsCompares the PR package wrapper, the installed native optional dependency binary, and the workspace release binary on the same large fixture. This identifies whether slow package results come from JavaScript wrapper overhead, the published native binary build, or the Rust core itself. Fixtures: Claude
Committed fixture performanceCommitted small fixtures for stable PR-to-PR feedback and explicit Claude/Codex command coverage. Fixtures: Claude
Large real-world-shaped fixture performanceGenerated fixtures shaped from aggregate local log statistics: thousands of JSONL files, many small sessions, and a long tail of larger sessions. No real prompts, paths, or outputs are stored in the fixtures. Fixtures: Claude
Artifact size
Lower medians and smaller artifacts are better. CI runner noise still applies; use same-run ratios as directional PR feedback, not release guarantees. |
Use remote tarball sizing only after the corresponding preview package was installed successfully. When either package URL times out, benchmark and size the available local checkout so fallback runs can still produce a complete report. Cover base and head source selection and verify both unavailable URLs through a committed-fixture smoke run.
|
@coderabbitai @cubic-dev-ai Addressed the valid fallback review finding in bf40013 and replied to all Cubic threads. Validation: 12 tests, 34 assertions, plus unavailable base/head URL fallback integration smoke. Please review the latest commit. |
@ryoppippi I have started the AI code review. It will take a few minutes to complete. |
|
✅ Action performedReview finished.
|
ccusage performance comparisonPR SHA: This compares the PR package against the configured base package on the same CI runner. Package runtime diagnosticsCompares the PR package wrapper, the installed native optional dependency binary, and the workspace release binary on the same large fixture. This identifies whether slow package results come from JavaScript wrapper overhead, the published native binary build, or the Rust core itself. Fixtures: Claude
Committed fixture performanceCommitted small fixtures for stable PR-to-PR feedback and explicit Claude/Codex command coverage. Fixtures: Claude
Large real-world-shaped fixture performanceGenerated fixtures shaped from aggregate local log statistics: thousands of JSONL files, many small sessions, and a long tail of larger sessions. No real prompts, paths, or outputs are stored in the fixtures. Fixtures: Claude
Artifact size
Lower medians and smaller artifacts are better. CI runner noise still applies; use same-run ratios as directional PR feedback, not release guarantees. |
ccusage performance comparisonPR SHA: This compares the Rust PR release binary against the configured base package on the same CI runner. Package runtime diagnosticsCompares the PR package wrapper, the installed native optional dependency binary, and the workspace release binary on the same large fixture. This identifies whether slow package results come from JavaScript wrapper overhead, the published native binary build, or the Rust core itself. Fixtures: Claude
Committed fixture performanceCommitted small fixtures for stable PR-to-PR feedback and explicit Claude/Codex command coverage. Fixtures: Claude
Large real-world-shaped fixture performanceGenerated fixtures shaped from aggregate local log statistics: thousands of JSONL files, many small sessions, and a long tail of larger sessions. No real prompts, paths, or outputs are stored in the fixtures. Fixtures: Claude
Artifact size
Lower medians and smaller artifacts are better. CI runner noise still applies; use same-run ratios as directional PR feedback, not release guarantees. |
…platforms Add a cancellable timeout to run-process and thread --package-runner-timeout-ms through the package URL probe, install, pnpm pack, and git rev-parse flows so a stalled child cannot outlive the deadline; give the curl probe and download explicit connect and read limits. measure-memory now warns once and skips gracefully when /usr/bin/time is unavailable (unsupported platforms) instead of throwing and aborting the entire benchmark run. Co-authored-by: Codesmith <[email protected]>
…pported platforms" This reverts commit 9140a99.
|
@coderabbitai @cubic-dev-ai Reverted the Codesmith auto-fix in 6c90fc2 because it reintroduced the two review suggestions already withdrawn for Nushell parity (global child timeouts and skipping unsupported RSS). The validated fallback fix in bf40013 remains. Tests: 12 tests, 34 assertions passed. Please review the latest commit. |
@ryoppippi I have started the AI code review. It will take a few minutes to complete. |
|
✅ Action performedReview finished.
|
ccusage performance comparisonPR SHA: This compares the Rust PR release binary against the configured base package on the same CI runner. Package runtime diagnosticsCompares the PR package wrapper, the installed native optional dependency binary, and the workspace release binary on the same large fixture. This identifies whether slow package results come from JavaScript wrapper overhead, the published native binary build, or the Rust core itself. Fixtures: Claude
Committed fixture performanceCommitted small fixtures for stable PR-to-PR feedback and explicit Claude/Codex command coverage. Fixtures: Claude
Large real-world-shaped fixture performanceGenerated fixtures shaped from aggregate local log statistics: thousands of JSONL files, many small sessions, and a long tail of larger sessions. No real prompts, paths, or outputs are stored in the fixtures. Fixtures: Claude
Artifact size
Lower medians and smaller artifacts are better. CI runner noise still applies; use same-run ratios as directional PR feedback, not release guarantees. |
ccusage performance comparisonPR SHA: This compares the PR package against the configured base package on the same CI runner. Package runtime diagnosticsCompares the PR package wrapper, the installed native optional dependency binary, and the workspace release binary on the same large fixture. This identifies whether slow package results come from JavaScript wrapper overhead, the published native binary build, or the Rust core itself. Fixtures: Claude
Committed fixture performanceCommitted small fixtures for stable PR-to-PR feedback and explicit Claude/Codex command coverage. Fixtures: Claude
Large real-world-shaped fixture performanceGenerated fixtures shaped from aggregate local log statistics: thousands of JSONL files, many small sessions, and a long tail of larger sessions. No real prompts, paths, or outputs are stored in the fixtures. Fixtures: Claude
Artifact size
Lower medians and smaller artifacts are better. CI runner noise still applies; use same-run ratios as directional PR feedback, not release guarantees. |
ccusage performance comparisonPR SHA: This compares the Rust PR release binary against the configured base package on the same CI runner. Package runtime diagnosticsCompares the PR package wrapper, the installed native optional dependency binary, and the workspace release binary on the same large fixture. This identifies whether slow package results come from JavaScript wrapper overhead, the published native binary build, or the Rust core itself. Fixtures: Claude
Committed fixture performanceCommitted small fixtures for stable PR-to-PR feedback and explicit Claude/Codex command coverage. Fixtures: Claude
Large real-world-shaped fixture performanceGenerated fixtures shaped from aggregate local log statistics: thousands of JSONL files, many small sessions, and a long tail of larger sessions. No real prompts, paths, or outputs are stored in the fixtures. Fixtures: Claude
Artifact size
Lower medians and smaller artifacts are better. CI runner noise still applies; use same-run ratios as directional PR feedback, not release guarantees. |
ccusage performance comparisonPR SHA: This compares the PR package against the configured base package on the same CI runner. Package runtime diagnosticsCompares the PR package wrapper, the installed native optional dependency binary, and the workspace release binary on the same large fixture. This identifies whether slow package results come from JavaScript wrapper overhead, the published native binary build, or the Rust core itself. Fixtures: Claude
Committed fixture performanceCommitted small fixtures for stable PR-to-PR feedback and explicit Claude/Codex command coverage. Fixtures: Claude
Large real-world-shaped fixture performanceGenerated fixtures shaped from aggregate local log statistics: thousands of JSONL files, many small sessions, and a long tail of larger sessions. No real prompts, paths, or outputs are stored in the fixtures. Fixtures: Claude
Artifact size
Lower medians and smaller artifacts are better. CI runner noise still applies; use same-run ratios as directional PR feedback, not release guarantees. |
* chore(ci): remove pullfrog because they dont serve free tokens anymore * Restore `pullfrog.yml` workflow * feat(statusline): show reasoning effort level next to model name (ccusage#1405) * feat(statusline): show reasoning effort level next to model name Claude Code 2.1.119+ includes an optional top-level effort.level field (low, medium, high, xhigh, or max) in the statusline hook JSON, reflecting the live /effort setting. Parse it from the hook input and append it to the model segment, e.g. '🤖 Fable 5 (high)'. The field is absent for models without the effort parameter and on older Claude Code versions, in which case the statusline keeps showing just the model label as before. * test(statusline): add Fable 5 fixture with effort level Adds a manual statusline fixture for the latest model shape, including the effort.level field, plus a test-statusline-fable5 recipe wired into test-statusline-all so the effort display can be smoke-tested from the CLI. * docs(statusline): document effort level next to the model name Updates the statusline guide examples to the current model display ('Fable 5 (high)') and explains that the reasoning effort level comes from Claude Code 2.1.119+, with a fallback example for models or versions that do not report it. * Revert "Restore `pullfrog.yml` workflow" This reverts commit 04f45b0. * fix(codex): skip forked session replay history (ccusage#1369) * feat(json): emit modelBreakdowns in per-agent JSON reports (ccusage#1395) Per-agent subcommands (ccusage pi|opencode|amp|hermes|... daily/weekly/ monthly/session --json) compute per-model cost breakdowns — the table view renders them with --breakdown — but the shared per-agent JSON serializer never emitted them, forcing JSON consumers to re-derive model costs they cannot actually reconstruct. Add "modelBreakdowns" to agent_summary_json, mirroring the unified serializers (summary_json / session_summary_json). Purely additive: every pre-existing key and value is unchanged; the codex-native serializer (models object) is deliberately untouched. Tests: shared-shape insta snapshot now shows populated breakdowns for all four report kinds; pi daily JSON asserts a full single-element breakdown array with non-zero cost (the motivating case); fixture- driven copilot (real pricing via read_otel_file) and qwen (real JSONL fixture line) assertions pin their entire breakdown arrays. * fix(pi): align unified session date filtering (ccusage#1394) * fix(pi): align unified session date filtering Filter default pi unified session entries by date before summarizing, matching `ccusage pi session --pi-path` behavior for inclusive `--until` days. * style: apply treefmt formatting --------- Co-authored-by: ryoppippi <[email protected]> * feat(unified): --sections and --by-agent for single-invocation reporting (ccusage#1396) * feat(unified): --sections and --by-agent for single-invocation reporting Dashboards polling ccusage today need one unified invocation per section plus one per-agent invocation per agent — every call re-scanning all stores. Two additive flags on the unified commands (and the bare root invocation) collapse that to a single call: ccusage daily --json --sections daily,monthly,session --by-agent --sections <csv> emits each requested grouping in one envelope from at most TWO store scans (daily/weekly/monthly share one Daily-kind base load; session adds one Session-kind load), one process, one pricing load. Every section is produced by exactly the code path its standalone command uses — load_sections delegates to the same load_rows machinery, so section output is identical to a standalone invocation by construction (covered by fixture equivalence tests including claude agent-progress usage lines and codex cross-session/model-alias dedupe cases). Envelope order is deterministic via a local ordered serializer: invoked section first, remaining sections in canonical order, totals last; single-section envelopes use the unchanged existing path. --by-agent adds an "agents" array to daily/weekly/monthly rows (the internal per-agent breakdowns, now serialized: tokens, cost, and modelBreakdowns per agent). Session rows are already per-agent, so the flag is a no-op there. Per-agent costs sum exactly to the combined row. Backward compatibility: without the new flags, JSON and table output are byte-identical to before (verified against a 20-invocation golden matrix on real stores). Tables render requested sections sequentially; --by-agent is JSON-only. * refactor(unified): address review feedback on duplication and detected agents - row_json now composes agent_json and layers on the row-level fields (period, metadata, agents), so the shared row shape has a single serialization path; output unchanged. - Extract parse_unified_report_arg so the root, unified-command, and top-level-session parse sites share one --all/--sections/--by-agent block; the root site keeps its mark_used bookkeeping. - Carry daily-load and session-load detected agents separately so each --sections table header shows the same detected list as the equivalent standalone invocation. * style: apply treefmt formatting --------- Co-authored-by: ryoppippi <[email protected]> * feat(pi): named pi-format stores as config-declared agents (ccusage#1397) * feat(pi): named pi-format stores as config-declared agents Tools built on pi (oh-my-pi and other forks) keep pi-format session stores at their own paths. ccusage could only read one pi path universe and labeled everything it found as agent "pi". Declare named extra stores in the config file: { "pi": { "stores": [ { "name": "omp", "path": "~/.omp/agent/sessions" } ] } } Each named store loads through the existing pi parser and surfaces as its OWN agent in the unified reports: rows tagged in metadata.agents, sessions with projectPath/lastActivity like pi, model labels prefixed "[<name>] ". Named stores are additive to the default pi store and use the same path-list semantics (comma-separated, ~-expansion, dedupe) and the same date-window filtering as `ccusage pi ... --pi-path`. Costs are computed from the unprefixed model name — the configurable store name never participates in pricing lookup (a store named "o3" cannot fabricate o3 pricing; regression-tested), while prefixed pricingOverrides keys are consulted first and keep working. Config validation: names match ^[a-z][a-z0-9_-]{0,31}$, reject collisions with built-in agents (single source of truth asserted against the unified loader's registry), duplicates, empty paths, and stores whose resolved paths overlap the default pi store or another store (silent double-counting is never possible). Invalid stores error through the same config-error path as other invalid config content. Absent store paths yield clean empty results, like default pi. Backward compatibility: without pi.stores configured, all output is byte-identical to before (verified against a golden matrix on real stores, including a known pre-existing until-day session-window quirk in the default pi unified path, deliberately preserved here and fixed in a separate patch). Committed config schema regenerated. No CLI surface changes: per-agent subcommands remain a closed set; named stores appear in unified reports only. * fix(pi): reject nested/partial named-store path overlaps, dedupe path parsing - Session files are collected recursively, so a named store rooted at an ancestor or descendant of the default pi store (or another named store) would ingest the same files twice under different dedupe identities. The resolver now rejects any overlap — equal, ancestor, or descendant — and partial collisions error instead of silently dropping the colliding path, matching the documented contract. Regression tests for a store nested inside the default pi path and a partial overlap across two stores. - Extract a shared existing_paths helper in pi/paths.rs; the default and named-store variants now differ only in their path mapper, with the deliberate ~-expansion difference documented. - Update config/pi docs for the stricter overlap wording. * docs(pi): move trailing space out of code spans (markdownlint MD038) --------- Co-authored-by: ryoppippi <[email protected]> * fix(kimi): support Kimi Code new wire format (ccusage#1362) Kimi Code (`~/.kimi-code`) emits a new `wire.jsonl` schema that the old adapter could not parse, so its usage was silently dropped (ccusage#1261). - Detect `~/.kimi-code` and the deeper layout `sessions/<ws>/<session>/agents/<agent>/wire.jsonl` (5 path components) alongside the legacy 3-component layout. - Parse top-level `type == "usage.record"` lines: camelCase token fields (`inputOther`, `inputCacheRead`, `inputCacheCreation`), `time` in milliseconds, and `model` prefixed with `kimi-code/` (stripped for pricing lookup). Skip cumulative `usageScope == "session"` records. - Deserialize `time` leniently so a float- or string-encoded timestamp degrades to the file-mtime fallback instead of dropping the whole line. - Walk the correct number of parents in `kimi_root_from_wire_path` for the deeper layout so config resolution looks at the right root. - Keep full backward compatibility with the old StatusUpdate format. - Update the Kimi guide and data-source docs for `~/.kimi-code`. Fixes ccusage#1261 Co-authored-by: Claude Opus 4.8 <[email protected]> * perf: cache PricingMap::find() results and skip redundant opencode pricing checks (ccusage#1407) * perf(pricing): cache PricingMap::find() results to avoid repeated fuzzy matching PricingMap::find() does an exact HashMap lookup followed by expensive fuzzy matching through all ~2,200 pricing entries when the exact model name is not in the map. When adapters repeatedly query the same model names, a large fraction of lookups miss the HashMap and trigger a full scan of the pricing table for every call. Add a OnceLock<Mutex<FxHashMap>> cache that memoizes find() results by model name (including None for models not found in pricing). Once a model name has been resolved, future lookups complete in O(1) instead of O(n) over the pricing table. Also add clear_find_cache() called from load_json_with_overrides(), load_models_dev_models(), and apply_overrides() so the cache stays consistent when the pricing table is mutated. * perf(opencode): skip redundant missing-pricing check when cost is known calculate_open_code_cost and missing_open_code_pricing independently iterate through the same model candidates. When the cost calculation already found a valid positive cost (either from a stored cost_usd field or from pricing lookup), skip the missing-pricing check entirely since pricing was already resolved. --------- Co-authored-by: turtton <[email protected]> * ci(release): migrate from bumpp to tagpr (ccusage#1406) * ci(release): add tagpr release PR automation Introduce Songmu/tagpr to manage releases via an auto-generated release PR: every push to main creates or updates a PR that bumps all nine workspace package.json versions (tagpr versionFile) and syncs the Rust workspace via the new `just sync-rust-version` recipe run as postVersionCommand. Merging the PR tags the merge commit. GitHub Release creation and CHANGELOG.md generation are disabled in .tagpr because changelogithub keeps generating the release notes in the existing style. Tags pushed with GITHUB_TOKEN do not trigger `on: push: tags` workflows, so tagpr.yaml dispatches release.yaml explicitly with `gh workflow run --ref <tag>`; release.yaml gains a workflow_dispatch trigger for that purpose. * chore(release): drop bumpp local release flow Releases are now driven by tagpr in CI, so the local `just release` recipe and the bumpp dependency are no longer needed. bump.config.ts is deleted because its cargo set-version hook moved to the `just sync-rust-version` recipe that tagpr runs as postVersionCommand. * docs(skills): document tagpr release flow Replace the removed `just release` recipe in the development skill command list with a note on the tagpr release PR flow and the minor/major bump labels. * ci(release): gate release jobs to tag refs and isolate actions:write Gate release.yaml build/publish/release jobs behind startsWith(github.ref, 'refs/tags/') so a workflow_dispatch from a branch cannot bypass the tag-only release flow. tagpr dispatches with --ref <tag>, so the intended path is unaffected. Move the release dispatch out of the tagpr job into a dependent dispatch-release job that alone holds actions: write, keeping tagpr on its documented least-privilege scopes (contents/pull-requests/issues). Co-authored-by: Codesmith <[email protected]> * ci(release): consolidate release pipeline into tagpr workflow Move the build/publish/release jobs from release.yaml into tagpr.yaml, gated on the tagpr job's tag output, and delete release.yaml. Running everything in one workflow removes the workflow_dispatch chaining that worked around GITHUB_TOKEN-pushed tags not triggering `push: tags` workflows, along with the dispatch-release job and its `actions: write` grant. The release jobs check out the freshly created tag explicitly. changelogithub resolves the release tag with `git tag --points-at HEAD`, not GITHUB_REF, so it picks the right release even though the run's ref is refs/heads/main. A failed release is retried with "Re-run failed jobs"; a full re-run finds no new tag and skips the release jobs. --------- Co-authored-by: Codesmith <[email protected]> * ci(release): use Conventional Commits title for tagpr release PRs (ccusage#1409) tagpr titles its release PRs "Release for vX.Y.Z", which fails the check-pr-title workflow because it has no Conventional Commits type prefix (seen on PR ccusage#1408). tagpr takes the first line of the rendered pull request template as the PR title, so point .tagpr at a custom template whose first line is "chore: release {{.NextVersion}}". The rest of the template mirrors tagpr's default body, minus the unused tag-prefix placeholder. The template is a Go text/template, and oxfmt's markdown rewrites break its <details> block and nested list structure, so exclude it from treefmt. This also restores the title style used by the previous bumpp-based release flow ("chore: release v20.0.14"). * feat(pricing): support OpenAI two-stage pricing and add the gpt-5.6 family (ccusage#1414) * feat(pricing): add gpt-5.6 family and OpenAI long-context tier rates OpenAI introduced two-stage (short/long context) pricing with gpt-5.6: requests with more than 272K input tokens are billed at higher long-context rates. The same tier also applies to gpt-5.5, gpt-5.5-pro, gpt-5.4, and gpt-5.4-pro on the current pricing page. The existing tier support hardcoded the LiteLLM 200K boundary, so Pricing gains a per-model long_context_threshold (defaulting to 200K for LiteLLM *_above_200k_tokens data) and tiered_cost takes the threshold as a parameter. New built-in entries cover gpt-5.6-sol, gpt-5.6-terra, and gpt-5.6-luna, including their cache-write rates. Long-context tier rates live in a builtin_long_context_rates overlay that is re-applied after every pricing load: a live LiteLLM refresh replaces whole entries, and LiteLLM currently publishes these models with flat rates only, so tier rates set directly on built-in entries would be silently dropped whenever a refresh succeeds. Entries that already carry tier rates are left untouched so upstream data wins once it exists. Date-pinned keys such as gpt-5.5-2026-04-23 share their base model's overlay rates. The gpt-5.6 context limits mirror the 1,050,000-token window of the other long-context GPT-5 flagship models until upstream data lands. * feat(codex): bill long-context requests at OpenAI two-stage rates OpenAI decides the pricing tier per request: once a request's input exceeds 272K tokens, every token of that request (input, cached input, and output) is billed at the long-context rates. Codex cost calculation runs on per-model sums aggregated across many requests, so the tier cannot be recovered from the totals afterwards. CodexModelUsage now tracks the portion of tokens that came from long-context requests. The split is recorded while token_count events are aggregated, where each event still represents a single request, and merged across parallel shards like the other counters. calculate_codex_model_cost prices the aggregated usage as two independent buckets: the short bucket at the flat rates and the long bucket at the *_above_200k rates, falling back to the flat rates for models without a long-context tier so their costs are unchanged. The existing fast-speed multiplier applies to both buckets. Report JSON and table output are unchanged; only costUSD values for long-context requests differ. * docs(pricing): explain all-or-nothing long-context overlay check Codex review suggested filling missing tier fields independently when a refreshed LiteLLM entry carries partial *_above_200k_tokens data. That would mix rates that assume the 200K LiteLLM boundary with built-in rates that assume the OpenAI 272K boundary under a single per-model threshold, mispricing both tiers, so the overlay defers to upstream entirely once any tier rate exists. Record that rationale next to the check. * fix(pricing): apply two-stage rates to whole request and per-model split Co-authored-by: Codesmith <[email protected]> --------- Co-authored-by: Codesmith <[email protected]> * chore: use black smith more * chore: release v20.0.15 (ccusage#1408) [tagpr] prepare for the next release Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * chore(ci): rename it back to releaese.yaml * chore: release v20.0.16 (ccusage#1416) [tagpr] prepare for the next release Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * docs: update Lineman affiliate links to CCUsage landing page (ccusage#1417) Point GitHub README and docs site sponsor links at the dedicated LinkJolt redirect for CCUsage traffic (free tier + voucher funnel). Co-authored-by: Cursor Agent <[email protected]> Co-authored-by: ryoppippi <[email protected]> * docs: update Star History chart (ccusage#1419) * docs: update Star History chart Switch the README and sponsorship guide to the current Star History chart endpoint, including light and dark variants. Allowlist the public read-only sealed chart token so secret scanning does not report a false positive. Co-authored-by: ryoppippi <[email protected]> * chore: exclude sealed token from spellcheck Mark the exact public Star History token allowlist line as a spellcheck exclusion so its random character sequence does not fail the documentation preflight. Co-authored-by: ryoppippi <[email protected]> * chore: format sealed token allowlist Use the repository's TOML formatting and bracket the random token with the supported spellchecker block directives. Co-authored-by: ryoppippi <[email protected]> * style: align Gitleaks TOML indentation Match the repository formatter's tab indentation for the multiline allowlist entry. Co-authored-by: ryoppippi <[email protected]> * fix: match full Star History token URL Configure the global Gitleaks allowlist to evaluate the full finding match so the narrowly scoped sealed_token pattern suppresses the six intentional chart URLs. Co-authored-by: ryoppippi <[email protected]> --------- Co-authored-by: Cursor Agent <[email protected]> Co-authored-by: ryoppippi <[email protected]> * fix(claude): count advisor model usage (ccusage#1423) * fix(claude): count advisor model usage Expand advisor_message iterations into distinct usage entries so their tokens and model-specific costs are included in every report path. Keep main-model iteration totals unchanged and cover both standard and daily loaders. Co-authored-by: ryoppippi <[email protected]> * docs(claude): clarify advisor cost modes Co-authored-by: ryoppippi <[email protected]> --------- Co-authored-by: Cursor Agent <[email protected]> Co-authored-by: ryoppippi <[email protected]> * chore: release v20.0.17 (ccusage#1418) [tagpr] prepare for the next release Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * perf(nix): keep dependency cache across releases (ccusage#1424) * build(perf): migrate benchmark harness to Babashka (ccusage#1432) * build(perf): migrate benchmark harness to Babashka Replace the large Nushell PR benchmark script with a Babashka implementation split by data, system, benchmark, report, and orchestration responsibilities. The new process boundary keeps argv, environment, and working-directory data explicit while preserving hyperfine, package installation, memory, size, and Markdown behavior. Move the CI caller and profiling guidance to the executable Babashka entry point. Add focused tests behind their own Nix shebang so contributors can run the harness suite without adding Babashka to the full development shell. * docs(agents): document implementation language choices Route small command-oriented automation to Nushell and data-heavy, testable automation to Babashka. Keep production binaries in Rust and npm-integrated APIs in TypeScript so future tooling changes follow the same criteria used by the benchmark migration. * test(ci): run Babashka harness tests Execute the self-contained benchmark harness test entry point in the CI test job so changes to CLI parsing, normalization, fallback decisions, and report rendering cannot bypass pull request validation. * fix(perf): harden platform and tarball paths Normalize version-qualified Windows os.name values to win32 so native executable and package paths use the expected suffixes. Resolve relative pnpm pack filenames against the temporary destination while preserving the absolute paths emitted by current pnpm versions. Add regression coverage for both platform normalization and relative or absolute tarball filenames. * fix(perf): size local package fallbacks Use remote tarball sizing only after the corresponding preview package was installed successfully. When either package URL times out, benchmark and size the available local checkout so fallback runs can still produce a complete report. Cover base and head source selection and verify both unavailable URLs through a committed-fixture smoke run. * fix(perf): bound harness child processes and skip RSS on unsupported platforms Add a cancellable timeout to run-process and thread --package-runner-timeout-ms through the package URL probe, install, pnpm pack, and git rev-parse flows so a stalled child cannot outlive the deadline; give the curl probe and download explicit connect and read limits. measure-memory now warns once and skips gracefully when /usr/bin/time is unavailable (unsupported platforms) instead of throwing and aborting the entire benchmark run. Co-authored-by: Codesmith <[email protected]> * Revert "fix(perf): bound harness child processes and skip RSS on unsupported platforms" This reverts commit 9140a99. --------- Co-authored-by: Codesmith <[email protected]> * build(perf): migrate fixture generator to Bun (ccusage#1433) * build(perf): migrate fixture generator to Bun Replace the Nushell fixture generator with a dependency-free Bun script.\n\nKeep the generated Claude and Codex fixture layouts and command-line\ninterface while using Bun file writers and Bun Shell for file operations. * build(perf): type Bun fixture script Add Bun development types so the fixture generator is checked alongside the package tooling.\n\nAwait file writer operations to preserve ordered writes and satisfy the\nrepository promise lint rule. * build(perf): avoid Bun type dependency Keep the fixture generator dependency-free by declaring its small Bun API surface locally.\n\nRemove the Bun type package and restore the package TypeScript configuration so\npublishing the fixture generator does not expand package dependencies. * chroe(ci): fix nix cache * ci: add GitHub-hosted runner fallback Keep Blacksmith runners for the upstream repository while allowing forks\nto use hosted runners by default. Forks with a Blacksmith subscription can\nopt in through HAS_BLACKSMITH=true. * ci: skip pkg-pr previews without the GitHub App Forks do not inherit the pkg-pr-new GitHub App installation. Skip\npreview publishing and its dependent E2E and performance jobs unless a fork\nexplicitly opts in with HAS_PKG_PR_NEW=true. * ci: keep Windows arm release runner defined Supply both matrix runner fields so the release workflow resolves its\nWindows ARM runner consistently with the other native package targets. --------- Co-authored-by: ryoppippi <[email protected]> Co-authored-by: pullfrog[bot] <226033991+pullfrog[bot]@users.noreply.github.com> Co-authored-by: sijie-ni-0214 <[email protected]> Co-authored-by: Ben Vargas <[email protected]> Co-authored-by: Mint Choco <[email protected]> Co-authored-by: Claude Opus 4.8 <[email protected]> Co-authored-by: turtton <[email protected]> Co-authored-by: Codesmith <[email protected]> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Cursor Agent <[email protected]> Co-authored-by: ryoppippi <[email protected]> Co-authored-by: axisrow <[email protected]>
Summary
Migrate the PR performance benchmark harness from Nushell to Babashka after parity testing showed the data-heavy reporting flow was substantially smaller and easier to test. The implementation is split into focused modules under
compare_pr_performance/, while executable entry points keep their Nix dependencies self-contained.What Changed
Testing
just ccusage::test-performance-harnessapps/ccusage/scripts/compare-pr-performance.bb --helpjust typecheckjust testjust checkNeed help on this PR? Tag
/codesmithwith what you need. Autofix is enabled.Summary by cubic
Migrated the PR performance benchmark harness from Nushell to Babashka. CI and docs now use the
.bbentry, run the harness tests, and reports stay complete via local sizing fallbacks when preview packages aren’t ready.Refactors
Bug Fixes
os.nametowin32so native and package paths resolve correctly.pnpm packtarball filenames against the temp destination; preserved absolute paths.Written for commit 6c90fc2. Summary will update on new commits.
Summary by CodeRabbit
justfilerecipe to run the performance harness.