Skip to content

perf(ci): try Intel runners for non-native jobs - #1305

Merged
ryoppippi merged 2 commits into
mainfrom
codex/compare-intel-ci-runners
Jun 12, 2026
Merged

perf(ci): try Intel runners for non-native jobs#1305
ryoppippi merged 2 commits into
mainfrom
codex/compare-intel-ci-runners

Conversation

@ryoppippi

@ryoppippi ryoppippi commented Jun 12, 2026

Copy link
Copy Markdown
Member

Switches CI jobs that do not build native packages from the ARM Blacksmith Ubuntu runner to the matching Intel runner for a timing comparison.

The native package build matrix and Linux arm64 preview E2E coverage remain unchanged, so architecture coverage is preserved while check, test, packaging, and perf-comment jobs run on Intel.

A separate formatter-only commit is included because the repository pre-push hook required current treefmt output before publishing the branch.

Testing:

  • git diff --check
  • nix develop --command just fmt
  • pre-push hook: clippy, treefmt, gitleaks, cargo test

View with Codesmith
Need help on this PR? Tag /codesmith with what you need. Autofix is enabled.


Summary by cubic

Switch non-native CI jobs from the ARM Blacksmith Ubuntu runner to the equivalent Intel runner to compare queue and wall-clock time. Native package builds and Linux arm64 preview E2E stay on ARM, so architecture coverage is unchanged.

  • Refactors
    • Moved check, test, packaging/publish, and performance-comment jobs to blacksmith-32vcpu-ubuntu-2404.
    • Applied treefmt to the Codex loader test file; formatting-only change.

Written for commit 43fc7cb. Summary will update on new commits.

Review in cubic

Summary by CodeRabbit

  • Chores

    • Updated CI workflow runner configurations.
  • Tests

    • Adjusted test assertion formatting and comment wording.

Switch the CI jobs that do not build native packages from the ARM Blacksmith Ubuntu runner to the matching Intel Ubuntu runner.

This keeps the native package matrix and Linux arm64 E2E coverage unchanged, so the PR can compare queue and wall-clock time for check, test, packaging, and performance-comment jobs without reducing architecture coverage.
Apply the repository formatter output required by the pre-push hook.

This keeps the formatter-only Rust test formatting separate from the CI runner experiment so the runner change remains independently reviewable.
@coderabbitai

coderabbitai Bot commented Jun 12, 2026

Copy link
Copy Markdown

Review Change Stack

📝 Walkthrough

Walkthrough

This PR migrates CI runner architecture from ARM-specific Blacksmith runners to standard Ubuntu 2404 across four jobs, and reformats a test assertion and comment in the Rust codebase without changing underlying logic.

Changes

CI Runner Architecture Migration

Layer / File(s) Summary
Runner platform migration
.github/workflows/ci.yaml
check, test, npm-publish-dry-run-and-upload-pkg-pr-now, and ccusage-perf-comment jobs all switched from blacksmith-32vcpu-ubuntu-2404-arm to blacksmith-32vcpu-ubuntu-2404 runner.

Test Code Cleanup

Layer / File(s) Summary
Test comment and assertion formatting
rust/crates/ccusage/src/adapter/codex/loader.rs
Comment rewording for unparsable timestamp fields; test assertion restructured from inline chain to line-broken closure with all(...)/is_some_and(...) while preserving boolean logic.

Estimated code review effort

🎯 1 (Trivial) | ⏱️ ~3 minutes

Possibly related PRs

  • ccusage/ccusage#1303: Updates the same Codex loader unit test around invalid event-date fallback logic and model-resolution validation in loader.rs.
  • ccusage/ccusage#1268: Modifies the same CI workflow jobs in test and related areas; switches runner architecture and updates test/coverage step logic.
  • ccusage/ccusage#1258: Updates the same four CI jobs (check, test, npm-publish-dry-run-and-upload-pkg-pr-now, ccusage-perf-comment) but changes job steps instead of runner platform.

Suggested reviewers

  • pullfrog

Poem

🐰 From ARM to standard grounds we hop,
With runners switched and tests retop,
No logic changed, just cleaner code,
On Ubuntu's broader, sunnier road! ✨

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Title check ✅ Passed The title 'perf(ci): try Intel runners for non-native jobs' accurately describes the main change: moving non-native CI jobs from ARM runners to Intel runners for performance comparison.
Docstring Coverage ✅ Passed Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch codex/compare-intel-ci-runners

Warning

Review ran into problems

🔥 Problems

Stopped waiting for pipeline failures after 30000ms. One of your pipelines takes longer than our 30000ms fetch window to run, so review may not consider pipeline-failure results for inline comments if any failures occurred after the fetch window. Increase the timeout if you want to wait longer or run a @coderabbit review after the pipeline has finished.


Comment @coderabbitai help to get the list of available commands and usage tips.

@ryoppippi

Copy link
Copy Markdown
Member Author

@coderabbitai review
@cubic-dev-ai review

@cubic-dev-ai

cubic-dev-ai Bot commented Jun 12, 2026

Copy link
Copy Markdown
Contributor

@coderabbitai review
@cubic-dev-ai review

@ryoppippi I have started the AI code review. It will take a few minutes to complete.

@coderabbitai

coderabbitai Bot commented Jun 12, 2026

Copy link
Copy Markdown

Tip

For best results, initiate chat on the files or code changes.

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# Get the PR diff
gh pr view 1305 --repo ccusage/ccusage --json files,title,body,additions,deletions,changedFiles

echo "=== PR DIFF ==="
gh pr diff 1305 --repo ccusage/ccusage

Length of output: 5813



Walkthrough

Two files changed, no logic modifications.

.github/workflows/ci.yaml

Four jobs switched from blacksmith-32vcpu-ubuntu-2404-armblacksmith-32vcpu-ubuntu-2404:

Job Purpose
check Clippy / lint
test Unit / integration tests
packaging/publish (~line 131) Preview publish
perf-comment (~line 313) Benchmark comment

The native package build matrix and the Linux arm64 preview E2E job are not touched, so architecture coverage is fully preserved. The change is consistent across all four affected jobs and reads as a clean A/B timing experiment.

rust/crates/ccusage/src/adapter/codex/loader.rs

  • Typo fix: "unparseable""unparsable" in a comment — minor but correct; unparsable is the standard technical term.
  • Reformatting: The chained assert!(events.iter().all(…)) call was reflowed by treefmt; no semantic change.

Overall: The PR is clean and well-scoped. No issues found. ✅

[approve]

@cloudflare-workers-and-pages

Copy link
Copy Markdown

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Preview URL Updated (UTC)
✅ Deployment successful!
View logs
ccusage-guide 43fc7cb Commit Preview URL

Branch Preview URL
Jun 12 2026, 04:05 PM

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No new issues found.

Reviewed changes — This PR switches 4 non-native CI jobs from ARM to Intel runners (blacksmith-32vcpu-ubuntu-2404-armblacksmith-32vcpu-ubuntu-2404) for a timing comparison, plus a treefmt formatting-only commit on the Codex loader test file.

  • Switch check/test/publish/perf-comment to Intel — Mechanical runs-on label change across 4 jobs that don't produce native artifacts
  • Apply treefmt to Codex loader test — Comment spelling fix and method-chain reformatting; no semantic change

Pullfrog  | View workflow run | Using Big Pickle (free via Pullfrog for OSS) | 𝕏

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No issues found across 2 files

Re-trigger cubic

@pkg-pr-new

pkg-pr-new Bot commented Jun 12, 2026

Copy link
Copy Markdown

Open in StackBlitz

ccusage

npx https://pkg.pr.new/ccusage@1305

@ccusage/ccusage-darwin-arm64

npx https://pkg.pr.new/@ccusage/ccusage-darwin-arm64@1305

@ccusage/ccusage-darwin-x64

npx https://pkg.pr.new/@ccusage/ccusage-darwin-x64@1305

@ccusage/ccusage-linux-arm64

npx https://pkg.pr.new/@ccusage/ccusage-linux-arm64@1305

@ccusage/ccusage-linux-x64

npx https://pkg.pr.new/@ccusage/ccusage-linux-x64@1305

@ccusage/ccusage-win32-x64

npx https://pkg.pr.new/@ccusage/ccusage-win32-x64@1305

commit: 43fc7cb

@github-actions

Copy link
Copy Markdown
Contributor

ccusage performance comparison

PR SHA: 43fc7cb75763
Base SHA: bfd28e031e0d

This compares the PR package against the configured base package on the same CI runner.

Package runner startup

Execution setup measures any pre-benchmark package materialization used by the execution benchmark. Bunx temp cache measures one bunx -p <url> ccusage --version run with an empty Bun install cache. Warm reuses that cache and reports the median of repeated runs.

Package SHA Execution setup Bunx temp cache Bunx warm median Warm samples
Base pkg.pr.new bfd28e031e0d 870.4ms 822.1ms 24.0ms 3
PR pkg.pr.new 43fc7cb 602.4ms 1.026s 22.7ms 3

Cached bunx execution performance

Runs the same large fixture through bunx -p <pkg.pr.new URL> ccusage after the Bun install cache has already been populated by the startup measurement. This separates cached package-runner execution from first-fetch package materialization.

Fixtures: Claude /home/runner/_work/_temp/ccusage-large-fixture (1.01 GiB, 2,597 files), Codex /home/runner/_work/_temp/ccusage-large-codex-fixture (1.01 GiB, 2,597 files)
Base package: bfd28e031e0d; PR package: 43fc7cb. Both run through bunx -p <pkg.pr.new URL> ccusage using the warmed Bun install cache from package runner startup, measured by hyperfine with 0 warmups and 1 runs.
Peak RSS is measured separately with /usr/bin/time using 1 runs. Lower RSS ratios are better.

Command Input Base median PR median PR vs base Base peak RSS PR peak RSS PR/base RSS Base throughput PR throughput
bunx -p <pkg> ccusage claude --offline --json 1.01 GiB 243.1ms 252.0ms 0.96x 740.00 MiB 735.00 MiB 0.99x 4.14 GiB/s 3.99 GiB/s
bunx -p <pkg> ccusage codex --offline --json 1.01 GiB 85.8ms 91.1ms 0.94x 89.50 MiB 89.50 MiB 1.00x 11.74 GiB/s 11.05 GiB/s

Package runtime diagnostics

Compares the PR package wrapper, the installed native optional dependency binary, and the workspace release binary on the same large fixture. This identifies whether slow package results come from JavaScript wrapper overhead, the published native binary build, or the Rust core itself.

Fixtures: Claude /home/runner/_work/_temp/ccusage-large-fixture (1.01 GiB, 2,597 files), Codex /home/runner/_work/_temp/ccusage-large-codex-fixture (1.01 GiB, 2,597 files)
All rows run --offline --json, measured by hyperfine with 0 warmups and 1 runs. This isolates wrapper overhead from the installed native optional dependency and the workspace release binary built on the runner.

Command Runtime Input Median Throughput Samples
claude --offline --json Package wrapper 1.01 GiB 241.8ms 4.16 GiB/s 1
claude --offline --json Installed native binary 1.01 GiB 241.6ms 4.17 GiB/s 1
codex --offline --json Package wrapper 1.01 GiB 86.9ms 11.58 GiB/s 1
codex --offline --json Installed native binary 1.01 GiB 72.8ms 13.83 GiB/s 1

Committed fixture performance

Committed small fixtures for stable PR-to-PR feedback and explicit Claude/Codex command coverage.

Fixtures: Claude apps/ccusage/test/fixtures/claude (0.00 MiB, 2 files), Codex apps/ccusage/test/fixtures/codex (0.00 MiB, 1 files)
Base runs the published ccusage package from pkg.pr.new, installed before measurement; PR runs the published ccusage package from pkg.pr.new, installed before measurement. Both run --offline --json, measured by hyperfine with 2 warmups and 7 runs.
Peak RSS is measured separately with /usr/bin/time using 1 runs. Lower RSS ratios are better.

Command Input Base median PR median PR vs base Base peak RSS PR peak RSS PR/base RSS Base throughput PR throughput
claude daily --offline --json 0.00 MiB 17.4ms 14.9ms 1.16x 43.25 MiB 42.75 MiB 0.99x 0.09 MiB/s 0.10 MiB/s
claude session --offline --json 0.00 MiB 15.5ms 14.7ms 1.05x 43.00 MiB 42.75 MiB 0.99x 0.10 MiB/s 0.10 MiB/s
codex daily --offline --json 0.00 MiB 14.7ms 14.5ms 1.01x 43.00 MiB 42.75 MiB 0.99x 0.06 MiB/s 0.06 MiB/s
codex session --offline --json 0.00 MiB 14.6ms 14.8ms 0.99x 43.00 MiB 43.00 MiB 1.00x 0.06 MiB/s 0.06 MiB/s

Large real-world-shaped fixture performance

Generated fixtures shaped from aggregate local log statistics: thousands of JSONL files, many small sessions, and a long tail of larger sessions. No real prompts, paths, or outputs are stored in the fixtures.

Fixtures: Claude /home/runner/_work/_temp/ccusage-large-fixture (1.01 GiB, 2,597 files), Codex /home/runner/_work/_temp/ccusage-large-codex-fixture (1.01 GiB, 2,597 files)
Base runs the published ccusage package from pkg.pr.new, installed before measurement; PR runs the published ccusage package from pkg.pr.new, installed before measurement. Both run --offline --json, measured by hyperfine with 0 warmups and 1 runs.
Peak RSS is measured separately with /usr/bin/time using 1 runs. Lower RSS ratios are better.

Command Input Base median PR median PR vs base Base peak RSS PR peak RSS PR/base RSS Base throughput PR throughput
claude --offline --json 1.01 GiB 240.1ms 249.7ms 0.96x 734.25 MiB 740.50 MiB 1.01x 4.19 GiB/s 4.03 GiB/s
codex --offline --json 1.01 GiB 73.9ms 77.8ms 0.95x 89.00 MiB 91.50 MiB 1.03x 13.63 GiB/s 12.94 GiB/s

Artifact size

Artifact Base PR Delta Ratio
packed ccusage-*.tgz 17.32 KiB 17.32 KiB +0.00 KiB 1.00x
installed native package binary 3968.69 KiB 3968.69 KiB +0.00 KiB 1.00x

Lower medians and smaller artifacts are better. CI runner noise still applies; use same-run ratios as directional PR feedback, not release guarantees.

@github-actions

Copy link
Copy Markdown
Contributor

ccusage performance comparison

PR SHA: 43fc7cb75763
Base SHA: bfd28e031e0d

This compares the Rust PR release binary against the configured base package on the same CI runner.

Package runner startup

Execution setup measures any pre-benchmark package materialization used by the execution benchmark. Bunx temp cache measures one bunx -p <url> ccusage --version run with an empty Bun install cache. Warm reuses that cache and reports the median of repeated runs.

Package SHA Execution setup Bunx temp cache Bunx warm median Warm samples
Base pkg.pr.new bfd28e031e0d 831.1ms 891.7ms 22.5ms 3
PR pkg.pr.new 43fc7cb 895.9ms 990.9ms 23.8ms 3

Cached bunx execution performance

Runs the same large fixture through bunx -p <pkg.pr.new URL> ccusage after the Bun install cache has already been populated by the startup measurement. This separates cached package-runner execution from first-fetch package materialization.

Fixtures: Claude /home/runner/_work/_temp/ccusage-large-fixture (1.01 GiB, 2,597 files), Codex /home/runner/_work/_temp/ccusage-large-codex-fixture (1.01 GiB, 2,597 files)
Base package: bfd28e031e0d; PR package: 43fc7cb. Both run through bunx -p <pkg.pr.new URL> ccusage using the warmed Bun install cache from package runner startup, measured by hyperfine with 0 warmups and 1 runs.
Peak RSS is measured separately with /usr/bin/time using 1 runs. Lower RSS ratios are better.

Command Input Base median PR median PR vs base Base peak RSS PR peak RSS PR/base RSS Base throughput PR throughput
bunx -p <pkg> ccusage claude --offline --json 1.01 GiB 251.7ms 261.5ms 0.96x 731.50 MiB 725.00 MiB 0.99x 4.00 GiB/s 3.85 GiB/s
bunx -p <pkg> ccusage codex --offline --json 1.01 GiB 92.3ms 103.4ms 0.89x 89.25 MiB 91.75 MiB 1.03x 10.91 GiB/s 9.74 GiB/s

Package runtime diagnostics

Compares the PR package wrapper, the installed native optional dependency binary, and the workspace release binary on the same large fixture. This identifies whether slow package results come from JavaScript wrapper overhead, the published native binary build, or the Rust core itself.

Fixtures: Claude /home/runner/_work/_temp/ccusage-large-fixture (1.01 GiB, 2,597 files), Codex /home/runner/_work/_temp/ccusage-large-codex-fixture (1.01 GiB, 2,597 files)
All rows run --offline --json, measured by hyperfine with 0 warmups and 1 runs. This isolates wrapper overhead from the installed native optional dependency and the workspace release binary built on the runner.

Command Runtime Input Median Throughput Samples
claude --offline --json Package wrapper 1.01 GiB 286.0ms 3.52 GiB/s 1
claude --offline --json Installed native binary 1.01 GiB 257.8ms 3.91 GiB/s 1
codex --offline --json Package wrapper 1.01 GiB 75.9ms 13.27 GiB/s 1
codex --offline --json Installed native binary 1.01 GiB 61.4ms 16.41 GiB/s 1

Committed fixture performance

Committed small fixtures for stable PR-to-PR feedback and explicit Claude/Codex command coverage.

Fixtures: Claude apps/ccusage/test/fixtures/claude (0.00 MiB, 2 files), Codex apps/ccusage/test/fixtures/codex (0.00 MiB, 1 files)
Base runs the published ccusage package from pkg.pr.new, installed before measurement; PR runs rust/target/release/ccusage directly. Both run --offline --json, measured by hyperfine with 2 warmups and 7 runs.
Peak RSS is measured separately with /usr/bin/time using 1 runs. Lower RSS ratios are better.

Command Input Base median PR median PR vs base Base peak RSS PR peak RSS PR/base RSS Base throughput PR throughput
claude daily --offline --json 0.00 MiB 16.0ms 2.0ms 8.11x 42.75 MiB 3.00 MiB 0.07x 0.10 MiB/s 0.78 MiB/s
claude session --offline --json 0.00 MiB 14.7ms 2.1ms 6.96x 43.25 MiB 2.75 MiB 0.06x 0.11 MiB/s 0.73 MiB/s
codex daily --offline --json 0.00 MiB 14.8ms 1.9ms 8.00x 43.00 MiB 3.00 MiB 0.07x 0.06 MiB/s 0.46 MiB/s
codex session --offline --json 0.00 MiB 14.5ms 1.8ms 8.15x 43.00 MiB 3.00 MiB 0.07x 0.06 MiB/s 0.48 MiB/s

Large real-world-shaped fixture performance

Generated fixtures shaped from aggregate local log statistics: thousands of JSONL files, many small sessions, and a long tail of larger sessions. No real prompts, paths, or outputs are stored in the fixtures.

Fixtures: Claude /home/runner/_work/_temp/ccusage-large-fixture (1.01 GiB, 2,597 files), Codex /home/runner/_work/_temp/ccusage-large-codex-fixture (1.01 GiB, 2,597 files)
Base runs the published ccusage package from pkg.pr.new, installed before measurement; PR runs rust/target/release/ccusage directly. Both run --offline --json, measured by hyperfine with 0 warmups and 1 runs.
Peak RSS is measured separately with /usr/bin/time using 1 runs. Lower RSS ratios are better.

Command Input Base median PR median PR vs base Base peak RSS PR peak RSS PR/base RSS Base throughput PR throughput
claude --offline --json 1.01 GiB 260.0ms 239.2ms 1.09x 701.75 MiB 718.00 MiB 1.02x 3.87 GiB/s 4.21 GiB/s
codex --offline --json 1.01 GiB 77.5ms 62.7ms 1.24x 94.25 MiB 90.50 MiB 0.96x 12.98 GiB/s 16.06 GiB/s

Artifact size

Artifact Base PR Delta Ratio
packed ccusage-*.tgz 17.32 KiB 17.32 KiB +0.00 KiB 1.00x
installed native package binary 3968.69 KiB 3968.69 KiB +0.00 KiB 1.00x

Lower medians and smaller artifacts are better. CI runner noise still applies; use same-run ratios as directional PR feedback, not release guarantees.

@ryoppippi
ryoppippi merged commit 5f962cd into main Jun 12, 2026
39 checks passed
@ryoppippi
ryoppippi deleted the codex/compare-intel-ci-runners branch June 12, 2026 16:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant