Skip to content

tools: add benchmark reporting utilities - #715

Draft
skku970412 wants to merge 4 commits into
ChampSim:masterfrom
skku970412:codex/benchmark-report-tools
Draft

skku970412 wants to merge 4 commits into
ChampSim:masterfrom
skku970412:codex/benchmark-report-tools

Conversation

@skku970412

Copy link
Copy Markdown

Summary

  • Add a Catch2 XML benchmark normalizer that writes stable JSON and Markdown reports.
  • Add a baseline/current benchmark comparison tool with JSON and Markdown output.
  • Add a manual, artifact-only benchmark report workflow.
  • Document the local and CI benchmark reporting flow.

Motivation

Issue #590 asks for benchmark history tracking and PR performance drift visibility. This first pass focuses on durable, machine-readable benchmark artifacts without changing simulator behavior or adding PR comments.

What changed

  • Added tools/bench/normalize_catch2_benchmarks.py.
  • Added tools/bench/compare_benchmark_results.py.
  • Added fixture-based Python tests for valid, malformed, duplicate, unit conversion, threshold, zero-baseline, Markdown, and JSON edge cases.
  • Added .github/workflows/benchmark-report.yml as a manual workflow with read-only permissions and uploaded artifacts.
  • Added workflow diagnostics for Catch2 reporters, test discovery, benchmark declaration discovery, generated XML validation, and strict artifact upload checks.
  • Added docs/src/Benchmark-reporting.rst.

Out of scope

  • No simulator behavior changes.
  • No automatic PR comments.
  • No benchmark history publishing.
  • No default CI gating on benchmark drift.

Verification

  • git diff --check
  • python3 -m compileall tools/bench test/python
  • python3 -m unittest discover -v --start-directory test/python passed 270 tests with 1 skipped test
  • ./config.sh
  • make -j"$(nproc)" test/bin/000-test-main
  • test/bin/000-test-main --list-reporters
  • test/bin/000-test-main --list-tests
  • grep -R "BENCHMARK" -n test/cpp src inc || true
  • Generated Catch2 XML contained BenchmarkResults
  • Catch2 XML benchmark output normalized successfully from a real XML run with 9 benchmarks
  • Self-compare with --fail-on-regression passed with 9 pass, 0 warn, and 0 fail
  • actionlint was not available locally
  • GitHub Actions workflow execution is not yet proven in CI. A fork-only push-trigger verification branch created a run, but GitHub did not start the job because the fork account is locked due to a billing issue.

Follow-ups

  • Add optional PR comments after maintainers choose a comment update policy.
  • Add benchmark history publishing after maintainers choose storage and retention.
  • Decide whether drift thresholds should ever fail CI.

@ngober

ngober commented Jul 7, 2026

Copy link
Copy Markdown
Collaborator

This is an interesting patch. It might do what the Issue is asking for. I have a couple of initial concerns:

  • Putting all of this code in the ChampSim repository means we need to maintain it over time. The principle of separation of concerns would suggest that this might be better as a standalone Github Action that we link with the uses key. But, then, why would this be better than an existing solution (such as the one I linked in the issue)? It may reinvent the wheel in that regard.
  • Why did you choose to convert the XML files to JSON, only to store the JSON files and compare them. Would it not make more sense to store the XML files and ingest them each time?

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants