Skip to content

Latest commit

 

History

History
 
 

Folders and files

NameName
Last commit message
Last commit date

parent directory

..
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

README.md

Benchmark harness

Measures the performance of generated Lua and meters how LuaJIT's tracing JIT treats it. Two kinds of output with two different purposes:

  • Wall-clock timings (./bench/run) — local numbers for defending an optimization win. Too noisy for CI.
  • Deterministic LuaJIT counters (./bench/ci) — byte-stable reports that CI diffs against the committed oracles in goldens/, so a codegen change that adds closure allocations or blacklists a loop shows up as a reviewable diff.

Layout

  • micro/ — hand-written Lua pairs: the shape the backend currently generates for a pattern (current) next to the idiomatic Lua it stands for (ideal). These are fixed reference points; they do not change when codegen changes.
  • macro/ — specs driving real linked modules. Each spec names a Bench.* PureScript module (sources in test/ps/src/Bench/, linked by ./bench/link into _build/), how to drive its export hot, and an ideal hand-written equivalent.
  • tools/ — the runners and meters. fnew_census.lua and trace_report.lua require LuaJIT (jit.util, jit.attach); the timing runners work under both PUC Lua and LuaJIT.
  • goldens/ — committed counter reports; the CI oracle.

Usage

./bench/run                 # all wall-clock benchmarks, all runtimes
lua bench/tools/run_micro.lua bench/micro/curried_apply.lua        # one bench
luajit bench/tools/run_macro.lua bench/macro/array_foldl.lua 5e6   # custom n
./bench/ci                  # regenerate counters, verify against goldens/
./bench/ci --accept         # rewrite goldens/ after a deliberate change

Timings use os.clock() — CPU time, not wall time. That is deliberate: the benchmarks are pure computation, and CPU time ignores scheduler noise. It would under-report I/O, so do not reuse the timing helpers for anything that waits. For quieter numbers, pin the process to a core (taskset -c N ./bench/run) and use the performance CPU governor; leave ASLR alone — disabling it trades a real security property for nearly nothing.

What the counters mean

fnew_census statically counts FNEW bytecodes (closure creation) in a linked artifact without running it, split by where the instruction lives: the main chunk runs once at load time, so its FNEWs are init cost; a function body runs per call, so its FNEWs are steady-state allocation — and each one aborts LuaJIT trace recording (NYI: bytecode FNEW), which is what keeps curried hot code interpreted. The census is a pure function of the artifact, hence byte-stable.

trace_report runs a macro spec hot and reports (a) the set of distinct trace-abort sites with reasons and (b) the end state of loop and function-entry bytecodes: LuaJIT rewrites an opcode to its J* form when it installs a trace there and to its I* form when it blacklists the spot. Raw abort counts are not reported: the retry-penalty step that leads to a blacklist draws on an entropy-seeded PRNG, so counts jitter across runs while the abort-site set and the opcode end state are far steadier. Steadier is not identical, though — a spot right at the hot-count boundary can root-trace in one process and not the next — so the report intersects several independent trials and keeps only what every trial agrees on (see below). Blacklisting is never logged by -jv/-jdump; the post-hoc opcode read is the only stable way to observe it.

Both reports record the LuaJIT version (runtime: header line): the abort reasons, the NYI set, and the opcode families are properties of a specific LuaJIT snapshot, so a toolchain bump that moves the counters shows up in the golden diff as an attributable header change, not a mystery regression.

Two caveats about golden stability. The trace reports pin source lines of both the linked artifact and the macro spec file itself, so any edit to bench/macro/*.lua — comments included — legitimately moves the goldens; rerun ./bench/ci --accept and review the diff. And trace formation is not deterministic per process: LuaJIT's hot-counters live in a small hashed table keyed by bytecode address, and the retry penalty draws on an entropy-seeded PRNG, so a spot on the hot-count boundary can form a trace in one run and not the next. trace_report.lua absorbs this by intersecting several independent trials and reporting only the sites and end states every trial agrees on, so the marginal ones drop out; the canonical report is then stable by construction. The FNEW census has no such channel — it never runs the code — so ./bench/ci still generates it twice and compares byte for byte.