Skip to content

Benchmark harness: PUC Lua timings and deterministic LuaJIT counters for generated code#195

Merged
Unisay merged 6 commits into
mainfrom
issue-172/bench-harness
Jul 7, 2026
Merged

Benchmark harness: PUC Lua timings and deterministic LuaJIT counters for generated code#195
Unisay merged 6 commits into
mainfrom
issue-172/bench-harness

Conversation

@Unisay

@Unisay Unisay commented Jul 7, 2026

Copy link
Copy Markdown
Collaborator

Closes #172.

What this adds

A bench/ directory, separate from the test suite, plus luajit in the dev shell and a CI step. Layout and usage are documented in bench/README.md.

./bench/run produces local wall-clock numbers: hand-written micro pairs (curried application, dictionary-driven comparison, constructor allocation + tag match, module-field access), each in its current generated shape next to the idiomatic Lua it stands for, and three macrobenchmarks driving real linked modules (Bench.Fib, Bench.ArrayFoldl, Bench.BindChain, sources under test/ps/src/Bench/). Everything runs under PUC Lua, LuaJIT, and luajit -joff. Timings stay out of CI: shared runners are too noisy for them.

./bench/ci is the CI check. It relinks the Bench.* artifacts with the current compiler, regenerates two kinds of LuaJIT counter reports, proves each is byte-stable (every report is generated twice and compared), and diffs them against the committed oracles in bench/goldens/. After a deliberate codegen change, ./bench/ci --accept rewrites the oracles and the diff gets reviewed like any golden. The CI step runs after the test suite on purpose: the suite's spago build leaves the full CoreFn output on disk, which is what bench/link links from.

Both meters read jit.util directly instead of parsing -bl/-jv text, whose format is not a contract and has already changed shape across LuaJIT versions:

  • fnew_census walks a linked artifact's prototypes statically (no execution) and counts FNEW bytecodes, split into main-chunk occurrences, which run once at load, and function-body occurrences, which allocate a closure per call and abort trace recording (NYI: bytecode FNEW). The census is a pure function of the artifact.
  • trace_report drives a macro spec hot and reports the set of distinct trace-abort sites plus the end state of loop and function-entry opcodes: LuaJIT rewrites them to J* forms when it installs a trace and to I* forms when it blacklists a spot. Blacklisting is never logged by -jv/-jdump, and raw abort counts jitter with the entropy-seeded retry-penalty PRNG (mainline LuaJIT has no jit.prngstate to pin it), so the post-hoc opcode read and the deduplicated site set are the stable signals. Aborts whose reason is merely "blacklisted" are filtered out: whether recording ever runs into an already-blacklisted spot depends on the retry count, and the I* opcode already carries that fact deterministically.

Measured baselines (this machine, LuaJIT 2.1.1741730670, PUC Lua 5.1.5)

Pattern current / linked ideal slowdown
curried hot loop acc = add(acc)(i) (2e7, LuaJIT) 0.981 s 0.008 s 120x
same, luajit -joff 1.002 s 0.109 s JIT buys nothing on current
curried hot loop (2e7, PUC) 0.960 s 0.295 s 3.3x
>= via ordInt dictionary (5e6, PUC) 0.816 s 0.027 s 30x
ctor alloc + tag match (2e6, PUC) 0.125 s 0.104 s 1.2x
M.field call vs local (1e7, PUC) 0.210 s 0.148 s 1.4x
linked fib(30) (PUC) 0.290 s 0.056 s 5.2x
linked foldl with curried step (5e6, LuaJIT) 0.325 s 0.014 s 24x
linked Maybe bind chain (1e6, LuaJIT) 0.485 s 0.0004 s interpreter-bound

The issue's verification bullets all reproduce. The curried hot loop runs the same with the JIT on and off, because every partial application emits an FNEW; the census shows those FNEWs in function bodies (12 in the Bench.ArrayFoldl artifact, one per curried level). The FFI fold loop ends up IFORL-blacklisted even though it is a plain for, while the hand-written fold compiles to a JFORL trace. In bind_chain the blacklist lands on the driver loop, whose callee cannot be inlined: the caller pays. All six counter reports came out identical over repeated runs (the trace reports were additionally checked over 6 consecutive runs), so CI can gate on them byte for byte.

Two comparator notes. The issue's one-off measurement quoted ~54x for the curried loop and ~85x for foldl; the ratios here differ in the expected directions. The curried gap grew to 120x because the uncurried loop compiles tighter under the current LuaJIT snapshot. The foldl gap shrank to 24-38x because the harness's ideal does the same total work as the linked artifact, array construction included; against a fold-only comparator like the issue's, the same runs give ~88x at 1e7 elements.

Notes

  • The macro modules are new Bench.* sources rather than the linked golden artifacts themselves: a module export can be driven and timed in-process without going through stdout, and the Bench. prefix keeps them out of the golden suite (the harness only collects Golden.*). Bench.Fib is source-identical to the golden's fib.

Unisay added 5 commits July 7, 2026 09:24
The benchmark harness meters generated code under LuaJIT (static FNEW
census via jit.util, trace abort/blacklist reports via jit.attach); the
shell so far only carried PUC Lua 5.1. nixpkgs' luajit is a mainline
2.1 snapshot, which is what generated code typically runs under.
Three small modules exercising the hot generated shapes end-to-end:
fib recursion with typeclass arithmetic, Foldable foldl over an Array
with a curried step (needs the arrays package, hence the new
dependency), and a three-step Maybe bind chain. The golden harness only
collects Golden.*-prefixed output directories, so these compile to
CoreFn alongside the goldens without joining the golden suite; the
bench scripts link them AsModule from that CoreFn.
./bench/run times micro codegen patterns (curried application,
dictionary-driven comparison, constructor allocation + tag match,
module-field access) and the linked Bench.* macro modules under PUC Lua
and LuaJIT with and without the JIT. Wall-clock numbers stay local:
they are too noisy for shared CI runners.

./bench/ci is the CI-grade check: byte-stable LuaJIT counters diffed
against committed oracles in bench/goldens/ (--accept rewrites them
after a deliberate codegen change), with every report generated twice
and compared to prove stability. Two meters, both built on jit.util
instead of parsing -bl/-jv text, whose format is not a contract:

* fnew_census statically walks a linked artifact's prototypes and
  counts FNEW bytecodes, split into main-chunk occurrences (run once at
  load) and function-body occurrences (allocate per call, and abort
  trace recording: closure creation is NYI in the 2.1 tracer).

* trace_report drives a macro spec hot and reports the set of distinct
  trace-abort sites plus the end state of loop/function-entry opcodes:
  the JIT rewrites them to J* forms when it installs a trace and to I*
  forms when it blacklists a spot. Blacklisting is never logged, and
  raw abort counts jitter with LuaJIT's entropy-seeded retry-penalty
  PRNG, so the post-hoc opcode read and the site set are the stable
  signals (aborts that merely ran into an already-blacklisted spot are
  filtered out for the same reason).

On this machine the harness reproduces the motivating measurements:
the curried hot loop runs ~120x slower than its uncurried shape under
LuaJIT and no faster with -joff (every partial application FNEWs, the
loop is blacklisted), and the FFI fold loop driving foldl with a
curried step ends up IFORL-blacklisted while the hand-written fold
compiles to a single trace.
The step runs after the test suite on purpose: the suite's spago build
leaves the full CoreFn output (including dependencies) on disk, which
is what ./bench/ci links the Bench.* artifacts from.
pslua does exit non-zero on a linker error; the earlier '$?' check that
seemed to say otherwise was reading tail's exit code through a pipe,
not pslua's. bench/link already runs under 'set -e', so a real
failure stops it without help.
@Unisay Unisay self-assigned this Jul 7, 2026
@Unisay Unisay marked this pull request as ready for review July 7, 2026 08:20
bench/link now rebuilds CoreFn unconditionally, clears stale _build
artifacts, and derives its module list from test/ps/src/Bench/*.purs —
the prior existence-check guard could silently link stale or missing
CoreFn after editing a bench source. fnew_census.lua and
trace_report.lua share their bytecode-walking logic via a new
bc_lib.lua, derive the J*/I* opcode families from vmdef.bcnames instead
of a hardcoded snapshot, and stamp jit.version into every report so a
LuaJIT bump shows up as an attributable golden diff instead of a
mystery one. The abort-reason formatter now falls back correctly for
builtin functions, which carry no source info. Spec name fields are
dropped in favor of deriving them from the file path, removing a
filename/name drift the report headers depended on. bench_lib.measure
guards against a zero/negative sample count. Goldens are regenerated
to match.
@Unisay Unisay merged commit 210766c into main Jul 7, 2026
2 checks passed
@Unisay Unisay deleted the issue-172/bench-harness branch July 7, 2026 08:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Benchmark harness: PUC Lua timings and LuaJIT trace/FNEW counters for generated code

1 participant