Benchmark harness: PUC Lua timings and deterministic LuaJIT counters for generated code#195
Merged
Conversation
The benchmark harness meters generated code under LuaJIT (static FNEW census via jit.util, trace abort/blacklist reports via jit.attach); the shell so far only carried PUC Lua 5.1. nixpkgs' luajit is a mainline 2.1 snapshot, which is what generated code typically runs under.
Three small modules exercising the hot generated shapes end-to-end: fib recursion with typeclass arithmetic, Foldable foldl over an Array with a curried step (needs the arrays package, hence the new dependency), and a three-step Maybe bind chain. The golden harness only collects Golden.*-prefixed output directories, so these compile to CoreFn alongside the goldens without joining the golden suite; the bench scripts link them AsModule from that CoreFn.
./bench/run times micro codegen patterns (curried application, dictionary-driven comparison, constructor allocation + tag match, module-field access) and the linked Bench.* macro modules under PUC Lua and LuaJIT with and without the JIT. Wall-clock numbers stay local: they are too noisy for shared CI runners. ./bench/ci is the CI-grade check: byte-stable LuaJIT counters diffed against committed oracles in bench/goldens/ (--accept rewrites them after a deliberate codegen change), with every report generated twice and compared to prove stability. Two meters, both built on jit.util instead of parsing -bl/-jv text, whose format is not a contract: * fnew_census statically walks a linked artifact's prototypes and counts FNEW bytecodes, split into main-chunk occurrences (run once at load) and function-body occurrences (allocate per call, and abort trace recording: closure creation is NYI in the 2.1 tracer). * trace_report drives a macro spec hot and reports the set of distinct trace-abort sites plus the end state of loop/function-entry opcodes: the JIT rewrites them to J* forms when it installs a trace and to I* forms when it blacklists a spot. Blacklisting is never logged, and raw abort counts jitter with LuaJIT's entropy-seeded retry-penalty PRNG, so the post-hoc opcode read and the site set are the stable signals (aborts that merely ran into an already-blacklisted spot are filtered out for the same reason). On this machine the harness reproduces the motivating measurements: the curried hot loop runs ~120x slower than its uncurried shape under LuaJIT and no faster with -joff (every partial application FNEWs, the loop is blacklisted), and the FFI fold loop driving foldl with a curried step ends up IFORL-blacklisted while the hand-written fold compiles to a single trace.
The step runs after the test suite on purpose: the suite's spago build leaves the full CoreFn output (including dependencies) on disk, which is what ./bench/ci links the Bench.* artifacts from.
pslua does exit non-zero on a linker error; the earlier '$?' check that seemed to say otherwise was reading tail's exit code through a pipe, not pslua's. bench/link already runs under 'set -e', so a real failure stops it without help.
bench/link now rebuilds CoreFn unconditionally, clears stale _build artifacts, and derives its module list from test/ps/src/Bench/*.purs — the prior existence-check guard could silently link stale or missing CoreFn after editing a bench source. fnew_census.lua and trace_report.lua share their bytecode-walking logic via a new bc_lib.lua, derive the J*/I* opcode families from vmdef.bcnames instead of a hardcoded snapshot, and stamp jit.version into every report so a LuaJIT bump shows up as an attributable golden diff instead of a mystery one. The abort-reason formatter now falls back correctly for builtin functions, which carry no source info. Spec name fields are dropped in favor of deriving them from the file path, removing a filename/name drift the report headers depended on. bench_lib.measure guards against a zero/negative sample count. Goldens are regenerated to match.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #172.
What this adds
A
bench/directory, separate from the test suite, plusluajitin the dev shell and a CI step. Layout and usage are documented inbench/README.md../bench/runproduces local wall-clock numbers: hand-written micro pairs (curried application, dictionary-driven comparison, constructor allocation + tag match, module-field access), each in its current generated shape next to the idiomatic Lua it stands for, and three macrobenchmarks driving real linked modules (Bench.Fib,Bench.ArrayFoldl,Bench.BindChain, sources undertest/ps/src/Bench/). Everything runs under PUC Lua, LuaJIT, andluajit -joff. Timings stay out of CI: shared runners are too noisy for them../bench/ciis the CI check. It relinks theBench.*artifacts with the current compiler, regenerates two kinds of LuaJIT counter reports, proves each is byte-stable (every report is generated twice and compared), and diffs them against the committed oracles inbench/goldens/. After a deliberate codegen change,./bench/ci --acceptrewrites the oracles and the diff gets reviewed like any golden. The CI step runs after the test suite on purpose: the suite'sspago buildleaves the full CoreFn output on disk, which is whatbench/linklinks from.Both meters read
jit.utildirectly instead of parsing-bl/-jvtext, whose format is not a contract and has already changed shape across LuaJIT versions:fnew_censuswalks a linked artifact's prototypes statically (no execution) and countsFNEWbytecodes, split into main-chunk occurrences, which run once at load, and function-body occurrences, which allocate a closure per call and abort trace recording (NYI: bytecode FNEW). The census is a pure function of the artifact.trace_reportdrives a macro spec hot and reports the set of distinct trace-abort sites plus the end state of loop and function-entry opcodes: LuaJIT rewrites them toJ*forms when it installs a trace and toI*forms when it blacklists a spot. Blacklisting is never logged by-jv/-jdump, and raw abort counts jitter with the entropy-seeded retry-penalty PRNG (mainline LuaJIT has nojit.prngstateto pin it), so the post-hoc opcode read and the deduplicated site set are the stable signals. Aborts whose reason is merely "blacklisted" are filtered out: whether recording ever runs into an already-blacklisted spot depends on the retry count, and theI*opcode already carries that fact deterministically.Measured baselines (this machine, LuaJIT 2.1.1741730670, PUC Lua 5.1.5)
acc = add(acc)(i)(2e7, LuaJIT)luajit -joffcurrent>=viaordIntdictionary (5e6, PUC)M.fieldcall vs local (1e7, PUC)foldlwith curried step (5e6, LuaJIT)The issue's verification bullets all reproduce. The curried hot loop runs the same with the JIT on and off, because every partial application emits an
FNEW; the census shows thoseFNEWs in function bodies (12 in theBench.ArrayFoldlartifact, one per curried level). The FFI fold loop ends upIFORL-blacklisted even though it is a plainfor, while the hand-written fold compiles to aJFORLtrace. Inbind_chainthe blacklist lands on the driver loop, whose callee cannot be inlined: the caller pays. All six counter reports came out identical over repeated runs (the trace reports were additionally checked over 6 consecutive runs), so CI can gate on them byte for byte.Two comparator notes. The issue's one-off measurement quoted ~54x for the curried loop and ~85x for
foldl; the ratios here differ in the expected directions. The curried gap grew to 120x because the uncurried loop compiles tighter under the current LuaJIT snapshot. Thefoldlgap shrank to 24-38x because the harness'sidealdoes the same total work as the linked artifact, array construction included; against a fold-only comparator like the issue's, the same runs give ~88x at 1e7 elements.Notes
Bench.*sources rather than the linked golden artifacts themselves: a module export can be driven and timed in-process without going through stdout, and theBench.prefix keeps them out of the golden suite (the harness only collectsGolden.*).Bench.Fibis source-identical to the golden'sfib.