Problem
The test suite checks the shape of generated code (structural goldens) and its semantics (eval goldens), but nothing measures performance. The upcoming optimization series (uncurrying #24, intrinsic inlining, inliner heuristics) needs every change to defend its win with a number, and wins must not silently regress afterwards.
Motivating measurements (PUC Lua 5.1.5 / LuaJIT 2.1, 2e7 iterations): a curried hot loop acc = add(acc)(i) runs 0.91 s under LuaJIT and 0.90 s with the JIT disabled via -joff — the JIT contributes nothing, because every trace recording aborts on NYI: bytecode FNEW (closure creation, executed on every partial application) and the loop start gets blacklisted. The uncurried equivalent compiles to a single trace and runs 0.017 s (54x). Driving the linked Data.Array.foldl with a curried step function reproduces this on real generated code: 0.99 s vs 0.012 s (85x), with the FFI fold loop itself blacklisted even though it is a plain for — the trace has to inline the curried callee.
Approach
A bench/ directory, separate from the test suite:
- Microbenchmarks of codegen patterns, timed under both PUC Lua and LuaJIT: curried application, dictionary-driven comparison (
greaterThanOrEq ordInt vs native >=), constructor allocation + tag match, M.<field> access vs a local.
- Macrobenchmarks driving linked real modules: the Fibonacci golden,
Data.Array.foldl over a large array, a monadic bind chain.
- Deterministic LuaJIT counters, suitable for CI: the number of
FNEW bytecodes in luajit -bl output of linked artifacts, split into main-chunk (runs once at init) vs function-body (runs per call) occurrences, and abort/blacklist event counts from luajit -jv runs of the macrobenchmarks.
Wall-clock timings stay in a local script — too noisy for CI. The counters are deterministic, so they can run as a CI smoke check and record the trend as optimizations land.
Requires adding luajit to the nix dev shell (currently only PUC Lua 5.1 is available there).
Prerequisites / Relations
Infrastructure with no hard prerequisite — self-contained apart from adding luajit to the nix dev shell. It should land early in the perf series: every perf change (#178, #179, #24, #180, #181, #173, #174, #185) cites this harness to defend its win, and the closing-audit issue #187 reads the trend it records.
Verification / Measurement
Done when the harness reproduces the baselines already recorded above on the current linked artifacts — curried application ~54x slower under LuaJIT, Data.Array.foldl ~85x, FNEW emitted on every partial application, the fold loop blacklisted — and the LuaJIT counters (main-chunk vs function-body FNEW counts; abort/blacklist counts from -jv) are byte-stable across runs so CI can assert on them. Wall-clock ratios come from the local script; CI gates on the deterministic counters.
Problem
The test suite checks the shape of generated code (structural goldens) and its semantics (eval goldens), but nothing measures performance. The upcoming optimization series (uncurrying #24, intrinsic inlining, inliner heuristics) needs every change to defend its win with a number, and wins must not silently regress afterwards.
Motivating measurements (PUC Lua 5.1.5 / LuaJIT 2.1, 2e7 iterations): a curried hot loop
acc = add(acc)(i)runs 0.91 s under LuaJIT and 0.90 s with the JIT disabled via-joff— the JIT contributes nothing, because every trace recording aborts onNYI: bytecode FNEW(closure creation, executed on every partial application) and the loop start gets blacklisted. The uncurried equivalent compiles to a single trace and runs 0.017 s (54x). Driving the linkedData.Array.foldlwith a curried step function reproduces this on real generated code: 0.99 s vs 0.012 s (85x), with the FFI fold loop itself blacklisted even though it is a plainfor— the trace has to inline the curried callee.Approach
A
bench/directory, separate from the test suite:greaterThanOrEq ordIntvs native>=), constructor allocation + tag match,M.<field>access vs a local.Data.Array.foldlover a large array, a monadic bind chain.FNEWbytecodes inluajit -bloutput of linked artifacts, split into main-chunk (runs once at init) vs function-body (runs per call) occurrences, and abort/blacklist event counts fromluajit -jvruns of the macrobenchmarks.Wall-clock timings stay in a local script — too noisy for CI. The counters are deterministic, so they can run as a CI smoke check and record the trend as optimizations land.
Requires adding
luajitto the nix dev shell (currently only PUC Lua 5.1 is available there).Prerequisites / Relations
Infrastructure with no hard prerequisite — self-contained apart from adding
luajitto the nix dev shell. It should land early in the perf series: every perf change (#178, #179, #24, #180, #181, #173, #174, #185) cites this harness to defend its win, and the closing-audit issue #187 reads the trend it records.Verification / Measurement
Done when the harness reproduces the baselines already recorded above on the current linked artifacts — curried application ~54x slower under LuaJIT,
Data.Array.foldl~85x,FNEWemitted on every partial application, the fold loop blacklisted — and the LuaJIT counters (main-chunk vs function-bodyFNEWcounts; abort/blacklist counts from-jv) are byte-stable across runs so CI can assert on them. Wall-clock ratios come from the local script; CI gates on the deterministic counters.