Repository navigation
Allocator (jemalloc, #379): 4-writer async insert p99 +29% and sessions peak memory doubled; keep, tune or revert #411
Description
Found by the performance bisect that followed #374 (benchmark host, paired and interleaved runs, 160 rounds, insert_async fixture with 4 writers).
Finding. Since #379 (jemalloc as the global allocator) the 4-writer async insert acknowledgement p99 is about 1.29x worse: 0.54 ms before, 0.71 ms after (ratio against the commit before #379: 1.297, 95% interval 1.257..1.339; against #379 itself 1.030; A/A 0.989). #367, #381 and #382 are cleared. The same merge makes the 4-writer p50 4% better and the serial p50 1-2% better; only the tail moved. The cost sits in each writer's first two requests after request threads start (a cold thread cache); without those two requests the p99 is +3%.
Related. The sessions fixture peak is about 127 MB on current main against a 63 MB baseline (87 MB when #375 was filed), and each login re-faults a 20 MB argon2 buffer under jemalloc (39,068 page faults over one round's logins against 4,917 under glibc); login rate measured 39/s against 62/s with a glibc-era fix that no longer applies.
Options (decision needed).
| Option | 4-writer ack p99 | Other effects |
|---|---|---|
| Keep #379 as it is | 0.71 ms (+29%) | Keeps memory bounded under sustained load (#376); 4-writer p50 4% better |
Tune oversize_threshold:0 |
0.934x of main (0.904..0.980) | No p50 cost; logins stop re-faulting 20 MB each; that memory stays resident until it decays (about 10 s); memory under the sustained soak not measured |
Tune tcache_nslots_small_max:20 |
0.868x of main (0.838..0.901) | 4-writer p50 5% worse |
| Start and keep request threads (prototype) | 0.815x of main, back to the level before #379 | Serial p50 24% worse; needs a request pool that reuses the most recently used thread |
| Revert #379 | back to 0.54 ms | Loses the memory bound (#376 reopens) |
Not settled: the memory effect of either setting under the sustained soak, and login latency as a paired measurement. Arena count, background purging and decay settings were measured and are not the cause.
The measurement record and raw samples are in perf-gate/baselines/async-insert-p99-2026-10-06.md in the engine repository. The gate already reports this metric as diagnostic (insert_async.concurrent_ack_us.p99).
Activity
jsam commentedon Oct 7, 2026
Measured on the delivery box (2026-10-07): with jemalloc oversize_threshold:0 the 4-writer async ack p99 did not improve (ratio 1.020 against main, 95% interval 0.995..1.068 over 800 paired rounds; main against itself 1.034), sessions peak RSS rose 17% (317 to 372 MB), and in the 30-minute sustained soak memory in the last 5 minutes was 13.3% above minute 5 (0 divergences; control with jemalloc defaults: 5.1%). The tuning does not meet its bars here and is not shipped; jemalloc stays with its defaults.
jsam commentedon Oct 8, 2026
oversize_threshold:0 did not meet the bars on the dev box (4-writer async insert p99 not improved); the tuning branch is discarded on the owner's decision of 2026-10-08. jemalloc stays with its defaults.
jsam commentedon Oct 10, 2026
The Argon2 arena fix landed in https://github.com/inputlayer/inputlayer-symbolic/pull/18, and the decision to keep jemalloc defaults was taken in this issue's comments. The 4-writer p99 tail and sessions RSS are not settled. Remaining work continues in https://github.com/inputlayer/inputlayer-symbolic/issues/132.