Repository navigation
fix(execution): use jemalloc as the global allocator so server memory stays bounded under sustained load - #379
Merged
Merged
Conversation
…stays bounded under sustained load (#376)
… as it may be once subscribed A slow or stalled consumer reads one frame at a time, also while it opens its session: deltas for its first queries can queue ahead of the reply to the next subscribe, and the engine's send timeout then closes it. The soak allowed that close once subscribed but failed on it during the open; on the benchmark host the sustained soak against the jemalloc server (#376) failed on it 48 s in, with 40 send timeouts logged.
jsam
force-pushed
the
fm/fix-mem-growth
branch
from
October 5, 2026 22:35
b13d663 to
ac4be1e
Compare
jsam
changed the title
jsam
marked this pull request as ready for review
October 5, 2026 22:39
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Intent
I want you to really hyper focus on making the symbolic engine, you know, crisp and really truly production ready. (Context: implement #376: the 30-minute soak showed server RSS growing 251 -> 1,012 MiB from glibc arena retention; capping arenas costs ~40% throughput.)
What Changed
MeteredAllocator(src/execution/memory.rs) now forwards to jemalloc (tikv-jemallocator) instead ofSystemon every non-MSVC target. glibc's per-thread arenas held on to freed memory, so RSS kept growing under sustained load (Server memory grows under sustained load: RSS 251 -> 1,012 MiB in a 30-minute soak (glibc arena retention) #376).release_freed_memoryused to call glibc'smalloc_trim. It now runs jemalloc'sarena.<all>.purgethroughmallctl, and a new unit test checks that the purge returns success.Cargo.tomladdstikv-jemallocator/tikv-jemalloc-sys0.6 as non-MSVC target dependencies. TheDockerfilesetsJEMALLOC_SYS_WITH_LG_PAGE=16for arm64 builds, so the image starts on kernels with 64 KiB pages as well as 4 KiB ones.Risk Assessment
arena.4096.purge, i.e. MALLCTL_ARENAS_ALL), the msvc fallback and the soak test relaxation all look correct.Testing
Release builds of the change and of base efa87aa were each run as a real inputlayer-server, under a systemd user scope (MemoryMax=16G, no swap), and driven with the real inputlayer-client. On the change, an unbounded closure query peaked at about 1 GiB and resident memory was back to 72 MiB within about 2 s; a small query still worked afterwards. With a 32 MiB query limit, both builds refused the runaway query and the same connection kept answering. The repo's /ws memory_limit_tests also passed on the jemalloc build, confirming the
resource_exhaustedcode. As the operator decided, the sustained-memory and slow-consumer-harness scenarios use the 30-minute soak on the benchmark host, which passed: RSS stayed flat between 95 and 124 MiB, against the glibc baseline's growth from 251 to 1,012 MiB. That run had 829 slow-subscriber disconnects and no failures. These are CLI transcripts and RSS logs; there is no UI, so there are no screenshots. All scratch servers and builds under /tmp were removed.?edge(0, Y)returned 2 rows. Base: 1,121 MiB to 96 M….session clearand?edge(0, Y)returned 2 rows on the same connecti…Evidence: RSS summary for checks (a) and (b), change vs base
a2-head: peak_rss=1003 MiB, final_rss=72 MiB, time from peak to <25% of peak=1.99s a2-base: peak_rss=1121 MiB, final_rss=96 MiB, time from peak to <25% of peak=3.04s b2-head: peak_rss=75 MiB (query refused at 32 MiB limit) b2-base: peak_rss=62 MiB (query refused at 32 MiB limit)~/.no-mistakes/evidence/01M471W8JV2JZ8MZPH16CM73JC/check-a-runaway-closure.txt)~/.no-mistakes/evidence/01M471W8JV2JZ8MZPH16CM73JC/check-b-query-memory-limit.txt)Evidence: RSS samples at 50 ms, change
~/.no-mistakes/evidence/01M471W8JV2JZ8MZPH16CM73JC/rss-a-base.log)Evidence: memory_limit_tests on jemalloc build
Evidence: Driver script and IQL programs
Pipeline
Updates from git push no-mistakes
✅ **intent** - passed
✅ No issues found.
✅ **Rebase** - passed
✅ No issues found.
src/execution/memory.rs:56- The new doc comment says jemalloc returns freed memory 'after about ten seconds unused'. Withoutbackground_thread(not enabled here: there is nomalloc_conf/_rjem_malloc_conf), jemalloc only applies decay when the same arena allocates or frees again. If a burst of threads goes quiet, their arenas can keep dirty pages until the nextrelease_freed_memory()purge, which only runs after a query with a peak above 64 MiB. RSS stays bounded, so this does not undo the Server memory grows under sustained load: RSS 251 -> 1,012 MiB in a 30-minute soak (glibc arena retention) #376 fix, but the comment overstates how promptly memory is returned. If idle-RSS return matters, setbackground_thread:truein the malloc conf, or reword the comment.🔧 **Test** - 2 issues found → no changes applied ✅
scripts/soak.sh --secs 600 --set MAX_RSS_GROWTH_PCT=25on the change and on base efa87aa, and run /tmp/nm-01M471/mem/run.sh on both builds.CARGO_TARGET_DIR=/tmp/nm-01M471/target-head cargo test --release --all-features --test differential_oracle --no-run(release build of the change; still compiling when the result was required)CARGO_TARGET_DIR=/tmp/nm-01M471/target-base cargo test --release --all-features --test differential_oracle --no-runon a copy of base commit efa87aa5 (still compiling)Prepared /tmp/nm-01M471/mem/run.sh: starts the real inputlayer-server with an isolated config, data directory and credentials; runs the 1500-node runaway closure through inputlayer-client; samples VmRSS every 50 ms (never executed)🔧 No changes applied.
✅ Re-checked - no issues remain.
?edge(0, Y)returned 2 rows. Base: 1,121 MiB to 96 M….session clearand?edge(0, Y)returned 2 rows on the same connecti…Checked that the base scratch source (src, Cargo.toml) is identical togit archive efa87aa5and that the change's source is clean at ac4be1ee; the change's server binary links jemalloc and base does not (strings | grep jemalloc)SCRIPT=closure.iql run2.sh target-head a2-head 18861 0and the same for target-base: each server undersystemd-run --user --scope -p MemoryMax=16G -p MemorySwapMax=0, a runaway transitive-closure query over the loaded edge graph, VmRSS sampled every 50 ms, then a small?edge(0, Y)query on the same serverSCRIPT=closure_then_small.iql run2.sh target-{head,base} b2-* 1887x 33554432: with max_query_memory_bytes = 32 MiB, the runaway rule and query are refused, then.session clearand?edge(0, Y)run on the same CLI connectionCARGO_BUILD_JOBS=4 cargo test --release --all-features --test memory_limit_tests(the change, jemalloc global allocator): /ws end-to-end checks ofcode == resource_exhaustedand that the connection stays usableCited the operator-provided 30-minute sustained soak on sam-dev-benchmarks (built from e2a0882e) for the soak and slow-consumer harness scenarios, as the operator decided✅ **Document** - passed
✅ No issues found.
✅ **Lint** - passed
✅ No issues found.
✅ **Push** - passed
✅ No issues found.