Skip to content

perf: reduce memory baseline for busybox and sensor#19999

Draft
davdhacs wants to merge 4 commits intomasterfrom
davdhacs/pr1-memory-baseline
Draft

perf: reduce memory baseline for busybox and sensor#19999
davdhacs wants to merge 4 commits intomasterfrom
davdhacs/pr1-memory-baseline

Conversation

@davdhacs
Copy link
Copy Markdown
Contributor

@davdhacs davdhacs commented Apr 14, 2026

Description

Umbrella PR for the memory baseline optimization series. This tracks the overall effort — individual changes are in separate PRs for focused review.

Merged

  • perf: combine logger zap sampling #19997Shared zap sampler (-46 MB sensor, -170 MB central). Replaces per-logger sampling with a single shared sampler core. Same flood protection, 1/100th the memory.

Open — Schema Lazy Loading

Open — Init-Time Reductions

Open — Logging Improvements

Not Yet Created

  • Process enricher cache scaling with ROX_MEMLIMIT
  • Additional PRs from the full roadmap (13 PRs total)

Combined Measurements (live GKE cluster)

Component Master After zap fix (#19997) Delta
Central 267 Mi 231 Mi -36 Mi
Sensor 142 Mi 123 Mi -19 Mi
Admission-control 40 Mi 36 Mi -4 Mi
Total 449 Mi 390 Mi -59 Mi

Note: These measurements are from the zap sampler change only. Additional savings expected from schema lazy loading and init-time reductions once deployed.

User-facing documentation

Testing and quality

  • the change is production ready
  • CI results are inspected

How I validated my change

This is a tracking PR. Individual PRs have their own validation.

AI-assisted.

@openshift-ci
Copy link
Copy Markdown

openshift-ci bot commented Apr 14, 2026

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

Copy link
Copy Markdown
Contributor

@sourcery-ai sourcery-ai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry @github-actions[bot], your pull request is larger than the review limit of 150000 diff characters

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Apr 14, 2026

🚀 Build Images Ready

Images are ready for commit 10e59b0. To use with deploy scripts:

export MAIN_IMAGE_TAG=4.11.x-661-g10e59b0d79

davdhacs and others added 4 commits April 15, 2026 06:53
Reduce init-time memory for the busybox binary by eliminating unnecessary
imports, deferring allocations with sync.OnceValue, and breaking heavy
transitive dependency chains.

Results (Linux amd64):
- Busybox: 16.1 MB -> 12.9 MB heap (-20%), 245K -> 173K mallocs (-29%)
- AC standalone: 9.1 MB -> 7.2 MB heap (-21%), 87K -> 51K mallocs (-41%)
- Binary size: 205 MB -> 194 MB (-5%)

Generated with assistance from AI

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Each logger that writes to a file spawns a lumberjack goroutine for
log rotation. With ~30 loggers writing to /var/log/stackrox/log.txt,
that's 30 idle goroutines + 30 independent file handles to the same
file. In container environments, logs go to stdout and are collected
by the container runtime — file logging is unnecessary overhead.

Set ROX_LOGGING_TO_FILE=false to disable file logging, saving:
- 30 goroutines and their stacks
- File I/O overhead
- lumberjack rotation processing

Default is true (unchanged behavior) for backward compatibility.

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Each CreateLogger call created an independent lumberjack.Logger for
the same log file, spawning its own rotation goroutine. With ~30
loggers, that's 30 goroutines + 30 file handles to the same file.

Share a single writer per path via a map. This reduces log rotation
goroutines from 30 to 1 and eliminates potential corruption from
concurrent uncoordinated writes to the same file.

GC sweet spot experiment findings (included in commit message for context):
- 128Mi: GC thrashing (84 GC/min, 200m CPU)
- 160Mi: Sweet spot (2 GC/min, 4m CPU)
- 192Mi: Comfortable (0 GC/min, 3m CPU)
- Rule: set limit to 1.3-1.5x natural heap size

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Process enrichment LRU cache was hardcoded at 100K entries — designed
for large enterprise clusters with thousands of containers. On a
50-container edge cluster, this is 2000x oversized.

Use pkg/sensor/queue.ScaleSize to scale based on ROX_MEMLIMIT:
- 128Mi limit → ~3K entries (sufficient for 50 containers)
- 4Gi limit → 100K entries (unchanged behavior)
- Minimum: 100 entries

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
@davdhacs davdhacs force-pushed the davdhacs/pr1-memory-baseline branch from 696081a to 10e59b0 Compare April 15, 2026 12:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant