Add performance counters/metrics for samplers - #811
Conversation
CI Test ResultsRun: #35935274258 | Commit:
Status Overview
Legend: ✅ passed | ❌ failed | ⚪ skipped | 🚫 cancelled Summary: Total: 32 | Passed: 32 | Failed: 0 Updated: 2026-09-24 00:05:43 UTC |
|
❌ 39 passed, 1 failed out of 40 expected configurations Test Matrix
Failure Detailsglibc-x64-openj9-jdk21Profiler-only: Links
|
There was a problem hiding this comment.
Two of the most critical issues are a shutdown race where probe counter updates escape inflight tracking (potentially corrupting profiling reports) and a missing extern "C" on Agent_OnUnload that silently prevents the JVM from invoking the shutdown callback on agentpath-loaded libraries. Additionally, J9Execution mode records to the wrong performance counter, and unsigned TSC subtraction can wrap on core migrations, both corrupting measurement data.
🤖 Bits Code Review · Commit eaf6b5b · @DataDog review to ask questions
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: eaf6b5b904
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Co-authored-by: datadog-datadog-prod-us1[bot] <88084959+datadog-datadog-prod-us1[bot]@users.noreply.github.com>
Co-authored-by: datadog-datadog-prod-us1[bot] <88084959+datadog-datadog-prod-us1[bot]@users.noreply.github.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a1eb67754e
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
❌ ErrorsYour PR has failed checks. Please review the issues below and take necessary action before merging. 🚦 2 Pipeline jobs failed
Useful? React with 👍 / 👎 This comment will be updated automatically if new data arrives.🔗 Commit SHA: 97b79ee | Docs | View more details | Give us feedback! |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 9774278e51
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 23f9c7c3f0
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Reliability & Chaos Results❌ 30 failure(s) detected Pipeline: https://gitlab.ddbuild.io/DataDog/java-profiler/-/pipelines/139232785 ❌ profiler gmalloc amd64Xjit❌ profiler gmalloc amd64Xmemory❌ profiler jemalloc aarch64Xjit❌ profiler jemalloc amd64Xjit❌ profiler tcmalloc amd64Xjit❌ profiler tcmalloc amd64Xmemory❌ profiler tracer tcmalloc amd64Xjit❌ chaos: profiler gmalloc aarch64 21 0 3 temXchaos❌ chaos: profiler gmalloc aarch64 25 0 3 temXchaos❌ chaos: profiler gmalloc amd64 21 0 3 temXchaos❌ chaos: profiler gmalloc amd64 25 0 3 temXchaos❌ chaos: profiler jemalloc aarch64 21 0 3 temXchaos❌ chaos: profiler jemalloc aarch64 25 0 3 temXchaos❌ chaos: profiler jemalloc amd64 25 0 3 temXchaos❌ chaos: profiler tcmalloc aarch64 21 0 3 temXchaos❌ chaos: profiler tcmalloc aarch64 25 0 3 temXchaos❌ chaos: profiler tcmalloc amd64 21 0 3 temXchaos❌ chaos: profiler tcmalloc amd64 25 0 3 temXchaos❌ chaos: profiler tracer gmalloc aarch64 21 0 3 temXchaos❌ chaos: profiler tracer gmalloc aarch64 25 0 3 temXchaos❌ chaos: profiler tracer gmalloc amd64 21 0 3 temXchaos❌ chaos: profiler tracer gmalloc amd64 25 0 3 temXchaos❌ chaos: profiler tracer jemalloc aarch64 21 0 3 temXchaos❌ chaos: profiler tracer jemalloc aarch64 25 0 3 temXchaos❌ chaos: profiler tracer jemalloc amd64 21 0 3 temXchaos❌ chaos: profiler tracer jemalloc amd64 25 0 3 temXchaos❌ chaos: profiler tracer tcmalloc aarch64 21 0 3 temXchaos❌ chaos: profiler tracer tcmalloc aarch64 25 0 3 temXchaos❌ chaos: profiler tracer tcmalloc amd64 21 0 3 temXchaos❌ chaos: profiler tracer tcmalloc amd64 25 0 3 temXchaos |
kaahos
left a comment
There was a problem hiding this comment.
Looks good to me! Just 1 nit comment regarding the documentation which might not be up-to-date - nothing blocking though.
What does this PR do?:
Adds an opt-in (
-PenableSamplerPerf), zero-cost-when-disabled instrumentation layer (SamplerPerf/SAMPLER_PERF_PROBE) that measures elapsed wall time and sample count per sampler family (CPU, wallclock, alloc, liveness, nativemem, nativesocket). Each sampler's signal handler or hot path gets a probe call (ctimer_linux.cpp,itimer.cpp,perfEvents_linux.cpp,wallClock.cpp,j9/j9WallClock.cpp,objectSampler.cpp,livenessTracker.cpp,mallocTracer.cpp,nativeSocketSampler.cpp). Counters are exposed through the existingCounters/JFR/getDebugCounters()machinery (counters.h) and a summary table is printed atProfiler::stop()viaSamplerPerf::report(). Also addsAgent_OnUnloadinvmEntry.cppso JVMTI-only (-agentpath) loads get the same shutdown path asJNI_OnUnload. New unit tests insamplerPerf_ut.cppand a new doc page,doc/reference/SamplerPerfCounters.md, explain the caveats around reading the numbers (elapsed vs CPU time, cpu/wall subset relationships, cross-sampler nesting, bailed-out samples, and a bounded shutdown-time undercount race).A sample output:
Motivation:
There was no way to measure each sampler's own per-sample overhead in isolation — existing counters track totals but not per-engine timing, making it hard to attribute latency regressions to a specific sampler.
Additional Notes:
Default builds are unaffected — without
-PenableSamplerPerfthe macro expands to((void)0)and the counters occupy no enum slot or storage.How to test the change?:
Build with
-PenableSamplerPerfand run the newsamplerPerf_ut.cppgtest suite (wired intoGtestTaskBuilder.kt/ConfigurationPresets.ktthe same way as the existingfaultInjectionflag). For an end-to-end check, run any profiled JVM workload with the flag enabled and confirm the[java-profiler] sampler performancetable prints at shutdown.For Datadog employees:
credentials of any kind, I've requested a security review (run the
dd:platform-security-reviewskill, or file a request via the PSEC review form).
bewairealso runs automatically on every PR.Unsure? Have a question? Request a review!