Conversation
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: daedd84e8c
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
There was a problem hiding this comment.
The EL7 stress job cannot pull its image. The nightly functional jobs also cannot start the script or load the required native test library.
🤖 Datadog Autotest · Commit daedd84 · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest
❌ ErrorsYour PR has failed checks. Please review the issues below and take necessary action before merging. 🚦 1 Pipeline job failed
Useful? React with 👍 / 👎 This comment will be updated automatically if new data arrives.🔗 Commit SHA: bcd7b1b | Docs | View more details | Give us feedback! |
CI Test ResultsRun: #35768595886 | Commit:
Status Overview
Legend: ✅ passed | ❌ failed | ⚪ skipped | 🚫 cancelled Summary: Total: 32 | Passed: 32 | Failed: 0 Updated: 2026-09-22 18:57:52 UTC |
83a2eb8 to
8a59fb1
Compare
|
Checked all 6 review threads — all point at the very first commit (daedd84), before the follow-up fixes landed. Each is already addressed:
Resolving all six. |
functional:x64-el7-jdk* previously just ran gradlew directly and let its raw exit code decide the job -- ddprof-test/quarantine.txt had no effect on it at all, unlike the GH Actions matrix (#777). Wraps the gradlew invocation in run_tests_with_retry.sh with cell "el7-<jdk>-<config>-amd64". MAX_ATTEMPTS defaults to 1 (no retry) here specifically: the default of 2 means a full second suite run, and this runner has already OOMKilled the container outright once at the current heap/concurrency (PR #805). Losing the flaky-vs-broken distinction is an acceptable trade for not doubling memory pressure on a runner already this tight; revisit once there's more headroom. Extends .functional_job's artifact paths with flake-evidence/, ci-outcome/ and build/logs/attempt.log so a quarantine decision on this job has evidence to show for it. Verified end-to-end inside the real Oracle Linux 7 image with a stub gradlew: a quarantined failure on its own cell now exits 0 through the full functional-tests.sh-shaped invocation. Co-Authored-By: Claude Sonnet 5 <[email protected]>
PROF-16014. Root-caused in PR #805: NativeSocketSampler's hooks install and fire correctly (confirmed via a debug-config diagnostic build), but the rate limiter rejects nearly every sample because NativeSocketTestBase.doTcpTransfer()'s assumption that the send buffer fills and blocks after ~32 iterations does not hold on this runner's network stack. Scoped to el7-8-release-amd64, the only cell this has actually been observed on -- not el7-*-release-amd64, so the nightly el7-17/-21 cells still gate for real if this doesn't reproduce there the same way. Co-Authored-By: Claude Sonnet 5 <[email protected]>
kaahos
left a comment
There was a problem hiding this comment.
I've left 2 comments, but nothing blocking. Thanks!
functional:x64-el7-jdk* previously just ran gradlew directly and let its raw exit code decide the job -- ddprof-test/quarantine.txt had no effect on it at all, unlike the GH Actions matrix (#777). Wraps the gradlew invocation in run_tests_with_retry.sh with cell "el7-<jdk>-<config>-amd64". MAX_ATTEMPTS defaults to 1 (no retry) here specifically: the default of 2 means a full second suite run, and this runner has already OOMKilled the container outright once at the current heap/concurrency (PR #805). Losing the flaky-vs-broken distinction is an acceptable trade for not doubling memory pressure on a runner already this tight; revisit once there's more headroom. Extends .functional_job's artifact paths with flake-evidence/, ci-outcome/ and build/logs/attempt.log so a quarantine decision on this job has evidence to show for it. Verified end-to-end inside the real Oracle Linux 7 image with a stub gradlew: a quarantined failure on its own cell now exits 0 through the full functional-tests.sh-shaped invocation. Co-Authored-By: Claude Sonnet 5 <[email protected]>
PROF-16014. Root-caused in PR #805: NativeSocketSampler's hooks install and fire correctly (confirmed via a debug-config diagnostic build), but the rate limiter rejects nearly every sample because NativeSocketTestBase.doTcpTransfer()'s assumption that the send buffer fills and blocks after ~32 iterations does not hold on this runner's network stack. Scoped to el7-8-release-amd64, the only cell this has actually been observed on -- not el7-*-release-amd64, so the nightly el7-17/-21 cells still gate for real if this doesn't reproduce there the same way. Co-Authored-By: Claude Sonnet 5 <[email protected]>
PR #790 moved build:x64 onto AlmaLinux 8, which took stresstest:x64 with it since it ran in the same image — runtime coverage at the shipped binary's glibc 2.17 floor went to zero. This follows up on the two items proposed there: - .stresstest_job now takes RUNTIME_IMAGE independently of BUILD_IMAGE, and stresstest:x64 points it at a new Oracle Linux 7 image so it goes back to running against glibc 2.17 while build:x64 stays on EL8. - adds a nightly functional:x64-el7-jdk{8,17,21} job matrix that runs the ddprof-test debug suite against the shipped x64 artifact on the same EL7 image, covering the libc-sensitive suites without multiplying the per-PR matrix. The new .gitlab/base/el7/Dockerfile is runtime-only (no compiler): both jobs consume the AlmaLinux-8-built .so via -Pskip-native -Pwith-libs. functional-tests.sh fetches the JDK under test from Adoptium into JAVA_TEST_HOME so Gradle itself keeps running on JDK 21 regardless of which JDK is under test. BUILD_IMAGE_X64_EL7 is a placeholder pin — the image still needs to be built and pushed via a real pipeline (REBUILD_IMAGES="x64-el7") before this can merge, same as #790's own image rebuild. Co-Authored-By: Claude Sonnet 5 <[email protected]>
Keeps the 3-JDK sweep nightly but lets one JDK's worth of EL7 functional coverage run on every pipeline, so it isn't gated entirely on the nightly schedule. Co-Authored-By: Claude Sonnet 5 <[email protected]>
Built and pushed successfully in pipeline 138399627, job 2056785023. Co-Authored-By: Claude Sonnet 5 <[email protected]>
…r probe stresstest:x64 failed with "No C++ compiler found": ConfigurationPresets. setupStandardConfigurations() calls PlatformUtils.findCompiler() unconditionally at project-configuration time, before -Pskip-native is consulted, so an image with no compiler at all fails to even configure the build. The stock EL7 gcc-c++ (4.8.5, base repo, no EPEL) satisfies that --version probe; -Pskip-native still gates every real compile/link task onlyIf, so it's never invoked to build anything. Verified the image still builds and g++ --version succeeds. This needs another `REBUILD_IMAGES="x64-el7"` run and a re-pin of BUILD_IMAGE_X64_EL7 before the fix takes effect. Co-Authored-By: Claude Sonnet 5 <[email protected]>
Rebuilt in pipeline 138438605 with the gcc-c++ fix from the previous commit; pushed and signed successfully before the rebuild-images log cuts off (mid-way through the unrelated datadog-ci image build). Co-Authored-By: Claude Sonnet 5 <[email protected]>
functional:x64-el7-jdk8 failed with "Permission denied" — the script was committed as mode 100644 instead of 100755. Co-Authored-By: Claude Sonnet 5 <[email protected]>
ContendedCallTraceStorageTest OOM'd with "GC overhead limit exceeded" on jdk8: it drives availableProcessors()*2 threads of CPU/allocation load against the standard -Xmx512m test heap, and the GitLab shared runner pod apparently gives it less headroom than GitHub's ubuntu-latest runners do. Adds an opt-in -PtestMaxHeap gradle property to ddprof-test (appended after the plugin's default -Xmx512m, so the last -Xmx wins) and passes -PtestMaxHeap=1536m from functional-tests.sh only. The shared default used by every other caller (GH Actions matrix, stresstest, etc.) is untouched. Co-Authored-By: Claude Sonnet 5 <[email protected]>
Several ddprof-test suites (NativeThread, DynamicNativeThread, ThreadEntryDetectionTest, ...) need ddprof-test-native's small JNI test helper on java.library.path regardless of which profiler binary is under test. Bare -Pskip-native disabled ALL native compilation project-wide, so those suites failed with UnsatisfiedLinkError/ NoClassDefFoundError on the EL7 functional job, which needs to skip only ddprof-lib's own build (substituted via -Pwith-libs) while still compiling that helper for real. PlatformUtils.isNativeSkipped() now supports -Pskip-native=<comma list of project names>, skipping only those; bare -Pskip-native (no value) keeps skipping everywhere, unchanged for every existing caller (stresstests.sh, build.sh, GH Actions). SimpleNativeLibPlugin (used by ddprof-test-native and malloc-shim) and ddprof-lib/build.gradle.kts's own skip-native check now go through it. functional-tests.sh passes -Pskip-native=ddprof-lib,malloc-shim (malloc-shim excluded too: it's unrelated to ddprof-test and its -std=c++17 flag doesn't compile under EL7's stock gcc 4.8.5). Verified locally: with -Pskip-native=ddprof-lib,malloc-shim, ddprof-test-native:linkLib actually compiles and links, while ddprof-lib:assembleDebugJar only runs copyExternalLibs (no native compile task in the graph at all). Co-Authored-By: Claude Sonnet 5 <[email protected]>
Now that -Pskip-native=ddprof-lib,malloc-shim actually compiles ddprof-test-native for the EL7 functional job, nativealloc.c failed: error: 'for' loop initial declarations are only allowed in C99 mode EL7's stock gcc (4.8.5) still defaults to gnu89; every other Linux gcc/ clang this project builds on already defaults to a C99+ dialect, so this was latent until a compiler this old actually built the file. Adds an explicit -std=gnu99. Verified against the real EL7 toolchain (oraclelinux:7 + gcc 4.8.5): all three ddprof-test-native source files compile and link. Co-Authored-By: Claude Sonnet 5 <[email protected]>
build:x64 (build.sh) only ever produces and ships the release-config
libjavaProfiler.so -- that IS the shipped artifact this job exists to
exercise. Running testDebug set ddprof_test.config=debug while the
loaded library was actually release-built, so debug-only assertions
failed outright instead of assumeTrue-skipping correctly, e.g.
JVMAccessTest.agentOnLoadVMFlagDetectionTest checking for
[TEST::INFO] log lines that only exist in debug builds
("initLibrary not found", "DebugNonSafepoints flag lookup log line
was not observed").
Co-Authored-By: Claude Sonnet 5 <[email protected]>
functional:x64-el7-jdk8 has NativeSocketEvent count: 0 across every retry (zero, not just fewer-than-expected), suggesting the send/recv/ write/read hooks (libraryPatcher_linux.cpp's GOT patching) never actually get installed in this environment -- but that can't be told apart from "installed but never firing" without the debug-only hook-fire counters, which only exist in a debug-config build. build:x64-debug-diag builds a real debug-config libjavaProfiler.so on the same AlmaLinux 8 toolchain build:x64 uses (release configs don't carry NativeSocketSampler's #ifdef DEBUG counters/TEST_LOG calls). functional:x64-el7-debug-diag runs just the NativeSocket* suite against it on the EL7 runtime (via functional-tests.sh's new TESTS_FILTER -> -Ptests passthrough), so --info output shows whether send_hook/recv_hook/write_hook/read_hook ever get called at all. Both are marked TEMPORARY and should be removed once this is root-caused. Co-Authored-By: Claude Sonnet 5 <[email protected]>
build:x64-debug-diag's first attempt failed to load on EL7: UnsatisfiedLinkError: libstdc++.so.6: version GLIBCXX_3.4.20 not found Debug configs link libstdc++ dynamically (only release gets -static-libstdc++), so the .so needed a newer GLIBCXX than EL7's stock gcc 4.8.5 provides. Statically linking (gated behind -Dddprof.debugStaticLibstdcxx=true, diagnostic-only) then surfaced a second, unrelated issue: -static-libstdc++ transitively pulls in libstdc++.a's cow-string-inst.o (legacy ABI string instantiations), which references std::random_device::_M_init -> getentropy@GLIBC_2.25 -- traced via the linker map (-Wl,-Map). That's dead code configureRelease's -Wl,--gc-sections already strips; debug's linkerArgs never had that flag. Adding --gc-sections and --exclude-libs,ALL alongside the static libstdc++ flags (same diagnostic gate) fixes both. Verified end to end: rebuilt in the real AlmaLinux 8 image, confirmed via objdump that the result has no libstdc++.so.6 NEEDED entry, no getentropy reference, and a max GLIBC_2.17 requirement, then loaded it inside the real EL7 image and saw VM::initLibrary log successfully. Co-Authored-By: Claude Sonnet 5 <[email protected]>
build:x64-debug-diag and functional:x64-el7-debug-diag did their job: they showed the socket hooks fire correctly on EL7 (send/recv/write/ read all called, PLT-patched libjvm.so/libnio.so/libnet.so), and that the real cause of NativeSocketEvent count: 0 is the sampler's rate limiter legitimately rejecting nearly every call because NativeSocketTestBase.doTcpTransfer()'s "the send buffer fills after ~32 iterations and blocks" assumption doesn't hold on this runner's network stack -- a real test bug, not a CI/EL7 problem. That will be quarantined separately now that #777 is merged. Drops the diagnostic-only -Dddprof.debugStaticLibstdcxx gate in ConfigurationPresets.kt and the now-unused TESTS_FILTER passthrough in functional-tests.sh along with the jobs that were the only callers of either. Co-Authored-By: Claude Sonnet 5 <[email protected]>
"(rebuilt with gcc-c++ added, see el7/Dockerfile)" and "restoring the runtime coverage build:x64 lost when it moved to AlmaLinux 8 (PR #790)" tell a reader what changed relative to old code, not why the current code is the way it is -- and the -std=gnu99 comment named the exact source construct that motivated it, which goes stale the moment that code changes. State the current rationale only. Co-Authored-By: Claude Sonnet 5 <[email protected]>
…sistency
- .gitlab-ci.yml: drop "unchanged from before" -- describes the diff
against the file's prior state, not the current mechanism.
- ddprof-lib/build.gradle.kts: its own skip-native check still used
bare hasProperty("skip-native"), not PlatformUtils.isNativeSkipped().
Both agree today (every caller either omits the property, uses it
bare, or lists "ddprof-lib" whenever it's meant to be skipped), so
this isn't a behavior change -- confirmed locally: assembleDebugJar
still runs only copyExternalLibs under
-Pskip-native=ddprof-lib,malloc-shim, and still depends on the real
compile/link without it. But a future -Pskip-native=<list not
containing ddprof-lib> would silently skip ddprof-lib's own build
too under the old bare check, contradicting isNativeSkipped()'s own
doc comment. Fixes the one call site the new scoped semantics
didn't reach.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
Flagged in review: functional-tests.sh's .jdk-cache/ was never in any cache paths -- .cache-config/.cache-config-pull only cover .gradle/caches/, .gradle/wrapper/, .m2/repository/ -- so every run re-downloaded the JDK tarball from Adoptium. Gives .functional_job its own two-entry cache instead of extending .cache-config-pull: the JDK tarball's natural cache key is the major version (TEST_JDK), not the branch/commit the Gradle cache is keyed on, and it doesn't change per-branch the way the Gradle cache's gradle-wrapper.properties hash does, so it gets pull-push with a static, ref-independent key while the Gradle entry stays pull-only exactly as before. Co-Authored-By: Claude Sonnet 5 <[email protected]>
Flagged in review: only ddprof-lib/build.gradle.kts and
SimpleNativeLibPlugin.kt went through PlatformUtils.isNativeSkipped()
after the earlier fix; GtestPlugin.kt, GtestTaskBuilder.kt,
FuzzTargetsPlugin.kt and ddprof-lib/benchmarks/build.gradle.kts still
used the bare hasProperty("skip-native") check, so
-Pskip-native=<list not containing ddprof-lib> would silently skip
gtest/fuzz/benchmarks too, contradicting the scoped semantics
everywhere else now documents.
GtestPlugin.kt/GtestTaskBuilder.kt are applied directly to ddprof-lib
(project.name == "ddprof-lib" there already), so those are a direct
swap. FuzzTargetsPlugin.kt and benchmarks/build.gradle.kts are applied
to the :ddprof-lib:fuzz and :ddprof-lib:benchmarks *subprojects*
(project.name == "fuzz"/"benchmarks") -- fuzzing and benchmarking are
part of ddprof-lib's native surface, not something a caller names on
its own, so those go through project.parent's identity instead of
their own.
Verified locally: -Pskip-native=ddprof-lib,malloc-shim still skips
:ddprof-lib:benchmarks:compileBenchmark (SKIPPED, not executed), and
it still compiles normally without the flag -- matching the old bare
check's behavior for every current caller, while now correctly
generalizing for a caller that doesn't name ddprof-lib.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
4285688 to
bcd7b1b
Compare
functional:x64-el7-jdk* previously just ran gradlew directly and let its raw exit code decide the job -- ddprof-test/quarantine.txt had no effect on it at all, unlike the GH Actions matrix (#777). Wraps the gradlew invocation in run_tests_with_retry.sh with cell "el7-<jdk>-<config>-amd64". MAX_ATTEMPTS defaults to 1 (no retry) here specifically: the default of 2 means a full second suite run, and this runner has already OOMKilled the container outright once at the current heap/concurrency (PR #805). Losing the flaky-vs-broken distinction is an acceptable trade for not doubling memory pressure on a runner already this tight; revisit once there's more headroom. Extends .functional_job's artifact paths with flake-evidence/, ci-outcome/ and build/logs/attempt.log so a quarantine decision on this job has evidence to show for it. Verified end-to-end inside the real Oracle Linux 7 image with a stub gradlew: a quarantined failure on its own cell now exits 0 through the full functional-tests.sh-shaped invocation. Co-Authored-By: Claude Sonnet 5 <[email protected]>
PROF-16014. Root-caused in PR #805: NativeSocketSampler's hooks install and fire correctly (confirmed via a debug-config diagnostic build), but the rate limiter rejects nearly every sample because NativeSocketTestBase.doTcpTransfer()'s assumption that the send buffer fills and blocks after ~32 iterations does not hold on this runner's network stack. Scoped to el7-8-release-amd64, the only cell this has actually been observed on -- not el7-*-release-amd64, so the nightly el7-17/-21 cells still gate for real if this doesn't reproduce there the same way. Co-Authored-By: Claude Sonnet 5 <[email protected]>
Summary
Follow-up to #790's "One coverage regression this introduces, and how to undo it" section.
Moving
build:x64to AlmaLinux 8 (#790) tookstresstest:x64with it, since it ran in the same image — runtime coverage at the shipped binary's actual glibc floor (2.17) went from thin to zero. This implements the two follow-ups #790 proposed:.stresstest_jobgets its own runtime image, independent of the image the artifact was built in.RUNTIME_IMAGEreplaces the reusedBUILD_IMAGEvariable in the job template;stresstest:x64now points it at a new Oracle Linux 7 image (glibc 2.17), whilebuild:x64stays on AlmaLinux 8 — matching ci: move the shipped binaries to AlmaLinux 8, get the test images off EOL Debian #790's own note that this "is better than the old arrangement, because it would exercise the el8-built artifact on 2.17 rather than a 2.17-built one." Every otherstresstest:*job keeps its existing image, just renamed to the new variable.functional:x64-el7-jdk{8,17,21}, runs theddprof-testsuite against the shipped x64 artifact on the same EL7 image — the "libc-sensitive suites" coverage ci: move the shipped binaries to AlmaLinux 8, get the test images off EOL Debian #790 called out.jdk17andjdk21are nightly-only (CI_PIPELINE_SOURCE == "schedule", same pattern as the existingchaos:buildjob);jdk8also runs on every pipeline — it's the oldest/most different from the JDK 21 that runs Gradle, so EL7 functional coverage isn't gated entirely on the nightly schedule.Details
.gitlab/base/el7/Dockerfile(new): Oracle Linux 7. Built and pushed for real viarebuild-images(BUILD_IMAGE_X64_EL7, currently pinned to the pipeline-138438605 digest,gcc-c++included — see below)..gitlab/scripts/functional-tests.sh(new): fetches the JDK under test from Adoptium intoJAVA_TEST_HOME, so Gradle itself keeps running on JDK 21 (Gradle 9's minimum) regardless of which JDK the suite targets. RunstestRelease, nottestDebug—build:x64only ever ships a release-config.so, sotestDebugwould setddprof_test.config=debugagainst a library that was actually built release, and debug-only assertions would fail outright instead of correctlyassumeTrue-skipping..gitlab/build-deploy/images.yml,rebuild-images.sh,check-image-updates.sh: wiredBASE_IMAGE_EL7/x64-el7through the same image-build, repin and staleness-check machinery as the existing images.PlatformUtils.isNativeSkipped()(build-logic):-Pskip-nativenow also accepts a comma-separated project list (-Pskip-native=ddprof-lib,malloc-shim) instead of only the bare, skip-everywhere form.functional-tests.shneeds to skip onlyddprof-lib's own native build (substituted via-Pwith-libs) while still compilingddprof-test-native's small JNI test helper for real — severalddprof-testsuites need it onjava.library.pathregardless of which profiler binary is under test, and the bare flag disabled that unconditionally.ddprof-test-native/build.gradle.kts: added-std=gnu99— one of its.cfiles uses a C99 for-loop-scoped declaration, which EL7's stock gcc 4.8.5 rejects under its defaultgnu89dialect (every other gcc/clang this project builds on already defaults to C99+, so this was latent until a compiler this old actually compiled the file).ddprof-test/build.gradle.kts: added an opt-in-PtestMaxHeapproperty (appended after the plugin's default-Xmx512m, so the last-Xmxwins), andfunctional-tests.shpasses-PtestMaxHeap=1536m. One suite (ContendedCallTraceStorageTest) drives more concurrent CPU/allocation load than this shared runner's pod apparently gives it headroom for at the default heap. Every other caller (GH Actions matrix,stresstest.sh) is unaffected..gitlab/base/el8/Dockerfile: addedgcc-c++. Gradle'sConfigurationPresets.setupStandardConfigurations()callsfindCompiler()unconditionally at configuration time, before-Pskip-nativeis even consulted — an image with no compiler at all fails to configure the build, even though-Pskip-nativemeans it's never actually invoked. (Applies to the EL7 imagefunctional-tests.shruns in, not the shipped AlmaLinux 8 build.)A real bug this surfaced (not fixed here)
With all of the above green,
functional:x64-el7-jdk8still fails — but now for a legitimate reason, not a CI bug. EveryNativeSocket*test reportsdatadog.NativeSocketEvent count: 0.Built a temporary debug-config diagnostic build (statically linked, since debug configs don't get
-static-libstdc++and needed it to even load on EL7 — see the now-removed diagnostic commits for the full trail) to getNativeSocketSampler's debug-only hook-fire counters. Confirmed the hooks are not the problem:send_hook/recv_hook/write_hook/read_hookall fire correctly, PLT patching succeeds acrosslibjvm.so/libnio.so/libnet.so. Thestop summarylines show the sampler's rate limiter rejecting nearly every single call (accept=0on almost every line, out of hundreds to thousands of calls) because none of them are actually slow —NativeSocketTestBase.doTcpTransfer()'s comment claims the persistent connection's send buffer fills after ~32 iterations and blocks for >1ms, but on this runner's network stack that assumption doesn't hold, so the time-weighted sampler correctly rejects everything as "too fast to be worth a sample."That's a real test-portability bug in
NativeSocketTestBase, unrelated to EL7/glibc-2.17 specifically. Tracked separately for quarantine (ddprof-test/quarantine.txt) now that #777 is merged, rather than worked around here.Test plan
.gitlab/base/el7/Dockerfilebuilds locally and the resulting image runsjava -versionagainst glibc 2.17x64-el7image built and pushed via a real pipeline, digest pinnedstresstest:x64andfunctional:x64-el7-jdk8run in real GitLab pipelines through several rounds of fixes (compiler probe, exec permission, C99, debug/release config, heap) — each fix verified against the actual failure it targetsfunctional:x64-el7-jdk8failure (NativeSocket*) root-caused via a temporary diagnostic build/job pair, now removed🤖 Generated with Claude Code