ci: wire the EL7 functional job into #777's quarantine machinery - #810
Conversation
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 53db933f4d
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
This comment has been minimized.
This comment has been minimized.
There was a problem hiding this comment.
Two critical issues block EL7 functional jobs: the pinned EL7 image does not contain Python 3 (causing immediate job failure when the wrapper is invoked), and the NativeSocket quarantine entries use exact method patterns that never match the Class.[n] IDs written by @RetryingTest, leaving those failures unquarantined. A separate concern is that a UDP test is incorrectly grouped inside a TCP zero-event quarantine block, risking silent suppression of unrelated UDP regressions.
🤖 Datadog Autotest · Commit 53db933 · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest
CI Test ResultsRun: #35983923982 | Commit:
Status Overview
Legend: ✅ passed | ❌ failed | ⚪ skipped | 🚫 cancelled Summary: Total: 32 | Passed: 32 | Failed: 0 Updated: 2026-09-24 10:12:12 UTC |
d1f4b1f to
02217fa
Compare
4285688 to
bcd7b1b
Compare
02217fa to
b4c3b83
Compare
Reliability & Chaos Results❌ 12 failure(s) detected Pipeline: https://gitlab.ddbuild.io/DataDog/java-profiler/-/pipelines/139438691 ❌ profiler tcmalloc aarch64Xmemory❌ profiler tracer tcmalloc amd64Xjit❌ chaos: profiler gmalloc aarch64 21 0 3 temXchaos❌ chaos: profiler gmalloc aarch64 25 0 3 temXchaos❌ chaos: profiler gmalloc amd64 21 0 3 temXchaos❌ chaos: profiler jemalloc amd64 25 0 3 temXchaos❌ chaos: profiler tcmalloc aarch64 21 0 3 temXchaos❌ chaos: profiler tracer gmalloc amd64 25 0 3 temXchaos❌ chaos: profiler tracer jemalloc aarch64 21 0 3 temXchaos❌ chaos: profiler tracer jemalloc aarch64 25 0 3 temXchaos❌ chaos: profiler tracer jemalloc amd64 21 0 3 temXchaos❌ chaos: profiler tracer jemalloc amd64 25 0 3 temXchaos |
bcd7b1b to
cb4d5a1
Compare
bcb69db to
8b799bb
Compare
Follow-up to #777, for wiring the GitLab EL7 functional job into quarantine: EL7's base-repo python3 is 3.6.8 (no SCL/newer-Python repo exists for Oracle Linux 7), and running the real scripts inside that image crashed twice: add_subparsers(dest="command", required=True) TypeError: __init__() got an unexpected keyword argument 'required' datetime.date.fromisoformat(review_by) AttributeError: type object 'datetime.date' has no attribute 'fromisoformat' Both need Python 3.7+. Drops required=True (checking args.command is None manually afterward) and replaces fromisoformat with strptime(s, "%Y-%m-%d").date() everywhere it was called. Verified end-to-end inside the real Oracle Linux 7 image: a quarantined failure on its own cell now goes green under python3 3.6.8, and the existing 59 test_quarantine.sh assertions (which run under whatever python3 is on the host, typically 3.9+) still pass unchanged. Also adds "el7" to KNOWN_LIBCS: the GitLab functional job's Oracle Linux 7 runtime is technically glibc but must not share the "glibc" cell token with the GitHub Actions Ubuntu matrix -- confirmed by a second smoke test that a quarantine entry scoped to el7-8-release-amd64 does not excuse the identical failure on glibc-8-release-amd64. Co-Authored-By: Claude Sonnet 5 <[email protected]>
functional-tests.sh is about to wrap its gradlew invocation in run_tests_with_retry.sh, which shells out to quarantine.py and flake_report.py. Co-Authored-By: Claude Sonnet 5 <[email protected]>
functional:x64-el7-jdk* previously just ran gradlew directly and let its raw exit code decide the job -- ddprof-test/quarantine.txt had no effect on it at all, unlike the GH Actions matrix (#777). Wraps the gradlew invocation in run_tests_with_retry.sh with cell "el7-<jdk>-<config>-amd64". MAX_ATTEMPTS defaults to 1 (no retry) here specifically: the default of 2 means a full second suite run, and this runner has already OOMKilled the container outright once at the current heap/concurrency (PR #805). Losing the flaky-vs-broken distinction is an acceptable trade for not doubling memory pressure on a runner already this tight; revisit once there's more headroom. Extends .functional_job's artifact paths with flake-evidence/, ci-outcome/ and build/logs/attempt.log so a quarantine decision on this job has evidence to show for it. Verified end-to-end inside the real Oracle Linux 7 image with a stub gradlew: a quarantined failure on its own cell now exits 0 through the full functional-tests.sh-shaped invocation. Co-Authored-By: Claude Sonnet 5 <[email protected]>
PROF-16014. Root-caused in PR #805: NativeSocketSampler's hooks install and fire correctly (confirmed via a debug-config diagnostic build), but the rate limiter rejects nearly every sample because NativeSocketTestBase.doTcpTransfer()'s assumption that the send buffer fills and blocks after ~32 iterations does not hold on this runner's network stack. Scoped to el7-8-release-amd64, the only cell this has actually been observed on -- not el7-*-release-amd64, so the nightly el7-17/-21 cells still gate for real if this doesn't reproduce there the same way. Co-Authored-By: Claude Sonnet 5 <[email protected]>
The example line was written to be deleted once the first real entry landed, but it's worth keeping as a permanent format reference -- the NativeSocket entries are tied to a bug that will get fixed and removed, and the file shouldn't go back to having no example once that happens. Co-Authored-By: Claude Sonnet 5 <[email protected]>
Inserting it in the module-level constants block left it wedged between two densely-packed constant groups with no blank-line separation, breaking the file's own two-blank-lines-around-defs convention. Moves it next to is_expired(), its first caller, matching how the file already places its other small helpers near their use. Co-Authored-By: Claude Sonnet 5 <[email protected]>
Flagged in review, and confirmed against both the test source and quarantine.txt's own documented format: every affected class's single test method is @RetryingTest, which -- like @ParameterizedTest and @testtemplate -- reports in JUnit XML as "Class.[1]", "[2]", ... with no method name at all. The exact-method entries could never match anything; only the already-class-wide NativeSocketStackTraceTest.* entry ever worked. Converts the other 8 to <class>.*. Also drops the NativeSocketUdpExcludedTest entry entirely rather than widening it: that test asserts *zero* events, which the bug's own symptom (near-zero events everywhere) would make pass, not fail. Its earlier observed failure has an unconfirmed, likely different cause, and quarantining it on this evidence risks hiding an unrelated regression later. Verified: find_entry() now matches the real "NativeSocketEnabledTest.[1]"-shaped id against the class-wide entry, and confirms the UDP test is not quarantined. All 59 test_quarantine.sh assertions and quarantine.py validate still pass. Co-Authored-By: Claude Sonnet 5 <[email protected]>
Built and pushed successfully in pipeline 139300629 (the job's overall failure was two unrelated images, arm64-musl and benchmarks-amd64). Confirmed python3 3.6.8 installed in the build log. Co-Authored-By: Claude Sonnet 5 <[email protected]>
The x64-el7 functional job was getting OOMKilled (exit 137) at the container level during pod cleanup, after the test suite had already finished and uploaded its artifacts. The JVM under test already has its own -Xmx1536m cap; the pod itself needed more headroom to run Gradle's daemon/workers, the gtest builds, and the JVM together. Co-Authored-By: Claude Sonnet 5 <[email protected]>
The Kotlin compile daemon outlives the build-logic compile and stays resident for the rest of the build, including the whole test run, next to the Gradle process and the test JVMs. Measured on a --no-daemon build: ~580MB Gradle + ~660MB Kotlin daemon with the default strategy, versus ~920MB Gradle and no daemon in-process. The out-of-process strategy was removed in Kotlin 2.4.0. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
…mon" This reverts commit 7864e74.
The Kotlin compile daemon (~660MB RSS) stays resident through the whole test run in this 6GB pod. Setting in-process globally via build-logic/gradle.properties broke the CodeQL Java job: its Kotlin extractor runs inside the compiler, and in-process that exhausted Gradle's default 512m heap. Scope the setting to this job instead. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This reverts commit a794f69.
8b799bb to
92c4a01
Compare
There was a problem hiding this comment.
Two distinct build/CI correctness issues were found: the skip-native=true property is misinterpreted as a project name in PlatformUtils.kt, silently enabling native compilation when it should be skipped; and the functional-test wrapper in functional-tests.sh can replace a SIGKILL exit code (137) with 0, masking suite-level failures in CI.
🤖 Bits Code Review · Commit 8b799bb · @DataDog review to ask questions
Findings that could not be posted inline
build-logic/conventions/src/main/kotlin/com/datadoghq/native/util/PlatformUtils.kt:396
Preserve the documented skip-native=true setting
Existing builds that use prebuilt libraries can fail when the compiler cannot build the native sources.
Assertion details
- Input: Set skip-native=true in gradle.properties, or pass -Pskip-native=true, when assembling a JAR with prebuilt libraries.
- Expected:
Preserve the global skip for skip-native=true, as documented in gradle.properties.template:22. Check this value before parsing project names. - Actual:
The new helper treats "true" as a project name and returns false. JAR assembly now includes native compilation.
Was this helpful? React 👍 or 👎
🤖 Bits Code Review · @DataDog review to ask questions · Open Bits AI session
…a signal A signal that kills the Gradle JVM itself (e.g. the OOM killer taking the --no-daemon build process) leaves nothing alive to print a cut-short marker, and with MAX_ATTEMPTS=1 there is no earlier attempt to measure a shortfall against. If a quarantined test had already written its failure XML, the report excused the run and the wrapper turned exit 137 into 0. Treat an exit status above 128 as the attempt being cut short. Gradle never exits above 128 on its own, however many tests fail. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
gradle.properties.template documents skip-native=true, but isNativeSkipped() only treated an empty value as the global skip and read anything else as a list of project names. "true" matched no project, so native compilation ran anyway. Treat "true" like a bare -Pskip-native, and "false" as skipping nothing. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
|
Re the Bits finding on |
Summary
Follow-up to #777 and #805 (both merged; this builds on
.gitlab/base/el7/Dockerfileandfunctional-tests.shfrom #805).#777 built a flaky-test quarantine mechanism, but wired it only into the GitHub Actions matrix (
run_tests_with_retry.shis only ever invoked from.github/workflows/test_workflow.yml).functional:x64-el7-jdk*(from #805) just rangradlewdirectly and let its raw exit code decide the job —ddprof-test/quarantine.txthad zero effect on it. This wires it in, so theNativeSocket*flake root-caused in #805 (PROF-16014) can actually be quarantined there.What's here
quarantine.py/flake_report.py. Running the real scripts inside the actual Oracle Linux 7 image crashed twice —add_subparsers(..., required=True)anddatetime.date.fromisoformat()both need Python 3.7+, but EL7's base-repopython3is 3.6.8 (there's no SCL/newer-Python repo for Oracle Linux 7 to reach for instead). Both are narrow, backward-compatible fixes; the existing 59test_quarantine.shassertions (which run under whateverpython3is on the host, typically 3.9+) pass unchanged."el7"added toquarantine.py'sKNOWN_LIBCS. The EL7 job is technically glibc, but it's a different environment (container, kernel, network stack) than the GitHub Actions Ubuntu glibc matrix and can fail the same test for unrelated reasons — sharing theglibccell token would let an entry meant to excuse one silently excuse the other. Verified this isolation directly: the same failure quarantined onel7-8-release-amd64still gates onglibc-8-release-amd64.python3added to.gitlab/base/el7/Dockerfile— rebuilt, withBUILD_IMAGE_X64_EL7re-pinned to the new digest in.gitlab-ci.yml.functional-tests.shwraps itsgradlewcall inrun_tests_with_retry.sh, withCELL="el7-${TEST_JDK}-<lowercased TEST_CONFIG>-amd64".MAX_ATTEMPTSdefaults to1(no retry) for this job specifically — the default of 2 means a full second suite run, and this runner has already OOMKilled the container outright at the current heap/concurrency. Losing the flaky-vs-broken distinction seemed like the right trade against not doubling memory pressure; easy to revisit later..functional_job's artifact paths extended withflake-evidence/,ci-outcome/,build/logs/attempt.log— without these a quarantine decision on this job has nothing to show for it besides the console log.ddprof-test/quarantine.txt: 10 entries for theNativeSocket*classes (PROF-16014), scoped toel7-8-release-amd64only — the one cell this has actually been observed on, notel7-*-release-amd64, so the nightlyel7-17/el7-21cells still gate for real if it turns out not to reproduce there the same way.flake_report.pygates a final attempt killed by a signal (exit > 128). WithMAX_ATTEMPTS=1there is no earlier attempt to measure a shortfall against, and a signal that kills the--no-daemonGradle JVM itself (e.g. the OOM killer) leaves nothing alive to log a cut-short marker. A quarantined test that had already written its failure XML used to excuse the run, turning exit 137 into 0. Found by Bits review; regression test added.skip-native=trueis honored as a global native skip (PlatformUtils.isNativeSkipped(), build-logic). Unrelated to quarantine, but flagged by review on this PR: ci: restore EL7 stresstest coverage, add nightly EL7 functional tests #805's project-list form of-Pskip-nativereadtrue(the spellinggradle.properties.templatedocuments) as a project name, so native compilation ran anyway. CI only uses bare-Pskip-nativeor a project list, so no CI job changes behavior.Verified
test_quarantine.sh+ 4test_generate_test_summary.sh+ 2test_prepare_reports.shassertions pass.quarantine.py validatepasses on the new entries.run_tests_with_retry.shend-to-end inside the actual Oracle Linux 7 image (not just locally) against a stub test command, in both directions:el7-8-release-amd64→ exit 0glibc-8-release-amd64→ exit 1 (still gates, confirming cell isolation)NativeSocketEnabledTest) through afunctional-tests.sh-shaped invocation → exit 0skip-nativefix::ddprof-lib:assembleReleaseJar --dry-runwith-Pskip-native=truepulled in the native compile/link/copy tasks before the fix and omits them after; no flag and=falsestill include them,=ddprof-lib,malloc-shimstill omits them.Not done here
flake_summary.py's rendering side) isn't wired up for this job — feasible later via thebootstrap-gh-tools/octo-sts pattern already used elsewhere in.gitlab/, but a separate increment.stresstest:x64doesn't go through this at all — it runs a different Gradle task (runStressTests, not:ddprof-test:test*) with a different output shape; out of scope here.🤖 Generated with Claude Code