Skip to content

ci: wire the EL7 functional job into #777's quarantine machinery - #810

Merged
rkennke merged 15 commits into
mainfrom
ci/el7-quarantine-wiring
Sep 24, 2026
Merged

rkennke merged 15 commits into
mainfrom
ci/el7-quarantine-wiring

Conversation

@rkennke

@rkennke rkennke commented Sep 21, 2026

Copy link
Copy Markdown
Contributor

Summary

Follow-up to #777 and #805 (both merged; this builds on .gitlab/base/el7/Dockerfile and functional-tests.sh from #805).

#777 built a flaky-test quarantine mechanism, but wired it only into the GitHub Actions matrix (run_tests_with_retry.sh is only ever invoked from .github/workflows/test_workflow.yml). functional:x64-el7-jdk* (from #805) just ran gradlew directly and let its raw exit code decide the job — ddprof-test/quarantine.txt had zero effect on it. This wires it in, so the NativeSocket* flake root-caused in #805 (PROF-16014) can actually be quarantined there.

What's here

  1. Python 3.6 compatibility fix in quarantine.py/flake_report.py. Running the real scripts inside the actual Oracle Linux 7 image crashed twice — add_subparsers(..., required=True) and datetime.date.fromisoformat() both need Python 3.7+, but EL7's base-repo python3 is 3.6.8 (there's no SCL/newer-Python repo for Oracle Linux 7 to reach for instead). Both are narrow, backward-compatible fixes; the existing 59 test_quarantine.sh assertions (which run under whatever python3 is on the host, typically 3.9+) pass unchanged.
  2. "el7" added to quarantine.py's KNOWN_LIBCS. The EL7 job is technically glibc, but it's a different environment (container, kernel, network stack) than the GitHub Actions Ubuntu glibc matrix and can fail the same test for unrelated reasons — sharing the glibc cell token would let an entry meant to excuse one silently excuse the other. Verified this isolation directly: the same failure quarantined on el7-8-release-amd64 still gates on glibc-8-release-amd64.
  3. python3 added to .gitlab/base/el7/Dockerfile — rebuilt, with BUILD_IMAGE_X64_EL7 re-pinned to the new digest in .gitlab-ci.yml.
  4. functional-tests.sh wraps its gradlew call in run_tests_with_retry.sh, with CELL="el7-${TEST_JDK}-<lowercased TEST_CONFIG>-amd64". MAX_ATTEMPTS defaults to 1 (no retry) for this job specifically — the default of 2 means a full second suite run, and this runner has already OOMKilled the container outright at the current heap/concurrency. Losing the flaky-vs-broken distinction seemed like the right trade against not doubling memory pressure; easy to revisit later.
  5. .functional_job's artifact paths extended with flake-evidence/, ci-outcome/, build/logs/attempt.log — without these a quarantine decision on this job has nothing to show for it besides the console log.
  6. ddprof-test/quarantine.txt: 10 entries for the NativeSocket* classes (PROF-16014), scoped to el7-8-release-amd64 only — the one cell this has actually been observed on, not el7-*-release-amd64, so the nightly el7-17/el7-21 cells still gate for real if it turns out not to reproduce there the same way.
  7. flake_report.py gates a final attempt killed by a signal (exit > 128). With MAX_ATTEMPTS=1 there is no earlier attempt to measure a shortfall against, and a signal that kills the --no-daemon Gradle JVM itself (e.g. the OOM killer) leaves nothing alive to log a cut-short marker. A quarantined test that had already written its failure XML used to excuse the run, turning exit 137 into 0. Found by Bits review; regression test added.
  8. skip-native=true is honored as a global native skip (PlatformUtils.isNativeSkipped(), build-logic). Unrelated to quarantine, but flagged by review on this PR: ci: restore EL7 stresstest coverage, add nightly EL7 functional tests #805's project-list form of -Pskip-native read true (the spelling gradle.properties.template documents) as a project name, so native compilation ran anyway. CI only uses bare -Pskip-native or a project list, so no CI job changes behavior.

Verified

  • All 61 test_quarantine.sh + 4 test_generate_test_summary.sh + 2 test_prepare_reports.sh assertions pass.
  • quarantine.py validate passes on the new entries.
  • Ran the real run_tests_with_retry.sh end-to-end inside the actual Oracle Linux 7 image (not just locally) against a stub test command, in both directions:
    • a quarantined failure on el7-8-release-amd64 → exit 0
    • the identical failure on glibc-8-release-amd64 → exit 1 (still gates, confirming cell isolation)
    • the exact real quarantine entry (NativeSocketEnabledTest) through a functional-tests.sh-shaped invocation → exit 0
  • skip-native fix: :ddprof-lib:assembleReleaseJar --dry-run with -Pskip-native=true pulled in the native compile/link/copy tasks before the fix and omits them after; no flag and =false still include them, =ddprof-lib,malloc-shim still omits them.

Not done here

  • The paste-ready quarantine-proposal PR comment (flake_summary.py's rendering side) isn't wired up for this job — feasible later via the bootstrap-gh-tools/octo-sts pattern already used elsewhere in .gitlab/, but a separate increment.
  • stresstest:x64 doesn't go through this at all — it runs a different Gradle task (runStressTests, not :ddprof-test:test*) with a different output shape; out of scope here.

🤖 Generated with Claude Code

@rkennke
rkennke requested a review from a team as a code owner September 21, 2026 13:30

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 53db933f4d

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread .gitlab/scripts/functional-tests.sh
Comment thread ddprof-test/quarantine.txt Outdated
Comment thread ddprof-test/quarantine.txt Outdated
@datadog-prod-us1-3

This comment has been minimized.

@datadog-prod-us1-3 datadog-prod-us1-3 Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Datadog Autotest: FAIL

Two critical issues block EL7 functional jobs: the pinned EL7 image does not contain Python 3 (causing immediate job failure when the wrapper is invoked), and the NativeSocket quarantine entries use exact method patterns that never match the Class.[n] IDs written by @RetryingTest, leaving those failures unquarantined. A separate concern is that a UDP test is incorrectly grouped inside a TCP zero-event quarantine block, risking silent suppression of unrelated UDP regressions.

Open Bits AI session

🤖 Datadog Autotest · Commit 53db933 · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest

Comment thread .gitlab/scripts/functional-tests.sh
Comment thread ddprof-test/quarantine.txt Outdated
Comment thread ddprof-test/quarantine.txt Outdated
@dd-octo-sts

dd-octo-sts Bot commented Sep 21, 2026

Copy link
Copy Markdown
Contributor

CI Test Results

Run: #35983923982 | Commit: 06fde44 | Duration: 15m 29s (longest job)

All 32 test jobs passed

Status Overview

JDK glibc-aarch64/debug glibc-amd64/debug musl-aarch64/debug musl-amd64/debug
8 - - -
8-ibm - - -
8-j9 - -
8-librca - -
8-orcl - - -
11 - - -
11-j9 - -
11-librca - -
17 - -
17-graal - -
17-j9 - -
17-librca - -
21 - -
21-graal - -
21-librca - -
25 - -
25-graal - -
25-librca - -

Legend: ✅ passed | ❌ failed | ⚪ skipped | 🚫 cancelled

Summary: Total: 32 | Passed: 32 | Failed: 0


Updated: 2026-09-24 10:12:12 UTC

@dd-octo-sts

dd-octo-sts Bot commented Sep 21, 2026

Copy link
Copy Markdown
Contributor

All 40 integration tests passed

📊 Dashboard · 👷 Pipeline · 📦 090bafbd

@rkennke
rkennke force-pushed the ci/el7-quarantine-wiring branch 2 times, most recently from d1f4b1f to 02217fa Compare September 22, 2026 15:01
@rkennke
rkennke force-pushed the ci/el7-stresstest-functional-coverage branch from 4285688 to bcd7b1b Compare September 22, 2026 18:39
@rkennke
rkennke force-pushed the ci/el7-quarantine-wiring branch from 02217fa to b4c3b83 Compare September 22, 2026 18:39

@kaahos kaahos left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me!

@dd-octo-sts

dd-octo-sts Bot commented Sep 23, 2026

Copy link
Copy Markdown
Contributor

Reliability & Chaos Results

12 failure(s) detected Pipeline: https://gitlab.ddbuild.io/DataDog/java-profiler/-/pipelines/139438691

❌ profiler tcmalloc aarch64Xmemory
Could not fetch ddprof 1.51.0-ci_el7-quarantine-wiring-SNAPSHOT jar
❌ profiler tracer tcmalloc amd64Xjit
Could not fetch ddprof 1.51.0-ci_el7-quarantine-wiring-SNAPSHOT jar
❌ chaos: profiler gmalloc aarch64 21 0 3 temXchaos
chaos.jar unavailable
❌ chaos: profiler gmalloc aarch64 25 0 3 temXchaos
chaos.jar unavailable
❌ chaos: profiler gmalloc amd64 21 0 3 temXchaos
chaos.jar unavailable
❌ chaos: profiler jemalloc amd64 25 0 3 temXchaos
chaos.jar unavailable
❌ chaos: profiler tcmalloc aarch64 21 0 3 temXchaos
chaos.jar unavailable
❌ chaos: profiler tracer gmalloc amd64 25 0 3 temXchaos
chaos.jar unavailable
❌ chaos: profiler tracer jemalloc aarch64 21 0 3 temXchaos
chaos.jar unavailable
❌ chaos: profiler tracer jemalloc aarch64 25 0 3 temXchaos
chaos.jar unavailable
❌ chaos: profiler tracer jemalloc amd64 21 0 3 temXchaos
chaos.jar unavailable
❌ chaos: profiler tracer jemalloc amd64 25 0 3 temXchaos
ddprof jar unavailable (Maven snapshot download failed)

@rkennke
rkennke force-pushed the ci/el7-stresstest-functional-coverage branch from bcd7b1b to cb4d5a1 Compare September 24, 2026 08:12
@rkennke
rkennke force-pushed the ci/el7-quarantine-wiring branch from bcb69db to 8b799bb Compare September 24, 2026 08:13
Base automatically changed from ci/el7-stresstest-functional-coverage to main September 24, 2026 08:54
rkennke and others added 13 commits September 24, 2026 10:55
Follow-up to #777, for wiring the GitLab EL7 functional job into
quarantine: EL7's base-repo python3 is 3.6.8 (no SCL/newer-Python repo
exists for Oracle Linux 7), and running the real scripts inside that
image crashed twice:

  add_subparsers(dest="command", required=True)
  TypeError: __init__() got an unexpected keyword argument 'required'

  datetime.date.fromisoformat(review_by)
  AttributeError: type object 'datetime.date' has no attribute 'fromisoformat'

Both need Python 3.7+. Drops required=True (checking args.command is
None manually afterward) and replaces fromisoformat with
strptime(s, "%Y-%m-%d").date() everywhere it was called. Verified
end-to-end inside the real Oracle Linux 7 image: a quarantined
failure on its own cell now goes green under python3 3.6.8, and the
existing 59 test_quarantine.sh assertions (which run under whatever
python3 is on the host, typically 3.9+) still pass unchanged.

Also adds "el7" to KNOWN_LIBCS: the GitLab functional job's Oracle
Linux 7 runtime is technically glibc but must not share the "glibc"
cell token with the GitHub Actions Ubuntu matrix -- confirmed by a
second smoke test that a quarantine entry scoped to
el7-8-release-amd64 does not excuse the identical failure on
glibc-8-release-amd64.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
functional-tests.sh is about to wrap its gradlew invocation in
run_tests_with_retry.sh, which shells out to quarantine.py and
flake_report.py.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
functional:x64-el7-jdk* previously just ran gradlew directly and let
its raw exit code decide the job -- ddprof-test/quarantine.txt had no
effect on it at all, unlike the GH Actions matrix (#777). Wraps the
gradlew invocation in run_tests_with_retry.sh with cell
"el7-<jdk>-<config>-amd64".

MAX_ATTEMPTS defaults to 1 (no retry) here specifically: the default
of 2 means a full second suite run, and this runner has already
OOMKilled the container outright once at the current heap/concurrency
(PR #805). Losing the flaky-vs-broken distinction is an acceptable
trade for not doubling memory pressure on a runner already this
tight; revisit once there's more headroom.

Extends .functional_job's artifact paths with flake-evidence/,
ci-outcome/ and build/logs/attempt.log so a quarantine decision on
this job has evidence to show for it.

Verified end-to-end inside the real Oracle Linux 7 image with a stub
gradlew: a quarantined failure on its own cell now exits 0 through
the full functional-tests.sh-shaped invocation.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
PROF-16014. Root-caused in PR #805: NativeSocketSampler's hooks
install and fire correctly (confirmed via a debug-config diagnostic
build), but the rate limiter rejects nearly every sample because
NativeSocketTestBase.doTcpTransfer()'s assumption that the send
buffer fills and blocks after ~32 iterations does not hold on this
runner's network stack.

Scoped to el7-8-release-amd64, the only cell this has actually been
observed on -- not el7-*-release-amd64, so the nightly el7-17/-21
cells still gate for real if this doesn't reproduce there the same
way.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
The example line was written to be deleted once the first real entry
landed, but it's worth keeping as a permanent format reference -- the
NativeSocket entries are tied to a bug that will get fixed and
removed, and the file shouldn't go back to having no example once
that happens.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Inserting it in the module-level constants block left it wedged
between two densely-packed constant groups with no blank-line
separation, breaking the file's own two-blank-lines-around-defs
convention. Moves it next to is_expired(), its first caller, matching
how the file already places its other small helpers near their use.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Flagged in review, and confirmed against both the test source and
quarantine.txt's own documented format: every affected class's single
test method is @RetryingTest, which -- like @ParameterizedTest and
@testtemplate -- reports in JUnit XML as "Class.[1]", "[2]", ... with
no method name at all. The exact-method entries could never match
anything; only the already-class-wide NativeSocketStackTraceTest.*
entry ever worked. Converts the other 8 to <class>.*.

Also drops the NativeSocketUdpExcludedTest entry entirely rather than
widening it: that test asserts *zero* events, which the bug's own
symptom (near-zero events everywhere) would make pass, not fail. Its
earlier observed failure has an unconfirmed, likely different cause,
and quarantining it on this evidence risks hiding an unrelated
regression later.

Verified: find_entry() now matches the real
"NativeSocketEnabledTest.[1]"-shaped id against the class-wide entry,
and confirms the UDP test is not quarantined. All 59
test_quarantine.sh assertions and quarantine.py validate still pass.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Built and pushed successfully in pipeline 139300629 (the job's
overall failure was two unrelated images, arm64-musl and
benchmarks-amd64). Confirmed python3 3.6.8 installed in the build log.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
The x64-el7 functional job was getting OOMKilled (exit 137) at the
container level during pod cleanup, after the test suite had already
finished and uploaded its artifacts. The JVM under test already has
its own -Xmx1536m cap; the pod itself needed more headroom to run
Gradle's daemon/workers, the gtest builds, and the JVM together.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
The Kotlin compile daemon outlives the build-logic compile and stays
resident for the rest of the build, including the whole test run, next to
the Gradle process and the test JVMs. Measured on a --no-daemon build:
~580MB Gradle + ~660MB Kotlin daemon with the default strategy, versus
~920MB Gradle and no daemon in-process. The out-of-process strategy was
removed in Kotlin 2.4.0.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
The Kotlin compile daemon (~660MB RSS) stays resident through the whole
test run in this 6GB pod. Setting in-process globally via
build-logic/gradle.properties broke the CodeQL Java job: its Kotlin
extractor runs inside the compiler, and in-process that exhausted
Gradle's default 512m heap. Scope the setting to this job instead.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
@rkennke
rkennke force-pushed the ci/el7-quarantine-wiring branch from 8b799bb to 92c4a01 Compare September 24, 2026 08:56
Comment thread .gitlab/scripts/functional-tests.sh

@datadog-prod-us1-3 datadog-prod-us1-3 Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bits Code Review: FAIL

Two distinct build/CI correctness issues were found: the skip-native=true property is misinterpreted as a project name in PlatformUtils.kt, silently enabling native compilation when it should be skipped; and the functional-test wrapper in functional-tests.sh can replace a SIGKILL exit code (137) with 0, masking suite-level failures in CI.

Open Bits AI session

🤖 Bits Code Review · Commit 8b799bb · @DataDog review to ask questions


Findings that could not be posted inline

build-logic/conventions/src/main/kotlin/com/datadoghq/native/util/PlatformUtils.kt:396

P2 Preserve the documented skip-native=true setting

Existing builds that use prebuilt libraries can fail when the compiler cannot build the native sources.

Assertion details
  • Input: Set skip-native=true in gradle.properties, or pass -Pskip-native=true, when assembling a JAR with prebuilt libraries.
  • Expected: Preserve the global skip for skip-native=true, as documented in gradle.properties.template:22. Check this value before parsing project names.
  • Actual: The new helper treats "true" as a project name and returns false. JAR assembly now includes native compilation.

Was this helpful? React 👍 or 👎
🤖 Bits Code Review · @DataDog review to ask questions · Open Bits AI session

rkennke and others added 2 commits September 24, 2026 11:33
…a signal

A signal that kills the Gradle JVM itself (e.g. the OOM killer taking the
--no-daemon build process) leaves nothing alive to print a cut-short marker,
and with MAX_ATTEMPTS=1 there is no earlier attempt to measure a shortfall
against. If a quarantined test had already written its failure XML, the
report excused the run and the wrapper turned exit 137 into 0.

Treat an exit status above 128 as the attempt being cut short. Gradle never
exits above 128 on its own, however many tests fail.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
gradle.properties.template documents skip-native=true, but isNativeSkipped()
only treated an empty value as the global skip and read anything else as a
list of project names. "true" matched no project, so native compilation ran
anyway. Treat "true" like a bare -Pskip-native, and "false" as skipping
nothing.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
@rkennke

rkennke commented Sep 24, 2026

Copy link
Copy Markdown
Contributor Author

Re the Bits finding on PlatformUtils.kt:396 (skip-native=true read as a project name): confirmed and fixed in 090bafb. isNativeSkipped() now treats skip-native=true like a bare -Pskip-native (skip everywhere) and false as skipping nothing; the comma-separated project-list form is unchanged. Checked with :ddprof-lib:assembleReleaseJar --dry-run: with -Pskip-native=true the native compile/link/copy tasks were in the graph before the fix and are gone after. No CI job changes behavior, since CI only uses bare -Pskip-native or a project list.

@rkennke
rkennke merged commit 5e5b208 into main Sep 24, 2026
112 checks passed
@rkennke
rkennke deleted the ci/el7-quarantine-wiring branch September 24, 2026 10:40
@github-actions github-actions Bot added this to the 1.51.0 milestone Sep 24, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants