Skip to content

Detect scope and continuation leaks in instrumentation tests - #12458

Open
amarziali wants to merge 21 commits into
masterfrom
andrea.marziali/diag2
Open

amarziali wants to merge 21 commits into
masterfrom
andrea.marziali/diag2

Conversation

@amarziali

@amarziali amarziali commented Sep 11, 2026

Copy link
Copy Markdown
Contributor

What Does This Do

Adds default-on scope and continuation lifecycle diagnostics to instrumentation tests.

The diagnostic records continuation and scope events during each test, waits briefly for asynchronous cleanup, and fails with an actionable timeline when it detects problems such as:

  • A continuation that was never released
  • A scope that was never closed
  • A continuation resolved more than once
  • Activation after resolution
  • Cleanup performed from an unexpected thread
  • Cleanup occurring after the test boundary

The timeline includes elapsed time, thread transitions, ownership state, and relevant call sites so the missing lifecycle edge can be identified without reconstructing it from a trace timeout.

  flowchart LR
      A[Capture continuation] --> B[Resume context]
      B --> C[Run asynchronous work]
      C --> D[Close scope]
      D --> E[Release continuation]
      E --> F[Write complete trace]

      C -. missing cleanup .-> G[Test ends]
      G --> H[Diagnostic failure]
      H --> I[Timeline with thread and callsite]

      style E fill:#d5f5e3,stroke:#239b56
      style H fill:#fadbd8,stroke:#c0392b
Loading

How is it enabled?

The diagnostic runs automatically for both instrumentation-test harnesses:

  • Spock tests extending InstrumentationSpecification
  • JUnit tests extending AbstractInstrumentationTest

A narrow opt-out is available only for tests that are incompatible with the diagnostic itself:

@TrackScopeContinuations(
enabled = false,
reason = "Explain the incompatibility and removal condition")

The reason is mandatory. The opt-out is not intended for unresolved continuation leaks.

Additional behavior

  • Allows a bounded quiescence period for legitimate asynchronous cleanup.
  • Recognizes iteration scopes registered for deferred cleanup without ignoring unrelated iteration-scope leaks.
  • Keeps recording isolated to the test harness.
  • Preserves an original test failure while attaching diagnostic failures as additional evidence.
  • Includes focused tests for lifecycle classification, reporting, configuration, stack filtering, and concurrent snapshots.
  • Adds a troubleshooting skill that guides an LLM from the emitted timeline to the broken ownership edge and a compact human-readable diagram.

Motivation

Strict trace writing can reveal that asynchronous work is still pending, but it does not explain where ownership was lost.

This diagnostic connects capture, activation, scope closure, and continuation resolution into one timeline, making continuation failures reproducible and actionable for both humans and automated integration work.

Fixed issues leveraging that:

Additional Notes

Contributor Checklist

Jira ticket: [PROJ-IDENT]

@amarziali amarziali added the tag: do not merge Do not merge changes label Sep 11, 2026
@datadog-datadog-us1-prod

This comment has been minimized.

@dd-octo-sts

dd-octo-sts Bot commented Sep 11, 2026

Copy link
Copy Markdown
Contributor

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite Status
Startup 🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results
Scenario Candidate master Δ (95% CI of mean)
startup:insecure-bank:iast:Agent 14.75 s 14.68 s [-0.4%; +1.3%] (no difference)
startup:insecure-bank:tracing:Agent 13.54 s 13.79 s [-2.7%; -0.9%] (maybe better)
startup:petclinic:appsec:Agent 17.52 s 17.48 s [-0.5%; +1.1%] (no difference)
startup:petclinic:iast:Agent 17.33 s 17.50 s [-1.7%; -0.3%] (maybe better)
startup:petclinic:profiling:Agent 17.52 s 17.37 s [-0.2%; +1.9%] (no difference)
startup:petclinic:sca:Agent 17.59 s 16.65 s [+1.3%; +9.9%] (significantly worse)
startup:petclinic:tracing:Agent 16.59 s 16.66 s [-1.4%; +0.4%] (no difference)

Commit: 9b23b489 · CI Pipeline · Benchmarking Platform UI


Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

@amarziali
amarziali added this pull request to stack #12465 September 11, 2026 13:34
@amarziali
amarziali removed this pull request from stack #12465 September 11, 2026 13:39
@amarziali
amarziali force-pushed the andrea.marziali/diag2 branch from aa21edf to 506a231 Compare September 14, 2026 12:33
@amarziali
amarziali force-pushed the andrea.marziali/diag2 branch from 4bfc4b6 to 113957f Compare September 15, 2026 15:20
@pr-commenter

pr-commenter Bot commented Sep 15, 2026

Copy link
Copy Markdown

Kafka / consumer-benchmark

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master andrea.marziali/diag2
git_commit_date 1790036134 1790164547
git_commit_sha dd9d42a 9b23b48
See matching parameters
Baseline Candidate
ci_job_date 1790165817 1790165817
ci_job_id 2071553460 2071553460
ci_pipeline_id 139452567 139452567
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
jdkVersion 11.0.31 11.0.31
jmhVersion 1.36 1.36
jvm /usr/lib/jvm/java-11-openjdk-amd64/bin/java /usr/lib/jvm/java-11-openjdk-amd64/bin/java
jvmArgs -Dhttp.proxyHost=127.0.0.1 -Dhttp.proxyPort=15002 -Dhttps.proxyHost=127.0.0.1 -Dhttps.proxyPort=15002 -Dhttp.nonProxyHosts=localhost *.localhost
kernel_version Linux runner-zfyrx7zua-project-304-concurrent-0-1x5u9brw 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux Linux runner-zfyrx7zua-project-304-concurrent-0-1x5u9brw 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
vmName OpenJDK 64-Bit Server VM OpenJDK 64-Bit Server VM
vmVersion 11.0.31+11-post-1ubuntu1-22.04.2-Ubuntu 11.0.31+11-post-1ubuntu1-22.04.2-Ubuntu

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 3 metrics, 0 unstable metrics.

See unchanged results
scenario Δ mean throughput
scenario:not-instrumented/KafkaConsumerBenchmark.benchConsume same
scenario:only-tracing-dsm-disabled-benchmarks/KafkaConsumerBenchmark.benchConsume same
scenario:only-tracing-dsm-enabled-benchmarks/KafkaConsumerBenchmark.benchConsume unsure
[+1017.540op/s; +7003.807op/s] or [+0.541%; +3.721%]

@amarziali
amarziali force-pushed the andrea.marziali/diag2 branch from e7e6409 to b7fa1e8 Compare September 16, 2026 09:27
@pr-commenter

pr-commenter Bot commented Sep 16, 2026

Copy link
Copy Markdown

Kafka / producer-benchmark

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master andrea.marziali/diag2
git_commit_date 1790036134 1790164547
git_commit_sha dd9d42a 9b23b48
See matching parameters
Baseline Candidate
ci_job_date 1790165776 1790165776
ci_job_id 2071553457 2071553457
ci_pipeline_id 139452567 139452567
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
jdkVersion 11.0.31 11.0.31
jmhVersion 1.36 1.36
jvm /usr/lib/jvm/java-11-openjdk-amd64/bin/java /usr/lib/jvm/java-11-openjdk-amd64/bin/java
jvmArgs -Dhttp.proxyHost=127.0.0.1 -Dhttp.proxyPort=15002 -Dhttps.proxyHost=127.0.0.1 -Dhttps.proxyPort=15002 -Dhttp.nonProxyHosts=localhost *.localhost
kernel_version Linux runner-zfyrx7zua-project-304-concurrent-0-803udo3y 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux Linux runner-zfyrx7zua-project-304-concurrent-0-803udo3y 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
vmName OpenJDK 64-Bit Server VM OpenJDK 64-Bit Server VM
vmVersion 11.0.31+11-post-1ubuntu1-22.04.2-Ubuntu 11.0.31+11-post-1ubuntu1-22.04.2-Ubuntu

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 3 metrics, 0 unstable metrics.

See unchanged results
scenario Δ mean throughput
scenario:not-instrumented/KafkaProduceBenchmark.benchProduce unsure
[+1740.802op/s; +56354.249op/s] or [+0.096%; +3.105%]
scenario:only-tracing-dsm-disabled-benchmarks/KafkaProduceBenchmark.benchProduce same
scenario:only-tracing-dsm-enabled-benchmarks/KafkaProduceBenchmark.benchProduce same

@amarziali
amarziali added this pull request to stack #12526 September 16, 2026 14:11
@amarziali amarziali changed the title Test scope diagnostics Detect scope and continuation leaks in instrumentation tests Sep 16, 2026
@amarziali
amarziali force-pushed the andrea.marziali/diag2 branch from 13947ba to c034c2c Compare September 18, 2026 11:55
@amarziali
amarziali marked this pull request as ready for review September 18, 2026 13:25
@amarziali
amarziali requested review from a team as code owners September 18, 2026 13:25
@amarziali
amarziali requested review from dougqh and jordan-wong and removed request for a team September 18, 2026 13:25
@dd-octo-sts

dd-octo-sts Bot commented Sep 18, 2026

Copy link
Copy Markdown
Contributor

Hi! 👋 Thanks for your pull request! 🎉

To help us review it, please make sure to:

  • Add at least one type, and one component or instrumentation label to the pull request

If you need help, please check our contributing guidelines.

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Sep 18, 2026

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review Completed 2026-09-22T08:09:17.044780Z 13ee8f8 Manual request
🔒 Security Review Completed 2026-09-22T08:06:28.421867Z 13ee8f8 Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@amarziali amarziali added type: feature Enhancements and improvements comp: testing Testing tag: no release notes Changes to exclude from release notes and removed tag: do not merge Do not merge changes labels Sep 18, 2026

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: c034c2c2b7

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread .agents/skills/fix-continuation-leakage/SKILL.md Outdated
Comment thread .agents/skills/fix-continuation-leakage/SKILL.md Outdated
@amarziali
amarziali force-pushed the andrea.marziali/diag2 branch from 7dcb090 to 9549280 Compare September 22, 2026 06:46
@amarziali

Copy link
Copy Markdown
Contributor Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

🛡️ Codex Security Review · Automatically triggered

Security review completed. No security issues were found in this pull request.

Reviewed commit: 9549280e8f

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 9549280e8f

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@amarziali

Copy link
Copy Markdown
Contributor Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

🛡️ Codex Security Review · Automatically triggered

Security review completed. No security issues were found in this pull request.

Reviewed commit: beed56821b

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: beed56821b

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@amarziali

Copy link
Copy Markdown
Contributor Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

🛡️ Codex Security Review · Automatically triggered

Security review completed. No security issues were found in this pull request.

Reviewed commit: 13ee8f8522

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 13ee8f8522

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@dougqh dougqh left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Generated with Claude Code

@dougqh

dougqh commented Sep 22, 2026

Copy link
Copy Markdown
Contributor

🤖 Generated with Claude Code

General concern separate from the inline nitpicks: "strict trace writes" isn't a hard guarantee the tracer can enforce everywhere — it depends on async completion racing against root-span close, which the tracer doesn't control for every integration. The old useStrictTraceWrites() { return false } overrides on Cassandra/Grizzly/Netty (and others) were explicit admissions of that for specific integrations with documented TODO/FIXME reasons.

This PR deletes those overrides outright rather than replacing them with the new @TrackScopeContinuations(enabled=false, reason=...) opt-out, so those known-non-strict cases lose their breadcrumb. The new leak detector is still valuable for integrations that genuinely can be made stricter and weren't checked before — but we shouldn't expect or require every test to pass under strict mode as a result of this change; some of the removed overrides likely need to be reinstated (via the new opt-out, with the original reasons preserved) rather than silently dropped.

🤖 Generated with Claude Code

@dougqh dougqh left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Generated with Claude Code

}
}

private static boolean isNotOnTop(Object scope) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wrong-thread-close detection (isNotOnTop, and the scopeSourceOf/continuationOf helpers it shares the pattern with) relies entirely on cached reflective Field/Method lookups against unqualified private/package-private members (scopeManager, scopeStack(), checkTop), with every failure path swallowed by catch (Throwable ignored) { return false; } and the null/failed result cached in a static field forever.

If ContinuableScope.scopeManager, ContinuableScopeManager.scopeStack(), or ScopeStack.checkTop are ever renamed or refactored, the cached lookup permanently fails for the rest of the JVM's lifetime, so the close-wrong-thread diagnostic silently stops firing for every subsequent test with no exception, log, or test failure — defeating one of this PR's stated detection goals without any signal that it broke.

🤖 Generated with Claude Code

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The current tests check the reflective members and verify that wrong-thread events, source metadata, and continuation links are actually recorded. Failed lookups also retry rather than being cached permanently.

These are CI regression guards, not per-test startup validation—my earlier “fail-fast” wording was imprecise. I consider this sufficient for this PR and propose resolving the thread.

@dougqh dougqh left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Generated with Claude Code

@amarziali
amarziali requested a review from dougqh September 23, 2026 10:51
@amarziali

Copy link
Copy Markdown
Contributor Author

@dougqh Regarding your general concern:

These are separate controls: disabling continuation tracking does not disable strict trace writes, so the annotation wouldn’t replace the removed overrides.

The diagnostic helped identify why those tests previously needed strict writes disabled. We fixed the underlying instrumentation and test-lifecycle issues in separate PRs, making those overrides unnecessary rather than ignoring their original reasons. Async work can legitimately outlive root-span closure; correctly captured continuations keep the trace pending.

Longer term, my intention is to remove the strict-write escape hatch from instrumentation tests entirely, so lifecycle problems are investigated rather than hidden. That’s a separate change, outside this PR. Any remaining demonstrated limitation should still be assessed and documented individually.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp: testing Testing tag: no release notes Changes to exclude from release notes type: feature Enhancements and improvements

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants