Drop APM traces when APM tracing is disabled and keep LLM Observability working by ncybul · Pull Request #12473 · DataDog/dd-trace-java

ncybul · 2026-09-11T19:20:37Z

What Does This Do

Two fixes for running LLM Observability without APM tracing (solves #10051):

DD_APM_TRACING_ENABLED=false now drops APM traces. The standalone sampler was only selected when ASM/IAST/SCA was enabled; every other configuration — including LLM Observability — fell through to the default sampler, which keeps traces, so APM traces kept flowing. On distributed traces the drop was defeated a second way: an extracted sampling priority is locked onto the span context at construction, so a request arriving with x-datadog-sampling-priority: 1 silenced the drop before it was ever decided locally.
DD_TRACE_ENABLED=false with DD_LLMOBS_ENABLED=true no longer throws. LLMObsSystem.start() installed the real SDK implementation even with no tracer installed, and AgentTracer.get().buildSpan() returns null on the no-op tracer, so every LLMObs.start*Span() call threw a NullPointerException out of application code.

Motivation

DD_APM_TRACING_ENABLED=false is documented as working in all SDKs. LLM Observability users who don't want APM tracing had no working way to turn it off: the APM flag was a no-op, and DD_TRACE_ENABLED=false broke the LLMObs SDK.

Additional Notes

Approach. This is a reduced port of #10989, which had gone stale. That PR introduced a ProductTraceSource.LLMOBS bit plus a StandaloneProduct enum and a unified StandaloneSampler replacing AsmStandaloneSampler. Two things changed since: 0x20 is now taken by AI_GUARD, and no other tracer allocates an LLMObs _dd.p.ts bit — in dd-trace-js the products are exactly APM/ASM/DSM/DJM/DBM/AI_GUARD, and in dd-trace-py exactly the same set. LLM Observability doesn't need one: its spans ship to the LLM Observability intake (/evp_proxy/v2/api/v2/llmobs), which is independent of APM sampling priority, so the correct behavior is to drop the APM traces and let the LLMObs events through.

That leaves the fix as the selection logic that other tracers already have — compare getProductRateLimiter in dd-trace-js (1/minute for appsec/iast, RateLimiter(0), i.e. drop-all, otherwise) and LLMObsProcessor.process_trace in dd-trace-py (returns None for the whole APM trace when APM tracing is disabled):

ASM enabled → unchanged: AsmStandaloneSampler, 1 APM trace per minute for service catalog / billing.
otherwise → ForcePrioritySampler(SAMPLER_DROP, DEFAULT).

Because a DEFAULT-mechanism priority cannot overwrite a priority that is already set, products that force-keep their own traces are unaffected — AI Guard (ai_guard.keep) and ASM (asm.keep) keep working exactly as before, which AIGuardSamplingTest covers.

Inherited sampling priorities. That same lock is what defeated the drop on distributed traces: DDSpanContext locks an extracted priority at construction, and DEFAULT is not one of the mechanisms SamplingMechanism.canAvoidSamplingPriorityLock exempts, so the drop-all vote was silently refused on exactly the traces the flag exists to suppress. ContextInterpreter.samplingPriorityOrDefault now leaves the extracted priority UNSET when APM tracing is disabled and no product claimed the trace (no _dd.p.ts), letting the local sampler decide. This is the same shape as dd-trace-js' standalone onSpanExtract hook (packages/dd-trace/src/standalone/index.js), which resets the upstream priority under the same condition.

It sits in ContextInterpreter because that is the one point every propagation style funnels through, so Datadog, W3C, B3, Haystack and XRay headers are all covered — including Haystack, whose defaultSamplingPriority() override manufactures a SAMPLER_KEEP out of headers that carry no priority at all.

Gating on _dd.p.ts is what keeps this from discarding a decision another product asked for: ASM- and AI-Guard-marked traces are already exempt from sampler override in TraceCollector.setSamplingPriorityIfNecessary, and they keep their upstream priority here too.

ASM standalone behavior is unchanged. AsmStandaloneSampler votes with SamplingMechanism.APPSEC, which canAvoidSamplingPriorityLock already exempts when APM tracing is disabled — and on that path DDSpanContext assigns rather than compare-and-swaps — so it already overrode an inherited priority before this change and reaches the same decision after it.

Testing.

SamplerTest: APM traces are dropped when APM tracing is disabled, with and without LLM Observability enabled — and the dropped trace is still handed to the writers, which is what keeps LLM Observability spans flowing to their own intake.
LLMObsSystemTest: the SDK stays no-op when tracing is disabled.
DatadogHttpExtractorTest.ApmTracingDisabled: an upstream SAMPLER_KEEP is dropped when no product marked the trace, and preserved when _dd.p.ts is set.
Existing AIGuardSamplingTest, ApmTracingDisabledChunkMarkerTest and the apm-tracing-disabled smoke tests pass unchanged.

Contributor Checklist

Format the title according to the contribution guidelines
Assign the type: and (comp: or inst:) labels in addition to any other useful labels
Avoid using close, fix, or any linking keywords when referencing an issue
Update the CODEOWNERS file on source file addition, migration, or deletion — not needed, all touched paths are already covered
Update public documentation with any new configuration flags or behaviors

Claude session: ba8ff7f0-b73e-41d6-a806-3603ab1c7329
Resume: claude --resume ba8ff7f0-b73e-41d6-a806-3603ab1c7329

🤖 Generated with Claude Code

…ty working Selecting a standalone sampler only when ASM/IAST/SCA is enabled left every other DD_APM_TRACING_ENABLED=false configuration on the default sampler, which keeps traces, so APM traces kept flowing. Drop them instead, as dd-trace-js and dd-trace-py do; LLM Observability spans are unaffected because they ship to the LLM Observability intake, which is independent of APM sampling priority. DD_TRACE_ENABLED=false installed the real LLM Observability SDK with no tracer to back it, and the no-op tracer's buildSpan() returns null, so every LLMObs.start*Span() call threw out of application code. Keep the SDK no-op. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

dd-octo-sts · 2026-09-11T19:43:42Z

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite	Status
Startup	🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results

Scenario	Candidate	master	Δ (95% CI of mean)
startup:insecure-bank:iast:Agent	14.03 s	13.94 s	[-0.5%; +1.8%] (no difference)
startup:insecure-bank:tracing:Agent	13.01 s	13.05 s	[-1.1%; +0.5%] (no difference)
startup:petclinic:appsec:Agent	16.97 s	17.01 s	[-0.9%; +0.4%] (no difference)
startup:petclinic:iast:Agent	16.94 s	16.88 s	[-0.3%; +1.0%] (no difference)
startup:petclinic:profiling:Agent	16.63 s	16.92 s	[-3.1%; -0.3%] (maybe better)
startup:petclinic:sca:Agent	16.76 s	16.80 s	[-1.2%; +0.7%] (no difference)
startup:petclinic:tracing:Agent	16.08 s	16.00 s	[-0.3%; +1.3%] (no difference)

Commit: a2a2a57d · CI Pipeline · Benchmarking Platform UI

Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

Co-Authored-By: Claude Opus 5 <[email protected]>

The two new Groovy smoke tests would have needed the `tag: override groovy enforcement` label. What they proved end-to-end — a dropped APM trace is still handed to the writers, which is how LLM Observability spans still reach their intake — is now asserted in SamplerTest instead. Co-Authored-By: Claude Opus 5 <[email protected]>

sabrenner

makes sense for agent obs, probably just needs java tracer signoff!

datadog-datadog-prod-us1

A sampled parent can lock a keep priority before the new sampler runs. The sampler then fails to drop that normal APM trace when APM tracing is off.

_{Open Bits AI session}

_{🤖 Datadog Autotest · Commit 5923093 · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest}

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 592309351a

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

chatgpt-codex-connector · 2026-09-15T18:08:09Z

+          if (isAsmEnabled(config)) {
+            log.debug(
+                "APM tracing is disabled, but ASM is enabled. Only 1 APM trace per minute will be sent.");
+            return new AsmStandaloneSampler(Clock.systemUTC());


Preserve standalone ASM sampling after remote activation

When DD_APM_TRACING_ENABLED=false and AppSec starts in its default ENABLED_INACTIVE state, isAsmEnabled(config) is false, so this permanently selects the drop-all sampler. Remote Configuration can later activate AppSec, but it changes AppSecSystem state rather than the immutable Config activation used here, and the sampler is not rebuilt; consequently non-event traces remain dropped instead of providing the one-trace-per-minute service-catalog and billing signal required by standalone ASM. The sampler selection needs to accommodate runtime AppSec activation rather than treating ENABLED_INACTIVE as permanently off.

Useful? React with 👍 / 👎.

I replaced that branch with the new ApmTracingDisabledSampler, which constructs both samplers up front and picks between them per sampling decision based on ActiveSubsystems.APPSEC_ACTIVE.

mhlidd

One comment about expected behavior

…d no product marked the trace Co-Authored-By: Claude Opus 5 <[email protected]>

Co-Authored-By: Claude Opus 5 <[email protected]>

mhlidd

LGTM following addressing Codex and Copilot comments 👍

Co-Authored-By: Claude Opus 5 <[email protected]>

datadog-datadog-prod-us1

The new drop sampler cannot replace manual keep decisions. APM traces can still leave the process when APM tracing is off. Remote AppSec activation and shutdown after single-span sampling also have trace-loss paths.

_{Open Bits AI session}

_{🤖 Datadog Autotest · Commit fe764e7 · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest}

datadog-datadog-prod-us1 · 2026-09-18T17:28:16Z

+          // catalog, so with it off we can drop every APM trace. Traces a product does want
+          // (asm.keep, ai_guard.keep) are force-kept on the span, which no sampler can override.
+          log.debug("APM tracing is disabled. APM traces will be dropped.");
+          return new ForcePrioritySampler(PrioritySampling.SAMPLER_DROP, SamplingMechanism.DEFAULT);


The drop sampler does not override manual keep decisions

Applications can send APM trace data after users turn APM tracing off.

Assertion details

Input: Set DD_APM_TRACING_ENABLED=false without ASM. Then set manual.keep, use the sampling-priority API, or enable the latency trace interceptor.

Expected: APM tracing off must drop ordinary APM traces. Only product-owned ASM and AI Guard decisions must keep traces.

Actual: The DEFAULT drop vote cannot replace a locked USER_KEEP decision. The writer then exports the APM trace.

Was this helpful? React 👍 or 👎
_{🤖 Datadog Autotest · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest} _{· Open Bits AI session}

datadog-datadog-prod-us1 · 2026-09-18T17:28:16Z

+          // and the trace is never dispatched. That is harmless for a sampled-out APM trace, but
+          // with APM tracing disabled the dropped trace also carries the LLM Observability spans,
+          // which have to reach their intake regardless of the APM sampling decision.
+          consumeFromSecondaryQueue();


The flush barrier omits pending span-sampling work

Shutdown can lose pending LLM Observability spans when single-span sampling rules are active.

Assertion details

Input: Enable LLM Observability and single-span sampling rules with APM tracing off. Finish a trace and close the writer before the span-sampling worker forwards it.

Expected: Flush must process all earlier span-sampling work before it reports completion.

Actual: A dropped trace can remain in the span-sampling queue when the primary flush marker runs. The secondary-queue sweep sees no trace. Close then stops the sampling worker and loses the LLM Observability event.

Was this helpful? React 👍 or 👎
_{🤖 Datadog Autotest · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest} _{· Open Bits AI session}

Co-Authored-By: Claude Opus 5 <[email protected]>

ncybul added type: bug fix Bug fix tag: ai generated Largely based on code generated by an AI or LLM comp: core Tracer core comp: mlobs ML Observability (LLMObs) labels Sep 11, 2026

This comment has been minimized.

Sign in to view

ncybul commented Sep 14, 2026

View reviewed changes

Comment thread dd-java-agent/agent-llmobs/src/main/java/datadog/trace/llmobs/LLMObsSystem.java Outdated

Comment thread dd-trace-core/src/main/java/datadog/trace/common/sampling/Sampler.java Outdated

ncybul and others added 2 commits September 14, 2026 11:53

Clarify why the tracer-disabled and drop-all-traces branches exist

5e30ea9

Co-Authored-By: Claude Opus 5 <[email protected]>

sabrenner approved these changes Sep 15, 2026

View reviewed changes

ncybul marked this pull request as ready for review September 15, 2026 17:59

ncybul requested review from a team as code owners September 15, 2026 17:59

ncybul requested review from mhlidd and removed request for a team September 15, 2026 17:59

Merge branch 'master' into nicole.cybul/llmobs-apm-tracing-disabled

6f19b69

datadog-datadog-prod-us1 Bot reviewed Sep 15, 2026

View reviewed changes

Comment thread dd-trace-core/src/main/java/datadog/trace/common/sampling/Sampler.java Outdated

chatgpt-codex-connector Bot reviewed Sep 15, 2026

View reviewed changes

mhlidd reviewed Sep 15, 2026

View reviewed changes

Comment thread dd-trace-core/src/main/java/datadog/trace/common/sampling/Sampler.java Outdated

ncybul and others added 2 commits September 16, 2026 16:01

Ignore the upstream sampling priority when APM tracing is disabled an…

9179365

…d no product marked the trace Co-Authored-By: Claude Opus 5 <[email protected]>

Start LLM Observability when CI Visibility installs the tracer

80e580e

Co-Authored-By: Claude Opus 5 <[email protected]>

ncybul force-pushed the nicole.cybul/llmobs-apm-tracing-disabled branch from 7860cdc to 80e580e Compare September 16, 2026 20:26

mhlidd approved these changes Sep 16, 2026

View reviewed changes

Dispatch the secondary queue before a flush completes

fe764e7

Co-Authored-By: Claude Opus 5 <[email protected]>

ncybul requested a review from a team as a code owner September 18, 2026 17:06

ncybul requested review from claponcet and removed request for a team September 18, 2026 17:06

datadog-datadog-prod-us1 Bot reviewed Sep 18, 2026

View reviewed changes

Keep the standalone ASM trickle when AppSec activates at runtime

a2a2a57

Co-Authored-By: Claude Opus 5 <[email protected]>

Conversation

ncybul commented Sep 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What Does This Do

Motivation

Additional Notes

Contributor Checklist

Uh oh!

This comment has been minimized.

dd-octo-sts Bot commented Sep 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🟢 Java Benchmark SLOs — All performance SLOs passed

Uh oh!

Uh oh!

Uh oh!

sabrenner left a comment

Choose a reason for hiding this comment

Uh oh!

datadog-datadog-prod-us1 Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Sep 15, 2026

Choose a reason for hiding this comment

Uh oh!

ncybul Sep 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

mhlidd left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

mhlidd left a comment

Choose a reason for hiding this comment

Uh oh!

datadog-datadog-prod-us1 Bot left a comment

Choose a reason for hiding this comment

Uh oh!

datadog-datadog-prod-us1 Bot Sep 18, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

datadog-datadog-prod-us1 Bot Sep 18, 2026

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

ncybul commented Sep 11, 2026 •

edited

Loading

dd-octo-sts Bot commented Sep 11, 2026 •

edited

Loading

ncybul Sep 18, 2026 •

edited

Loading