Skip to content

Add block_failure telemetry for Tomcat blocking enforcement failures - #12493

Open
jandro996 wants to merge 9 commits into
masterfrom
block-telemetry-2
Open

jandro996 wants to merge 9 commits into
masterfrom
block-telemetry-2

Conversation

@jandro996

@jandro996 jandro996 commented Sep 15, 2026 •

Copy link
Copy Markdown
Member

What Does This Do

  • Reports block_failure on the appsec.waf.requests metric when Tomcat's blocking-commit path fails to enforce a decided block, mirroring the Netty implementation (Add block-outcome telemetry for Netty blocking enforcement failures #12316).
  • Adds BlockFailureReporter.tryCommitAndReport(RequestContext, Flow.Action.RequestBlockingAction) in tomcat-common as a shared choke point: commits the blocking response via the registered BlockResponseFunction and, if the commit fails (returns false), reports failure through AppSecContext.reportBlockFailure() (resolved from RequestContext.getData(RequestContextSlot.APPSEC)). Exceptions thrown by the commit attempt are deliberately propagated (not converted into a false return), so the calling advice's suppress = Throwable.class aborts the whole advice and skips its blocking success-path side effects; such exception-based failures are a known telemetry gap shared with the Netty implementation.
  • Wires all Tomcat/GlassFish blocking-commit call sites (5.5/6.0/7.0) through this shared helper: CommitActionInstrumentation (5.5, 7.0), ParsedBodyParametersInstrumentation (5.5, 6.0), ParsePartsInstrumentation (7.0), GlassFishBlockingHelper/GlassFishMultipartInstrumentation (7.0), and TomcatServerInstrumentation (5.5 base).
  • Wires all three of TomcatBlockingHelper.commitBlockingResponse()'s reachable failure paths to report block_failure: GET_OUTPUT_STREAM == null, the response already committed by something other than us, and the write to the client throwing. The "already committed" / "write threw" branches keep their existing true return value.
  • Guards GlassFishBlockingHelper.tryBlock()'s Servlet API fallback branch so reportBlockFailure() only fires when a response was actually available to commit through (i.e. a commit was genuinely attempted), not on the brf == null / fallbackResp == null case where nothing was attempted at all.
  • Adds a volatile boolean blockingResponseInitiated guard to TomcatDecorator.TomcatBlockResponseFunction, mirroring the existing pattern in Netty's NettyBlockResponseFunction. Tomcat can evaluate blocking more than once per request (e.g. multipart parsing), so once a block has committed successfully, later calls for the same request short-circuit instead of re-evaluating the commit - this is what makes it safe to report block_failure on the "already committed" branch without false positives for a legitimate repeat block.

Split of TomcatBlockingHelper

  • Extracted tryCommitAndReport()/reportBlockFailure(RequestContext) into a new class, BlockFailureReporter (tomcat-common), that has no dependency on org.apache.catalina.* types.
  • TomcatBlockingHelper keeps only the catalina-typed methods (commitBlockingResponse(), start(), tryWriteWithOutputStream/Writer, the private reportBlockFailure(Request)).
  • Reason: helperClassNames() injects a helper's entire class bytecode, and Gradle's muzzle task validates every reference in that bytecode against each declared library version group. Before the split, instrumentations whose target library range does not guarantee org.apache.catalina.connector.Request/Response presence (e.g. some tomcat-appsec-5.5/6.0/7.0 groups) failed muzzle just from listing TomcatBlockingHelper, even though their advice only ever called the catalina-independent tryCommitAndReport/reportBlockFailure(RequestContext) methods.
  • RequestInstrumentation/ResponseInstrumentation (tomcat-5.5) also needed BlockFailureReporter added to helperClassNames(), since they list TomcatBlockingHelper transitively (via TomcatDecorator) and TomcatBlockingHelper's own reportBlockFailure(Request) still delegates to BlockFailureReporter.reportBlockFailure(reqCtx).

Motivation

APPSEC blocking-outcome telemetry currently only exists for Netty (#12316). This extends coverage to Tomcat so that a WAF-decided block that silently fails to actually block the request (e.g. response already committed, BlockResponseFunction throwing) is observable via the block_failure tag on appsec.waf.requests, instead of looking like a successful block.

Additional Notes

  • tomcat-common's new test dependencies (tomcat:catalina/tomcat-coyote/tomcat-util, javax.servlet:servlet-api:2.4, JUnit 5, Mockito) required regenerating gradle.lockfile for that module (tomcat-appsec-6.0's lockfile needs no change: its new implementation project(...) dependency adds no external dependency to that module's classpath).
  • Fixed a pre-existing bug in the spring-webmvc test fixtures (TestSpringBlockResponseFunction): it was unconditionally returning true/false instead of delegating to the previously registered BlockResponseFunction when RequestContextHolder is not yet populated by Spring, which clobbered Tomcat's real BlockResponseFunction in latestDepTest runs and surfaced as 10 unrelated test failures once tryCommitAndReport started being exercised on that path.
  • The "already committed" / "write threw" branches were initially left out of scope; extended after review feedback (discussion) once the blockingResponseInitiated guard made it safe to report on them without false positives.

Contributor Checklist

Jira ticket: APPSEC-62696

Note: Once your PR is ready to merge, add it to the merge queue by commenting /merge. /merge -c cancels the queue request. /merge -f --reason "reason" skips all merge queue checks; please use this judiciously, as some checks do not run at the PR-level. For more information, see this doc.

@jandro996 jandro996 added type: feature Enhancements and improvements comp: asm waf Application Security Management (WAF) inst: others All other instrumentations labels Sep 15, 2026
@jandro996

Copy link
Copy Markdown
Member Author

@codex review

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Sep 15, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-09-15T20:20:59.237999Z 6fc0248 Draft marked ready
🔒 Security Review ✅ Completed 2026-09-15T20:23:26.239841Z 6fc0248 Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Hooray!

Reviewed commit: b6b1546544

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@chatgpt-codex-connector

Copy link
Copy Markdown

🛡️ Codex Security Review · Automatically triggered

Security review completed. No security issues were found in this pull request.

Reviewed commit: b6b1546544

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

@datadog-prod-us1-6

This comment has been minimized.

@dd-octo-sts

dd-octo-sts Bot commented Sep 15, 2026 •

Copy link
Copy Markdown
Contributor

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite Status
Startup 🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results
Scenario Candidate master Δ (95% CI of mean)
startup:insecure-bank:iast:Agent 13.97 s 13.92 s [-0.3%; +1.1%] (no difference)
startup:insecure-bank:tracing:Agent 12.89 s 13.01 s [-1.7%; -0.1%] (maybe better)
startup:petclinic:appsec:Agent 16.94 s 16.85 s [-0.4%; +1.4%] (no difference)
startup:petclinic:iast:Agent 16.90 s 16.94 s [-1.0%; +0.6%] (no difference)
startup:petclinic:profiling:Agent 16.47 s 16.72 s [-2.7%; -0.4%] (maybe better)
startup:petclinic:sca:Agent 16.24 s 16.77 s [-7.5%; +1.1%] (no difference)
startup:petclinic:tracing:Agent 16.16 s 16.22 s [-1.1%; +0.4%] (no difference)

Commit: de443584 · CI Pipeline · Benchmarking Platform UI


Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

@jandro996

Copy link
Copy Markdown
Member Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. 👍

Reviewed commit: f3cbd0a658

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@chatgpt-codex-connector

Copy link
Copy Markdown

🛡️ Codex Security Review · Automatically triggered

Security review completed. No security issues were found in this pull request.

Reviewed commit: f3cbd0a658

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

@jandro996

Copy link
Copy Markdown
Member Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

🛡️ Codex Security Review · Automatically triggered

Security review completed. No security issues were found in this pull request.

Reviewed commit: f34f16feea

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: f34f16feea

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

jandro996 added a commit that referenced this pull request Sep 15, 2026
…ailureReporter

Remove the catch(Exception) around brf.tryCommitBlockingResponse() in
tryCommitAndReport() so it matches the Netty reference implementation
(PR #12316), which only branches on the boolean return value.

Several call sites (CommitActionInstrumentation in 5.5/7.0,
ParsePartsInstrumentation, ParsedBodyParametersInstrumentation) rely on
their enclosing advice's suppress = Throwable.class to abort the whole
method when the commit throws, gating success-path side effects (closing
the connection, injecting a BlockingException, marking the segment
effectively blocked). Swallowing the exception inside tryCommitAndReport
made those side effects run unconditionally even when nothing was
actually committed to the client. Removing the catch restores the
pre-existing control flow; exception-based commit failures are now a
known gap not reported to block_failure telemetry, same as Netty.

Addresses Codex review comment on PR #12493.
@jandro996

Copy link
Copy Markdown
Member Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Nice work!

Reviewed commit: 1e718e1484

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@chatgpt-codex-connector

Copy link
Copy Markdown

🛡️ Codex Security Review · Automatically triggered

Security review completed. No security issues were found in this pull request.

Reviewed commit: 1e718e1484

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

@jandro996
jandro996 marked this pull request as ready for review September 15, 2026 20:16
@jandro996
jandro996 requested review from a team as code owners September 15, 2026 20:16
@jandro996
jandro996 requested review from AlexeyKuznetsov-DD and jordan-wong and removed request for a team September 15, 2026 20:16

@datadog-prod-us1-6 datadog-prod-us1-6 Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Datadog Autotest: PASS

More details

All changed Tomcat and GlassFish commit paths use the shared failure reporter. Each affected instrumentation includes the required helper.

Was this helpful? React 👍 or 👎

Open Bits AI session

🤖 Datadog Autotest · Commit 6fc0248 · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest

Comment thread dd-java-agent/instrumentation/tomcat/tomcat-common/build.gradle
jandro996 added a commit that referenced this pull request Sep 15, 2026
Regenerate the lock file to reflect the testImplementation
dependencies added in TODO-9 (catalina/coyote/util, servlet-api,
JUnit 5, Mockito) for the sun.misc.Unsafe-based test. No change
needed for tomcat-appsec-6.0's lockfile: its new project(':...:tomcat-common')
dependency does not add any external dependency to its classpath.

Addresses review comments from AlexeyKuznetsov-DD on PR #12493.
Report block_failure on appsec.waf.requests when Tomcat's blocking-commit
path fails to enforce a decided block, mirroring the Netty implementation
(#12316). Adds a shared TomcatBlockingHelper.tryCommitAndReport choke point
used by all Tomcat/GlassFish blocking-commit call sites (5.5/6.0/7.0), and
guards it against exceptions thrown by the registered BlockResponseFunction.
…lemetry

Extract tryCommitAndReport()/reportBlockFailure(RequestContext) into a new
catalina-independent BlockFailureReporter class in tomcat-common, since
helperClassNames() injects the whole class and muzzle validates all its
references against every declared library version - including the
catalina-independent code path that doesn't need it.
…Function

TestSpringBlockResponseFunction now delegates to the previously registered
BlockResponseFunction (Tomcat's own, in production tests) instead of
returning true/false unconditionally when RequestContextHolder is not yet
populated by Spring, fixing 10 latestDepTest failures exposed by the
tryCommitAndReport refactor.
…fallback response

Guard the Servlet API fallback branch of tryBlock() so reportBlockFailure()
only fires when a response was actually available to commit through (i.e. a
commit was genuinely attempted). Previously it reported unconditionally
whenever the fallback commit failed, including the brf == null / fallbackResp
== null case where nothing was attempted at all.
…ailureReporter

Remove the catch(Exception) around brf.tryCommitBlockingResponse() in
tryCommitAndReport() so it matches the Netty reference implementation
(PR #12316), which only branches on the boolean return value.

Several call sites (CommitActionInstrumentation in 5.5/7.0,
ParsePartsInstrumentation, ParsedBodyParametersInstrumentation) rely on
their enclosing advice's suppress = Throwable.class to abort the whole
method when the commit throws, gating success-path side effects (closing
the connection, injecting a BlockingException, marking the segment
effectively blocked). Swallowing the exception inside tryCommitAndReport
made those side effects run unconditionally even when nothing was
actually committed to the client. Removing the catch restores the
pre-existing control flow; exception-based commit failures are now a
known gap not reported to block_failure telemetry, same as Netty.

Addresses Codex review comment on PR #12493.
Regenerate the lock file to reflect the testImplementation
dependencies added in TODO-9 (catalina/coyote/util, servlet-api,
JUnit 5, Mockito) for the sun.misc.Unsafe-based test. No change
needed for tomcat-appsec-6.0's lockfile: its new project(':...:tomcat-common')
dependency does not add any external dependency to its classpath.

Addresses review comments from AlexeyKuznetsov-DD on PR #12493.
… in Tomcat

Follow-up to PR review comment r4067498401: TomcatBlockingHelper.commitBlockingResponse()
now reports block_failure when the response was already committed by something other than
us, and when writing the blocking response throws. Safe against false positives from
Tomcat's multiple per-request blocking evaluations via a new blockingResponseInitiated
guard on TomcatBlockResponseFunction, mirroring Netty's existing pattern.
Resolves lockfile conflicts introduced by rebasing block-telemetry-2 onto
current master (dd-instrument-java, spotbugs, byte-buddy version bumps).
Master (#12519) added a default BlockResponseFunction.tryCommitBlockingResponse(RequestContext, RequestBlockingAction)
overload alongside the existing TraceSegment-based one, making bare any()
matchers ambiguous. Disambiguate with any(TraceSegment.class), matching
the overload GlassFishBlockingHelper actually invokes via BlockFailureReporter.

@datadog-prod-us1-6 datadog-prod-us1-6 Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bits Code Review: PASS

More details

The shared helper reports failed commit results. The request guard prevents repeat block attempts from causing false failure telemetry.

Was this helpful? React 👍 or 👎

Open Bits AI session

🤖 Bits Code Review · Commit de44358 · @DataDog review to ask questions

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp: asm waf Application Security Management (WAF) inst: others All other instrumentations type: feature Enhancements and improvements

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants