Add block-outcome telemetry for Netty blocking enforcement failures - #12316
Conversation
Report failed AppSec block-response commits to WafMetricCollector so appsec.waf.requests carries an accurate block_failure tag, scoped to Netty as the first framework. AppSecContext gains a module-boundary-safe reportBlockFailure() that AppSecRequestContext delegates to the existing setWafRequestBlockFailure() field.
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 2c99b3ee39
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
🎯 Code Coverage (details) 🔗 Commit SHA: 788dde9 | Docs | View more details | Give us feedback! |
NettyBlockResponseFunction.tryCommitBlockingResponse() now tracks whether a blocking response was already initiated for the current request. HttpPostRequestDecoderInstrumentation's advice can invoke tryBlock() multiple times across separate decoder invocations for the same request (e.g. one per multipart chunk); once an earlier call already committed the block successfully, ServerRequestContext.isPending() returns false for later calls, which previously caused a spurious block_failure to be reported even though the block had actually succeeded.
|
@codex review |
|
Codex Review: Didn't find any major issues. What shall we delve into next? Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: cdd1140d3d
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
An off-event-loop block task can fail after Netty accepts it. The new flag still marks it as complete, so later block attempts return success without enforcement or failure telemetry.
🤖 Datadog Autotest · Commit cdd1140 · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest
NettyBlockResponseFunction.tryCommitBlockingResponse() marked blockingResponseInitiated = true immediately after scheduling the off-event-loop commitBlockingResponse() call, without waiting for it to actually run. If that async commit failed once executed, the failure was silently swallowed and every later tryBlock() call for the same request would short-circuit to true, hiding the failure from block_failure telemetry. The flag is now set only inside the scheduled task, after commitBlockingResponse() actually returns true, matching the same-thread branch's existing behavior.
|
Good catch. Fixed in 4a2bd4d: the off-event-loop branch of |
|
@codex review |
|
Codex Review: Didn't find any major issues. Delightful! Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
🟢 Java Benchmark SLOs — All performance SLOs passed
PR vs. master results
Commit: Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion. |
vandonr
left a comment
There was a problem hiding this comment.
approving to unblock but I'm not really best positioned to review this code
|
/merge |
|
View all feedbacks in Devflow UI.
The expected merge time in
|
…ailureReporter Remove the catch(Exception) around brf.tryCommitBlockingResponse() in tryCommitAndReport() so it matches the Netty reference implementation (PR #12316), which only branches on the boolean return value. Several call sites (CommitActionInstrumentation in 5.5/7.0, ParsePartsInstrumentation, ParsedBodyParametersInstrumentation) rely on their enclosing advice's suppress = Throwable.class to abort the whole method when the commit throws, gating success-path side effects (closing the connection, injecting a BlockingException, marking the segment effectively blocked). Swallowing the exception inside tryCommitAndReport made those side effects run unconditionally even when nothing was actually committed to the client. Removing the catch restores the pre-existing control flow; exception-based commit failures are now a known gap not reported to block_failure telemetry, same as Netty. Addresses Codex review comment on PR #12493.
Report block_failure on appsec.waf.requests when Tomcat's blocking-commit path fails to enforce a decided block, mirroring the Netty implementation (#12316). Adds a shared TomcatBlockingHelper.tryCommitAndReport choke point used by all Tomcat/GlassFish blocking-commit call sites (5.5/6.0/7.0), and guards it against exceptions thrown by the registered BlockResponseFunction.
…ailureReporter Remove the catch(Exception) around brf.tryCommitBlockingResponse() in tryCommitAndReport() so it matches the Netty reference implementation (PR #12316), which only branches on the boolean return value. Several call sites (CommitActionInstrumentation in 5.5/7.0, ParsePartsInstrumentation, ParsedBodyParametersInstrumentation) rely on their enclosing advice's suppress = Throwable.class to abort the whole method when the commit throws, gating success-path side effects (closing the connection, injecting a BlockingException, marking the segment effectively blocked). Swallowing the exception inside tryCommitAndReport made those side effects run unconditionally even when nothing was actually committed to the client. Removing the catch restores the pre-existing control flow; exception-based commit failures are now a known gap not reported to block_failure telemetry, same as Netty. Addresses Codex review comment on PR #12493.
What Does This Do
void reportBlockFailure();toAppSecContext(internal-api), alongsideisManuallyKept().reportBlockFailure()inAppSecRequestContextby delegating to the existingsetWafRequestBlockFailure()volatile field — no new state added.NettyMultipartHelper#tryBlock(): captures the boolean returned byBlockResponseFunction#tryCommitBlockingResponse(...); onfalse, resolvesAppSecContextviactx.getData(RequestContextSlot.APPSEC)(the canonicalinstanceofpattern already used byLambdaAppSecHandler) and callsreportBlockFailure(). The method still throwsBlockingExceptionunconditionally, unchanged.NettyBlockResponseFunction#tryCommitBlockingResponse()idempotent per request: a single request can trigger multiple blocking evaluations (e.g. onetryBlock()call per multipart chunk inHttpPostRequestDecoderInstrumentation's advice). Once an earlier call has already committed the block successfully,ServerRequestContext.isPending()legitimately turnsfalsefor the request — without this guard, a later call would misread that as a commit failure and report a spuriousblock_failurefor a block that actually succeeded.tryCommitBlockingResponse()(event-loopexecute()path): theblockingResponseInitiatedidempotency flag is now only set inside the scheduled callback, aftercommitBlockingResponse(...)actually confirms success — not immediately after scheduling the task. Found by Datadog Autotest during review: the previous ordering silently hid any commit failure that happened inside the callback.MaybeBlockResponseHandler#write()'swriteAndFlushlistener flagging the async post-commit write-failure path as a known, currently-unreported gap for a future framework PR to pick up.WAFModuleSpecification.groovyassertingWafMetricCollector.raspRuleMatch(RuleType, boolean blocked)is called with the correctblockedvalue after RASP action processing.NettyMultipartHelperBlockFailureTest.java(new JUnit 5 test) verifyingreportBlockFailure()fires whentryCommitBlockingResponsereturnsfalse, and does NOT fire when it returnstrue.Scope
This is the first PR of a per-framework rollout — only Netty is covered here. Other frameworks (Tomcat, Jetty, Vert.x, Undertow, Grizzly, Play, Akka-http, Ratpack, RESTEasy, Jersey, Liberty, Spring-webmvc, commons-fileupload, okhttp) are explicitly out of scope and will get their own follow-up PRs. The async post-commit write failure in
MaybeBlockResponseHandler(channel write happens after the block decision, listener-driven) is also deliberately deferred — wiring it up requires investigating the ordering between span/request close and the channel listener to avoid introducing a race.Motivation
appsec.waf.requestscurrently has no signal for when an AppSec block action was decided but its commit to the client actually failed (e.g. the response was already committed, or the channel write failed). This closes that gap for the synchronous Netty commit path, so failed block enforcement becomes visible in telemetry instead of silently looking like a successful block.Additional Notes
Known, unaddressed gap (flagged by Codex during review, not fixed in this PR):
NettyMultipartHelper'sreportBlockFailure()call can raceGatewayBridge.onRequestEnded's singlewafRequest(...)telemetry emission. If multipart parsing resolves the commit outcome after the request has already ended (and the AppSec context closed), the failure report only mutates a field that is never read again — it is silently counted as a success. This is a distinct issue from theMaybeBlockResponseHandlerasync-write gap noted above, and applies to any future framework wiring ofblock_failurethat can resolve asynchronously relative to request-end.Contributor Checklist
type:and (comp:orinst:) labels in addition to any other useful labelsclose,fix, or any linking keywords when referencing an issueUse
solvesinstead, and assign the PR milestone to the issueJira ticket: APPSEC-62696
Note: Once your PR is ready to merge, add it to the merge queue by commenting
/merge./merge -ccancels the queue request./merge -f --reason "reason"skips all merge queue checks; please use this judiciously, as some checks do not run at the PR-level. For more information, see this doc.