fix(cli): expose trigger delivery id so tools can detect redeliveries - #7325
Open
devansh173 wants to merge 3 commits into
Open
devansh173 wants to merge 3 commits into
devansh173 wants to merge 3 commits into
Conversation
Pub/Sub and Eventarc redeliver a message when the trigger endpoint returns a non-2xx status, and each delivery runs the agent in a new session. The Pub/Sub messageId was only logged, so a tool that had already caused a side effect (a payment, an email) had nothing to deduplicate on and repeated it. Store the delivery identity in the new session's state under `trigger_delivery`: the source, the Pub/Sub messageId or CloudEvents id, and the source-specific metadata. For Pub/Sub-wrapped Eventarc events without a ce-id, fall back to the messageId. Tools can read it from tool_context.state and use it as an idempotency key. Fixes google#7322
Session state is new on every delivery, so a tool cannot dedupe there. Point tools at the provider or an external store, and key on subscription + messageId (or event_source + id for Eventarc), since the id alone is only unique per topic or source.
Check, act and record against a store still pays twice after an ambiguous failure or an overlapping redelivery. Prefer the provider's idempotency key; otherwise reserve the key atomically before the effect and plan for reconciling an ambiguous attempt. Raise on a key-in-use response so the message is nacked and redelivered.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Link to Issue or Description of Change
1. Link to an existing issue (if applicable):
Problem:
When a Pub/Sub trigger run fails after a tool has already caused a side effect (a payment, an email, a ticket), the endpoint returns 500 and Pub/Sub redelivers the message. Every delivery runs in a new session (
uuid4()), and the Pub/SubmessageIdis only logged, never passed to the agent. So a tool has nothing stable to deduplicate on, and the side effect happens again. The same applies to Eventarc: Pub/Sub-wrapped events don't carryce-idinto the agent's input.Solution:
This implements option 1 from the issue: pass the delivery identity into the run.
When a trigger creates its session, it now seeds the session state with a
trigger_deliveryentry (exposed asTRIGGER_DELIVERY_STATE_KEY):source"pubsub""eventarc"idmessage.messageIdid(body orce-idheader), falling back to the wrapped Pub/SubmessageIdsubscription,publish_timeevent_source,typeSession state is new on every delivery, so the deduplication itself has to happen at the provider or in an external store, not in state. A tool derives an idempotency key from this value and passes it on. A Pub/Sub
messageIdis only unique per topic, so the key includes the subscription (for Eventarc,event_source+id):Why this approach:
messageId(option 2) would not help on its own.Option 2 (a deterministic
session_id) changes redelivery behaviour and would need a guard against concurrent redeliveries, so I left it out of this PR. I'm happy to follow up on it, or on the docs (option 3) in adk-docs, if maintainers want either.Testing Plan
Unit Tests:
Added
TestTriggerDeliveryIdentitytotests/unittests/cli/test_trigger_routes.py(7 tests):messageId,subscriptionandpublishTimeare stored in session state.messageIdstill recordssource, withidset toNone.messageIdkey checked outside the session (the scenario from the issue: first delivery returns 500, the redelivery returns 200, one payment).id,sourceandtypeare stored.ce-*headers.ce-idfalls back to themessageId.All 7 new tests fail on
mainand pass with this change.Same result on Python 3.10, 3.11 and 3.14. The rest of
tests/unittests/clialso passes; the only failures on my Windows machine are 32 platform-specific tests (symlinks, path separators, gcloud deploy), and they fail identically onmain.pyink,isort,ruff,addlicense,codespelland the compliance checks pass. mypy reports no new errors intrigger_routes.py.Manual End-to-End (E2E) Tests:
I used the reproduction script from #7322 unchanged: a stock
LlmAgenton theGeminiclass pointed at a local fake Gemini endpoint that returns 503 once after thepay_invoicecall, with a loop that redelivers the same push envelope like Pub/Sub. No network access or API key is needed.I ran it twice: once with the original tool, and once with the tool changed to pass a subscription +
messageIdidempotency key to the (fake) provider, whose key store lives outside the ADK session. The second run then sends a new message, to check that a newmessageIdis still charged:Note:
DONEis an in-memory check-and-add ledger that stands in for the provider's idempotency store, to keep the harness self-contained. It is not the recommended pattern: check → act → record pays twice after an ambiguous failure or an overlapping redelivery. TheTRIGGER_DELIVERY_STATE_KEYdocstring recommends the provider's idempotency key, or an atomic reservation before the effect when using your own store.Before (
main@ 044a1ec): the tool sees no delivery identity and pays twice for one message.After (this PR): both deliveries carry the same key, so the provider charges once; the next message has a new
messageIdand is charged normally.The unmodified script from the issue still reports 2 payments, as expected: this change gives tools what they need to deduplicate, and doesn't change what happens to a tool that ignores it.
Checklist
Additional context
The key is documented on
TRIGGER_DELIVERY_STATE_KEY(including where to dedupe and how to build the key) and in both endpoints' OpenAPI descriptions. The ack-deadline and dead-letter guidance from the issue will follow as a separate PR in adk-docs.