Skip to content

fix(tools): reap ADK sessions of dead MCP connections in to_mcp_server - #7155

Open
harshal-96 wants to merge 5 commits into
google:mainfrom
harshal-96:fix/mcp-agent-server-session-reaping
Open

harshal-96 wants to merge 5 commits into
google:mainfrom
harshal-96:fix/mcp-agent-server-session-reaping

Conversation

@harshal-96

@harshal-96 harshal-96 commented Sep 17, 2026 •

Copy link
Copy Markdown

Fixes #7154

Description

to_mcp_server keeps one ADK session per MCP connection in a weakref.WeakKeyDictionary. When a connection is garbage collected the map entry disappears, but the ADK session it pointed to stays in the session service forever, with its full event history. A long running server therefore accumulates one dead conversation per closed connection, and a stateless streamable HTTP deployment, where the MCP SDK builds a fresh transport for every request, leaks one session per tool call.

This PR tracks the id of every session entered into the connection map. Tool calls start a background reap, at most one at a time, that deletes the sessions whose connection is no longer reachable through the weak map.

Design notes:

  • Reaping is started from tool calls rather than from a GC finalizer, because finalizers can fire without a running event loop. It runs as a background task, so a slow session service (database or Vertex backed) never adds delete latency to the live call. A task left over from another event loop does not block reaping in the current one.
  • Each reap repeats until no orphan is left, including connections that close while it runs, so a backlog drains without waiting for further calls. Each batch is deleted by a fixed pool of 8 workers sharing one iterator: concurrently, because deleting one at a time capped throughput at one session per delete round trip, and through a fixed pool rather than a task per session, because after an outage the backlog can hold many thousands of ids.
  • Once as many deletes have failed as there are workers, the workers stop deleting and the pass ends. An unavailable service therefore sees fewer than twice the pool size of attempts and one summary warning per pass, instead of one of each per waiting session, while a single session that keeps failing does not hold up the rest of the batch. Every id the pass did not delete is retried on a later call.
  • Each delete is bounded by a 10 second timeout, the bound runners.py gives toolset cleanup. Only one reap runs at a time, so a delete that never returned would otherwise stop reaping for good. The wait uses asyncio.wait rather than asyncio.wait_for: before Python 3.12, wait_for can swallow a cancellation that arrives as the delete finishes.
  • A reap cancelled mid-pass (for example at event loop shutdown) puts back both the ids still in flight and the ones no worker had started, so none are dropped from tracking; deleting a session that is already gone is a no-op in the built-in session services, so the retry is safe. Cancellation also stops the reap immediately instead of letting it drain its backlog.
  • The reap task and its done callback are created in an empty contextvars.Context. Both capture the current context by default, and on MCP SDK 1.x the request lives in a ContextVar that references the connection, so a copied context would keep the very connection whose session the reap should delete alive for as long as the task runs.
  • No timer, TTL, or policy surface is added. Connection lifetime is already governed by the MCP transport layer (client disconnects, the SDK's idle session timeout, stateless per request teardown); the fix makes ADK sessions follow that lifetime instead of outliving it. A session in use by an in-flight call is always strongly referenced through the call context, so an active conversation can never be reaped.
  • Delete failures and unexpected reap errors are logged, and neither ever fails a tool call.
  • Behavior change: with a persistent session service, orphaned conversations were previously retained forever and are now deleted by default. to_mcp_server(..., delete_orphaned_sessions=False) disables reaping entirely and leaves session lifecycle to the caller, for deployments that read those records after the fact. Only sessions created by the running server process are tracked, so with a persistent session service, sessions orphaned before a restart are not deleted.
  • The to_mcp_server docstring now documents stateless streamable HTTP behavior: each call is a fresh single turn conversation whose session is reclaimed.

Testing plan

18 new unit tests in tests/unittests/tools/mcp_tool/test_agent_to_mcp.py:

  • test_reap_deletes_only_sessions_no_longer_reachable: the reap deletes exactly the unreachable ids.
  • test_session_of_a_collected_connection_is_reaped: a conversation does not outlive its connection.
  • test_per_request_connections_do_not_accumulate_sessions: the stateless per request pattern stays flat instead of leaking one session per call.
  • test_reap_failure_does_not_raise_and_is_retried: a session service outage does not fail the live tool call, and the orphan is deleted once the service recovers.
  • test_reap_partial_failure_requeues_only_the_failed_ids: in a batch, a failed delete is retried later without undoing the deletes that were in flight and succeeded.
  • test_reap_stops_after_a_failure_instead_of_hammering_the_service: with the session service down and 80 sessions waiting, a reap makes fewer than twice the pool size of delete attempts, logs one summary warning, and keeps every id for retry.
  • test_one_failing_session_does_not_hold_up_the_rest: a session the service keeps rejecting does not stop the other 100 from being deleted in the same pass.
  • test_cancelled_reap_stops_promptly_and_keeps_unfinished_ids: with 400 sessions waiting and 50 ms deletes, a reap cancelled after 0.1 s stops without draining the backlog, and every id is either deleted or still tracked.
  • test_hung_delete_times_out_instead_of_stopping_reaping: a delete that never returns times out and is retried later, and the rest of the batch is still deleted.
  • test_reap_task_count_stays_bounded_for_a_large_backlog: a backlog of 1000 is drained without a task per waiting session.
  • test_slow_deletes_run_concurrently_up_to_the_cap: with a slow session service, a backlog is deleted concurrently and never with more than the cap in flight.
  • test_reap_drains_orphans_that_appear_while_it_runs: sessions orphaned while a reap is deleting are picked up by the same reap.
  • test_call_tool_reaps_conversation_of_closed_connection: end to end through a real in-memory MCP client and server.
  • test_call_tool_retains_sessions_when_deletion_is_opted_out: delete_orphaned_sessions=False keeps every session.
  • test_call_tool_does_not_wait_on_slow_session_deletes: with the session service's delete blocked, the tool call still completes; it times out with an inline reap.
  • test_background_reap_failure_is_logged_not_raised: an unexpected reap error is logged and the call succeeds.
  • test_background_reap_does_not_inherit_the_callers_context: a ContextVar set around the call is not visible inside the reap.
  • test_reap_is_not_blocked_by_a_task_from_another_event_loop: a reap stuck in a stopped event loop does not stop reaping in the next one.

Each safeguard was checked by removing it and confirming its test fails, on mcp 1.26.0 and 2.2.0: putting back in-flight and unstarted ids on cancellation, the delete timeout, the failure threshold (both never stopping and stopping at the first failure), the bounded worker pool, concurrent deletes, the drain loop, per-id failure retry, background start, the done callback, the empty context, and the event loop check. Switching the delete back to asyncio.wait_for fails the cancellation test in 4 of 5 runs on Python 3.10 and 3.11 (the race does not exist on 3.12).

Results:

  • pytest tests/unittests/tools/mcp_tool/test_agent_to_mcp.py: 31 passed on Python 3.10, 3.11, and 3.12, each with mcp 1.24.0, 1.26.0, and 2.2.0 (the pin admits 1.x and 2.x); 20 consecutive runs stable on each of 3.10, 3.11, and 3.12.
  • pytest tests/unittests/tools/mcp_tool: 389 passed (Python 3.12, mcp 2.2.0), and 443 passed on Python 3.10, 3.12 and 3.14 with the branch merged onto current main (f44d512). The full tests/unittests suite merged onto f44d512 gives 16573 passed versus 16555 on main: the 18 new tests, with the same 3 failures as main (unrelated, fixed in fix(live): stop an early-closed merged live stream from hanging #7358 and fix: keep google-auth transport imports out of Agent import #7359).
  • Real transport check: server run with run_streamable_http_async(stateless_http=True) on localhost, one fresh streamable_http_client connection per call. 9 calls leave 9 sessions in the service on main and 1 with this PR (the most recent call's session, reclaimed on the next call).
  • Load check (a new connection and call every 20 ms, 300 calls) against a stub session service with 50 ms deletes, mcp 1.26.0: the service holds 2 to 3 sessions after 100 and 300 calls and 0 after a second idle (three runs), against 34 to 49, 99 to 140, and 81 to 124 with one-at-a-time deletes (four runs).
  • The same load against ADK's real Runner and session services, mcp 1.26.0: InMemorySessionService and DatabaseSessionService on SQLite (also with 50 ms added per delete) hold 1 to 2 sessions over three runs, and no delete failed, so concurrent deletes do not trip SQLite locking. With the real Runner each call takes longer, so at this load the one-at-a-time version mostly kept up as well (2 to 6 sessions); the difference shows once calls outpace deletes, as in the stub run. With mcp 2.2.0 the sessions still held belong to connections not yet garbage collected, which are not orphans; after a collection, one call reaps them all.
  • Session service outage with 200 sessions waiting: one pass makes 15 delete attempts and logs 1 warning, against 200 of each before, and keeps all 200 ids for retry.
  • One session the service keeps rejecting plus 100 healthy ones: the 100 are deleted in one pass (five runs).
  • Cancellation with 400 sessions waiting and 50 ms deletes: the reap stops immediately on Python 3.10, 3.11, and 3.12 and no id is lost; with asyncio.wait_for it kept deleting for about 2.7 s on 3.10 and 3.11.
  • Large backlog: draining 100,000 waiting ids peaks at about 5 MB of extra memory with the worker pool, against about 15 MB per 10,000 ids (147 MB per 100,000) with a task per id.
  • Repro script from the issue: sessions left in the service: 10 on main, 1 with this PR.

Formatting and checks: pyink 25.12, isort 8.0.1, ruff 0.15.17, codespell 2.4.2, and scripts/compliance_checks.py are clean; mypy reports no errors in the changed module, the same as main; pylint with the repository pylintrc rates it 10.00/10.

to_mcp_server keeps one ADK session per MCP connection in a
WeakKeyDictionary. When a connection is garbage-collected the map entry
disappears, but the ADK session it pointed to stays in the session
service forever, with its full event history. A long-running server
therefore accumulates one dead conversation per closed connection, and a
stateless streamable HTTP deployment, where the SDK builds a fresh
transport for every request, leaks one session per tool call.

Track the id of every session entered into the connection map and, at
the start of each tool call, delete the sessions whose connection is no
longer reachable. Reaping runs lazily from the tool call rather than a
GC callback because finalizers may fire without a running event loop.

Also document how a stateless streamable HTTP deployment behaves: each
call is a fresh single-turn conversation whose session is reclaimed.

Tested with mcp 1.26.0 and 2.2.0: 17 passed each.
@google-cla

google-cla Bot commented Sep 17, 2026

Copy link
Copy Markdown

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

@harshal-96

Copy link
Copy Markdown
Author

@googlebot I signed it!

Callers who wire to_mcp_server to a persistent session service may want
finished conversations to remain readable after their connection dies,
e.g. for audit. delete_orphaned_sessions=False disables the reaping and
leaves session lifecycle to the caller. The default stays True so a
long-running server's memory is bounded out of the box.
The reap ran inline before the live tool call, so with a slow session
service (e.g. database or Vertex backed) the first call after a burst of
disconnects waited on every delete. Start it as a background task
instead:

- At most one reap runs at a time, and the task is referenced so it is
  not garbage-collected mid-run. A task left over from another event
  loop does not block reaping in the current one.
- The task and its done callback are created in an empty contextvars
  Context. Both capture the current context by default, and a copy of
  the tool call's would keep the request, and on MCP SDK 1.x the
  connection, alive for as long as the task runs, so a stuck reap would
  pin the very connection whose session it should delete.
- A done callback logs unexpected reap errors instead of leaving them
  for asyncio to report as never retrieved.
The background reap deleted orphaned sessions one at a time and a new
reap only started once the previous one finished, so throughput was
capped at one session per delete round trip. With a 50 ms delete and a
call every 20 ms the service held a growing backlog that only drained
when further calls arrived.

Each reap now repeats until no orphan is left, picking up connections
that close while it runs, and deletes each batch with a fixed pool of
_MAX_CONCURRENT_DELETES workers sharing one iterator, so a backlog of
any size costs a bounded number of tasks and a database-backed service
sees a bounded number of concurrent deletes.

Once as many deletes have failed as there are workers, the workers stop
deleting and the pass ends, so an unavailable service sees fewer than
twice the pool size of attempts and one summary warning per pass
instead of one of each per waiting session, while a single session that
keeps failing does not hold up the rest of the batch. Every id the pass
did not delete is put back for a later call to retry. Each delete is
bounded by _DELETE_TIMEOUT_SECONDS: only one reap runs at a time, so a
delete that never returned would otherwise stop reaping for good.
A reap cancelled mid-pass, for example at event loop shutdown, dropped
the ids it had claimed but not yet deleted: the in-flight ones and the
ones no worker had started. They were never deleted afterwards. Put both
back so a later reap retries them; deleting a session that is already
gone is a no-op in the built-in session services.

Each delete now waits with asyncio.wait instead of asyncio.wait_for.
Before Python 3.12, wait_for can swallow a cancellation that arrives as
the delete finishes, which let a cancelled reap keep draining its whole
backlog. With 400 orphans and 50 ms deletes, cancelling took about 2.7 s
on Python 3.10 and 3.11 and now returns immediately.

Also document that only sessions created by this server process are
tracked, so with a persistent session service, sessions orphaned before
a restart are not deleted.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

to_mcp_server never deletes the ADK session of a dead MCP connection; a stateless streamable HTTP deployment leaks one session per tool call

2 participants