google · RainieLLM · Sep 30, 2026
diff --git a/docs/guides/evaluation/eval_service/index.md b/docs/guides/evaluation/eval_service/index.md
@@ -159,6 +159,23 @@ conversation, which is why the manager is a constructor argument rather than
 something you pass per request. An `InferenceResult` naming a case the manager
 does not have raises `NotFoundError`.
 
+### Session state for metrics
+
+`perform_inference` stores the actual session state before the first turn and
+after the last turn in `InferenceResult.initial_session_state` and
+`InferenceResult.final_session_state`. For an existing session, the initial
+snapshot comes from that session. `SessionInput.state` is the seed for a new
+session and may differ from the actual initial state.
+
+`evaluate` passes these snapshots to each metric through `EvaluationContext`,
+along with `expected_final_session_state` from `EvalCase.final_session_state`.
+Each metric receives its own copy, so changes in one metric do not affect another.
+The expected state preserves the eval case value, whose default is `{}`. An
+omitted expected state therefore differs from an explicit `None`.
+To read these fields, override
+[`Evaluator.evaluate_with_context`](../evaluator/index.md#write-an-evaluator-subclass).
+Existing evaluators and custom metric functions keep their original arguments.
+
 ### Statuses
 
 Each metric produces an `EvalMetricResult` with a score and an `EvalStatus`.
@@ -289,8 +306,17 @@ async for case_result in eval_service.evaluate(
   ...
 ```
 
+Actual state snapshots survive serialization, so a later session update does
+not change the state a metric sees. Old results without snapshots load with
+`None` for those fields. The service does not fill missing snapshots from the
+live session. An empty dictionary is a valid snapshot and differs from `None`.
+
+`EvalCaseResult.session_details` still comes from the session service at evaluation
+time. It may reflect later updates, or be `None` if the session was deleted. Metrics
+use the inference snapshots instead.
+
 The eval sets manager must still hold the same cases under the same ids, since
-`evaluate` re-reads the expected conversation from it.
+`evaluate` re-reads the expected conversation and expected final state from it.
 
 ### Evaluate a subset
 
@@ -325,10 +351,9 @@ collecting a list, because callers rely on that to report progress.
 *   **It needs the evaluation extra.** `base_eval_service` imports on a base
     install, but `local_eval_service` pulls in `vertexai` through
     `google-cloud-aiplatform[evaluation]`. Install `google-adk[eval]`.
-*   **Nothing is re-exported at package level.** Import from
+*   **Service types use module imports.** Import from
     `google.adk.evaluation.base_eval_service` and
-    `google.adk.evaluation.local_eval_service`; `google.adk.evaluation` exports
-    only `AgentEvaluator`.
+    `google.adk.evaluation.local_eval_service` directly.
 *   **Failures are absorbed at both layers.** A failed inference becomes a
     `FAILURE` result, and a metric that raises becomes `NOT_EVALUATED`. Neither
     reaches the caller as an exception, so a job that does not check statuses

diff --git a/docs/guides/evaluation/evaluator/index.md b/docs/guides/evaluation/evaluator/index.md
@@ -26,11 +26,11 @@ and the module it names.
 
 An **`Evaluator` subclass** is the heavier one. It exists for a metric that needs
 per-run construction, such as a client to build, an expensive model to load
-once, or a criterion type of its own with extra config keys. A config file
-cannot name a class, only a function, so a subclass has to be registered from
-Python before the run starts.
+once, a criterion type of its own with extra config keys, or session state.
+A config file cannot name a class, only a function, so a subclass has to be
+registered from Python before the run starts.
 
-Take the function unless one of those three needs applies, because a function
+Take the function unless one of those needs applies, because a function
 costs you nothing beyond the function, while a subclass adds a registration step
 that has to run in the same process as the evaluation.
 
@@ -198,16 +198,41 @@ both the scores and the statuses, as the example above does, satisfies both.
 
 ## Advanced applications
 
-The class-based route is what makes per-run construction and extra configuration
-keys available, and those two capabilities are worth taking separately.
+The class-based route supports per-run construction, session state, and extra
+configuration keys.
 
 ### Write an `Evaluator` subclass
 
-Subclass `Evaluator` when the metric needs setup that should happen once per
-run rather than once per call. The constructor is invoked with a single keyword
-argument, `eval_metric=`, and `evaluate_invocations` may be sync or async.
+Subclass `Evaluator` when the metric needs setup once per run or access to
+session state. The constructor receives a single keyword argument, `eval_metric=`.
+Override `evaluate_invocations` for invocation-only metrics, or
+`evaluate_with_context` for metrics that need state. Either method may be sync
+or async. The default `evaluate_with_context` calls `evaluate_invocations` with
+its original arguments, so existing evaluators need no changes.
+
+`EvaluationContext` provides state for the whole eval case:
+
+| Field | Description |
+| :--- | :--- |
+| `initial_session_state` | Actual state before the first turn. |
+| `final_session_state` | Actual state after the last turn. |
+| `expected_final_session_state` | Expected state from `EvalCase.final_session_state`. |
+
+Each field is a dictionary, or `None` when unavailable. An empty dictionary is
+valid state. Actual states are snapshots from inference, and each metric receives
+its own copy. For details on saved results, see the
+[eval service guide](../eval_service/index.md#session-state-for-metrics).
+
+The expected state preserves `EvalCase.final_session_state` as supplied. Its
+default is `{}`, so an omitted value means an expected empty state. Set it to
+`None` explicitly when no expected state is available.
+
+The following metric checks whether the final state exactly matches the expected
+state. It returns `NOT_EVALUATED` if either state is absent. The same case-level
+score appears on each invocation to satisfy the result contract.
 
 ```python
+from google.adk.evaluation import EvaluationContext
 from google.adk.evaluation.eval_metrics import BaseCriterion
 from google.adk.evaluation.eval_metrics import EvalMetric
 from google.adk.evaluation.evaluator import EvaluationResult
@@ -216,45 +241,45 @@ from google.adk.evaluation.evaluator import EvalStatus
 from google.adk.evaluation.evaluator import PerInvocationResult
 
 
-class ResponseLengthEvaluator(Evaluator):
-  """Scores 1.0 when the final response stays under a character budget."""
+class FinalStateMatchEvaluator(Evaluator):
+  """Scores 1.0 when the final state matches the expected state."""
 
   criterion_type = BaseCriterion
 
   def __init__(self, eval_metric: EvalMetric):
-    self._threshold = eval_metric.criterion.threshold
-
-  def evaluate_invocations(
+    if eval_metric.criterion is not None:
+      self._threshold = eval_metric.criterion.threshold
+    elif eval_metric.threshold is not None:
+      self._threshold = eval_metric.threshold
+    else:
+      raise ValueError("final_state_match requires a threshold.")
+
+  def evaluate_with_context(
       self,
       actual_invocations,
       expected_invocations=None,
       conversation_scenario=None,
+      *,
+      context: EvaluationContext,
   ) -> EvaluationResult:
-    results = []
-    for invocation in actual_invocations:
-      response = invocation.final_response
-      parts = (response.parts or []) if response else []
-      length = len("".join(part.text or "" for part in parts))
-      score = 1.0 if length <= 200 else 0.0
-      results.append(
-          PerInvocationResult(
-              actual_invocation=invocation,
-              score=score,
-              eval_status=(
-                  EvalStatus.PASSED if score else EvalStatus.FAILED
-              ),
-          )
-      )
-
-    overall = sum(r.score for r in results) / len(results)
+    actual = context.final_session_state
+    expected = context.expected_final_session_state
+    if actual is None or expected is None or not actual_invocations:
+      return EvaluationResult()
+
+    score = 1.0 if actual == expected else 0.0
+    status = (
+        EvalStatus.PASSED if score >= self._threshold else EvalStatus.FAILED
+    )
     return EvaluationResult(
-        overall_score=overall,
-        overall_eval_status=(
-            EvalStatus.PASSED
-            if overall >= self._threshold
-            else EvalStatus.FAILED
-        ),
-        per_invocation_results=results,
+        overall_score=score,
+        overall_eval_status=status,
+        per_invocation_results=[
+            PerInvocationResult(
+                actual_invocation=invocation, score=score, eval_status=status
+            )
+            for invocation in actual_invocations
+        ],
     )
 ```
 
@@ -268,13 +293,13 @@ from google.adk.evaluation.metric_evaluator_registry import DEFAULT_METRIC_EVALU
 
 DEFAULT_METRIC_EVALUATOR_REGISTRY.register_evaluator(
     metric_info=MetricInfo(
-        metric_name="response_length",
-        description="Penalizes over-long final responses.",
+        metric_name="final_state_match",
+        description="Checks the expected final session state.",
         metric_value_info=MetricValueInfo(
             interval=Interval(min_value=0.0, max_value=1.0)
         ),
     ),
-    evaluator=ResponseLengthEvaluator,
+    evaluator=FinalStateMatchEvaluator,
 )
 ```
 
@@ -312,10 +337,10 @@ that two-stage validation works.
     the first registration and logs it. Constructing any registry also emits an
     experimental-feature warning, `MetricEvaluatorRegistry` being marked
     experimental.
-*   **Nothing is re-exported at package level.** Import from
+*   **Evaluator and registry types use module imports.** Import from
     `google.adk.evaluation.evaluator` and
-    `google.adk.evaluation.metric_evaluator_registry` directly; the package
-    `__init__` exports only `AgentEvaluator`.
+    `google.adk.evaluation.metric_evaluator_registry` directly.
+    `EvaluationContext` is also available from `google.adk.evaluation`.
 *   **Errors are swallowed.** A metric that raises degrades to `NOT_EVALUATED`
     with a log line rather than surfacing the exception to the caller.
 

diff --git a/src/google/adk/evaluation/__init__.py b/src/google/adk/evaluation/__init__.py
@@ -14,9 +14,11 @@
 
 import logging
 
+from .evaluator import EvaluationContext
+
 logger = logging.getLogger('google_adk.' + __name__)
 
-__all__ = []
+__all__ = ['EvaluationContext']
 
 try:
   from .agent_evaluator import AgentEvaluator

diff --git a/src/google/adk/evaluation/base_eval_service.py b/src/google/adk/evaluation/base_eval_service.py
@@ -27,6 +27,7 @@
 
 from .constants import DEFAULT_LIVE_TIMEOUT_SECONDS
 from .eval_case import Invocation
+from .eval_case import SessionState
 from .eval_metrics import EvalMetric
 from .eval_result import EvalCaseResult
 
@@ -171,6 +172,18 @@ class InferenceResult(BaseModel):
       description="""Error message if the inference failed.""",
   )
 
+  initial_session_state: Optional[SessionState] = Field(default=None)
+  """Actual state before the first turn, or None if no snapshot was captured.
+
+  For a reused session, this is its stored state, not SessionInput.state.
+  """
+
+  final_session_state: Optional[SessionState] = Field(default=None)
+  """Actual state after inference, or None if the session was unavailable.
+
+  Older inference results without state snapshots also leave this as None.
+  """
+
 
 class EvaluateRequest(BaseModel):
   model_config = ConfigDict(

diff --git a/src/google/adk/evaluation/evaluation_generator.py b/src/google/adk/evaluation/evaluation_generator.py
@@ -65,6 +65,7 @@
 from .eval_case import InvocationEvent
 from .eval_case import InvocationEvents
 from .eval_case import SessionInput
+from .eval_case import SessionState
 from .eval_set import EvalSet
 from .request_intercepter_plugin import _RequestIntercepterPlugin
 from .simulation.user_simulator import BaseUserSimulatorConfig
@@ -158,6 +159,20 @@ async def _get_or_create_eval_session(
   )
 
 
+async def _capture_final_session_state(
+    session_service: BaseSessionService, session: Session
+) -> Optional[SessionState]:
+  """Reads the final state from storage, since the runner may use another copy."""
+  final_session = await session_service.get_session(
+      app_name=session.app_name,
+      user_id=session.user_id,
+      session_id=session.id,
+  )
+  return (
+      copy.deepcopy(final_session.state) if final_session is not None else None
+  )
+
+
 # Keyword-argument names accepted by `Runner`, used when building the eval
 # Runner kwargs so the strings are not duplicated at each call site.
 _APP_NAME_KEY = "app_name"
@@ -747,12 +762,18 @@ async def _generate_inferences_from_root_agent_live(
       memory_service: Optional[BaseMemoryService] = None,
       live_timeout_seconds: int = DEFAULT_LIVE_TIMEOUT_SECONDS,
       app: Optional[App] = None,
+      session_state_callback: Optional[
+          Callable[[SessionState, Optional[SessionState]], None]
+      ] = None,
   ) -> list[Invocation]:
     """Scrapes the root agent in coordination with the user simulator in live mode.
 
     Mirrors `_generate_inferences_from_root_agent`: when `app` is provided the
     Runner carries the App's plugins and configuration, otherwise the bare
     `root_agent` is used.
+
+    If set, `session_state_callback` receives detached initial and final states
+    after the live session and runner close. A missing final session yields None.
     """
     if not session_service:
       session_service = InMemorySessionService()
@@ -763,6 +784,11 @@ async def _generate_inferences_from_root_agent_live(
     session = await _get_or_create_eval_session(
         session_service, initial_session, session_id
     )
+    initial_state = (
+        copy.deepcopy(session.state)
+        if session_state_callback is not None
+        else {}
+    )
     app_name = session.app_name
     user_id = session.user_id
     session_id = session.id
@@ -852,11 +878,18 @@ async def _generate_inferences_from_root_agent_live(
               events, request_intercepter_plugin
           )
       )
-      return EvaluationGenerator.convert_events_to_eval_invocations(
+      inferences = EvaluationGenerator.convert_events_to_eval_invocations(
           EvaluationGenerator._normalize_live_transcriptions(events),
           app_details_by_invocation_id,
       )
 
+    if session_state_callback is not None:
+      session_state_callback(
+          initial_state,
+          await _capture_final_session_state(session_service, session),
+      )
+    return inferences
+
   @staticmethod
   async def _generate_inferences_from_root_agent(
       root_agent: BaseAgent | BaseNode,
@@ -868,6 +901,9 @@ async def _generate_inferences_from_root_agent(
       artifact_service: Optional[BaseArtifactService] = None,
       memory_service: Optional[BaseMemoryService] = None,
       app: Optional[App] = None,
+      session_state_callback: Optional[
+          Callable[[SessionState, Optional[SessionState]], None]
+      ] = None,
   ) -> list[Invocation]:
     """Scrapes the root agent in coordination with the user simulator.
 
@@ -877,6 +913,9 @@ async def _generate_inferences_from_root_agent(
     application-wide configuration. Otherwise the Runner is built from
     the bare `root_agent` with only the internal eval plugins, matching
     the legacy behavior.
+
+    If set, `session_state_callback` receives detached initial and final states
+    after the runner closes. A missing final session yields None.
     """
 
     if not session_service:
@@ -888,6 +927,11 @@ async def _generate_inferences_from_root_agent(
     session = await _get_or_create_eval_session(
         session_service, initial_session, session_id
     )
+    initial_state = (
+        copy.deepcopy(session.state)
+        if session_state_callback is not None
+        else {}
+    )
     app_name = session.app_name
     user_id = session.user_id
     session_id = session.id
@@ -961,10 +1005,17 @@ async def _generate_inferences_from_root_agent(
               events, request_intercepter_plugin
           )
       )
-      return EvaluationGenerator.convert_events_to_eval_invocations(
+      inferences = EvaluationGenerator.convert_events_to_eval_invocations(
           events, app_details_by_invocation_id, durations_by_invocation_id
       )
 
+    if session_state_callback is not None:
+      session_state_callback(
+          initial_state,
+          await _capture_final_session_state(session_service, session),
+      )
+    return inferences
+
   @staticmethod
   def convert_events_to_eval_invocations(
       events: list[Event],