Skip to content

fix(core): drop session ID and order instructions for prompt-cache reuse - #51960

Merged
rekram1-node merged 3 commits into
v2from
instructions-env-last
Sep 29, 2026
Merged

rekram1-node merged 3 commits into
v2from
instructions-env-last

Conversation

@rekram1-node

@rekram1-node rekram1-node commented Sep 28, 2026 •

Copy link
Copy Markdown
Collaborator

The instructions baseline started with the <env> block, and its first line was the session ID. So every session's prompt differed from the start of the instructions block, and nothing after it could be reused from cache across sessions. In particular, sibling subagents with identical instructions never shared a cached prefix.

This PR removes the session ID from <env> and orders the instruction sources from most to least shared across sessions. The date and environment go last, just before per-session API entries, because the date changes daily and the environment by directory. Built-in session tools default to the current session, and cross-session prompts record their origin, so the model doesn't need its own ID. Removing it also fixes forks, which started out showing their source session's ID.

What the model sees

 [system] agent prompt
 [system] instructions
-  <env>
-    Current conversation session ID: ses_…   ← unique per session
-    Working directory: …
-    …
-  </env>
-  Today's date: …
   Code Mode catalog                          ← user-level
+  MCP guidance
+  References
+  Skills
   AGENTS.md (global, then project)           ← project-level
-  Skills
-  References
-  MCP guidance
+  Today's date: …                            ← changes daily
+  <env>
+    Working directory: …                     ← changes by directory
+    …
+  </env>
   API instruction entries                    ← per session
 [messages] …

Existing sessions re-render their baseline once in the new order. The removed line shows up as one appended environment update.

How we measured it

We replayed recorded V2 usage offline: 2,372 sessions over 63 days, 124k model steps across 86 projects. The data comes from one heavy user, and one project is 91% of the sessions, but every project with 15 or more cold starts ranks the layouts the same way.

  • Rendering. Each session's stored instruction values are re-rendered with the real Instructions renderers. The result matched recorded update text byte for byte (Code Mode 59/59, env 13/13, date 8/8, AGENTS.md 10/10), with 0 hash mismatches across 2,672 stored values.
  • Tokens. o200k counts. Anthropic counts were calibrated against its token-counting API on real baselines, at about 1.15× o200k. The tool and agent prompt block ahead of the instructions is sized from observed first-step cache reads per agent.
  • Cold starts. We measured each session's first request, the first request after a cache-lifetime gap, and the first request after a compaction. For each one we computed how much of the static prefix another warm session could have served. The prefix is tools, agent prompt and instructions, about 8.2k tokens on average.
  • Provider cache models:
    • OpenAI prefix caching: longest shared prefix, 1,024-token minimum, 128-token steps, only between requests with the same prompt_cache_key. We also modeled an optimistic variant that ignores routing.
    • Block caching: gpt-5.6-sol reuses only a byte-identical tools and instructions block. We confirmed this with live API probes.
    • Anthropic: hits only at cache breakpoints written by a warm request, org-wide.
  • Validation. For today's layout, the model predicted whether a first step would get a cache hit correctly for 98.1% of Claude subagents (0 false negatives) and 99.5% of GPT sessions. In the recorded data, instruction text was never shared across sessions, which is what the session ID predicts.
  • Uncertainty. 95% intervals come from bootstrapping over session trees.

Results

Share of the static prefix served from cache on cold starts, with 95% intervals and a 1-hour cache lifetime unless noted:

Layout OpenAI prefix, routed OpenAI prefix, optimistic Block caching (sol), routed Anthropic, 5 min
Before 17.7% [14–21] 35.8% [35–37] 0% 29.0% [27–31]
Date and env last, ID kept 44.8% [36–53] 76.6% [73–80] 0% 29.0%
This PR (ID removed, date and env last) 45.4% [36–54] 77.4% [74–81] 45.3% [36–54] 59.8% [54–65]

For sibling subagents, the pairs with realistic reuse under key routing, OpenAI prefix reuse goes from 38% to 93–95%.

  • The ID is what matters. Moving it to the end only helps prefix-caching OpenAI models. Block-caching models and Anthropic cache a system part all or nothing, so any ID in the system prompt prevents reuse there.
  • The order of the stable sources barely matters. Across all 120 orders of Code Mode, MCP, references, skills and AGENTS.md, the spread is at most 1–2 points, and only in the optimistic case. With key routing or on Anthropic it is under 0.1 points. The order used here puts user-level sources before project-level ones, which the data favors for sessions that share caches across projects.
  • Parent and child sessions share nothing either way. They run different agents with different tools, and tools come first in the prefix. The realistic gains are between siblings.

@rekram1-node rekram1-node changed the title fix(core): render date and environment after shared instructions fix(core): keep session-specific data out of the shared instructions prefix Sep 28, 2026
@rekram1-node rekram1-node changed the title fix(core): keep session-specific data out of the shared instructions prefix fix(core): drop session ID and order instructions for prompt-cache reuse Sep 29, 2026
@rekram1-node
rekram1-node enabled auto-merge (squash) September 29, 2026 00:29
@rekram1-node
rekram1-node merged commit bda798b into v2 Sep 29, 2026
9 of 10 checks passed
@rekram1-node
rekram1-node deleted the instructions-env-last branch September 29, 2026 00:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant