pstack: clarify eval bakeoff safeguards

cursoragent · cursoragent · commit e76324318826 · 2026-07-19T05:30:50.000Z
diff --git a/pstack/skills/poteto-mode/playbooks/eval.md b/pstack/skills/poteto-mode/playbooks/eval.md
@@ -20,11 +20,11 @@ Evals are blinded, one-shot bakeoffs for deciding whether to promote or reject a
 1. **Frame.** State the variant and the promote-or-reject claim. Write a judge-only rubric with 3-6 concrete criteria. Grade task success and the intended behavioral shape. Never make a turn-1 skill load, a particular file read, a citation, or "did the skill trigger?" a pass condition.
 2. **Author an organic prompt set.** Include at least one task where the behavior should apply. If the variant changes a description, routing, sticky behavior, or when-to-apply rule, include at least one task where it should not engage and add false-positive cost to the rubric. Write what a user would type. Never name the behavioral tell the rubric grades (if you measure dated headings, do not say "dated note" in the prompt). No other leakage of what is measured. If the task prompt itself is the target, write matched current and proposed versions here; otherwise every arm gets the same prompt.
 3. **Build comparison arms.** Before editing, snapshot any prior skill contents the control will need. Variant gets the proposed skill, structure, or prompt. Control gets the current version. For skill presence or content changes, the control may be skill-absent when absence is the realistic baseline, otherwise the snapshotted prior skill. Never plant the ablation-target skill into a skill-absent control. Hold the project skeleton, model mix, and every non-target input constant. The control is another sanitized label. Promote only when the variant beats the control on the rubric.
-4. **Set up isolated trials.** Fresh per-trial workspace with only that arm's variant and organic-task context. Identical project skeleton across arms. A fresh workspace does not clear skills from workspace `.cursor/skills/`, user `~/.cursor/skills/`, or plugin installs: for a skill-absent arm, use a workspace-local isolation (or disable that is restored before any other arm runs). Never apply a shared user/plugin disable that also strips the skill from the variant arm. Preflight resolved sources and fail setup if the skill remains visible on a skill-absent arm or missing on a variant arm. For sticky, mode, description, or other always-on triggers, preflight that the variant reaches the candidate the way production does (reminder in context, description always loaded, and so on). If the harness cannot inject it that way, stop: the bakeoff is invalid for that variant class. Record each trial's workspace path and transcript ID as orchestrator-only metadata. Cheap deterministic preflights aid synthesis only; they never replace the blinded rubric.
-5. **Run 2-3 one-shot trials per prompt and arm.** Launch each runner directly in its recorded workspace with that arm's isolated context. Fan out in parallel with no shared grounding and no candidate-visible files across workspaces. Match model and trial pairings across arms. If the skill ships across models, use at least two model families; matched pairings on one family are not enough. Ask only for the organic task output, not a graft rationale. Missing output fails the trial. No retries, coaching, or repair. When budget binds, prefer 2 trials on fewer models over 1 on many.
+4. **Set up isolated trials.** Fresh per-trial workspace with only that arm's variant and organic-task context. Identical project skeleton across arms. A fresh workspace does not clear skills from workspace `.cursor/skills/`, user `~/.cursor/skills/`, or plugin installs: for a skill-absent arm, use a workspace-local isolation (or disable that is restored before any other arm runs). Never apply a shared user/plugin disable that also strips the skill from the variant arm. Preflight resolved sources and fail setup if the skill remains visible on a skill-absent arm or missing on a variant arm. For sticky, mode, description, or other always-on triggers, preflight that each non-absent arm's skill reaches the candidate the way it would in production (reminder in context, description always loaded, and so on). If the harness cannot deliver every non-absent arm's skill that way, stop: the bakeoff is invalid for that variant class. Record each trial's workspace path and transcript ID as orchestrator-only metadata. Cheap deterministic preflights aid synthesis only; they never replace the blinded rubric.
+5. **Run 2-3 one-shot trials per prompt and arm.** Launch each runner directly in its recorded workspace with that arm's isolated context. Fan out in parallel with no shared grounding and no candidate-visible files across workspaces. Match model and trial pairings across arms. If the skill ships across models, use at least two model families; matched pairings on one family are not enough. Ask only for the organic task output, not a graft rationale. Missing output fails the trial. No retries, coaching, or repair. When budget binds, prefer 2 trials on fewer models over 1 on many, but never go below two model families for a skill that ships across models.
 6. **Spawn one blinded judge** on a different model family after every trial finishes. In one pass, score every output by randomized sanitized label against the same rubric. Mark each criterion and output pass or fail. Programmatic checks may filter obvious fails before the judge; they do not decide promote or reject. Do not run the arena pick/graft workflow. This bakeoff ends at arm-level scoring.
 7. **Inspect transcripts after scoring to explain how, not to decide pass or fail.** Read only the recorded transcript for each trial from that workspace's transcript directory (normally `~/.cursor/projects/<trial-workspace-slug>/agent-transcripts/`), using the session or transcript ID from setup. Derive the slug from the recorded workspace path. Do not glob across `~/.cursor/projects/*/` or open unregistered workspaces. Transcripts verify isolation and explain the output. They are not a pass gate.
-8. **Read the outputs yourself.** At small N, read every output end to end. At large N, read every fail plus a stated random sample of passes; silent skim is not enough. Report pass rates by arm and prompt, then compare with the judge. Promote only when the variant beats the control overall without adding false positives. Otherwise reject. Explain disagreements as judge bias, contamination, or rubric ambiguity.
+8. **Read the outputs yourself.** At small N, read every output end to end. At large N, read every fail and every output for prompts where the behavior should not engage, plus a stated random sample of the remaining passes; silent skim is not enough. Report pass rates by arm and prompt, then compare with the judge. Promote only when the variant beats the control overall without adding false positives. Otherwise reject. Explain disagreements as judge bias, contamination, or rubric ambiguity.
 
 **Related:** Shipped skills may keep a separate standing regression pack of 5-20 cases. It is distinct from this bakeoff.
 
