pstack: fix eval bakeoff workflow

cursoragent · cursoragent · commit 01ad992a572e · 2026-07-19T03:20:47.000Z
diff --git a/pstack/skills/poteto-mode/playbooks/eval.md b/pstack/skills/poteto-mode/playbooks/eval.md
@@ -18,13 +18,13 @@ Evals are blinded, one-shot bakeoffs for deciding whether to promote or reject a
 **Steps:**
 
 1. **Frame.** State the variant and the promote-or-reject claim. Write a judge-only rubric with 3-6 concrete criteria. Grade task success and the intended behavioral shape. Never make a turn-1 skill load, a particular file read, a citation, or "did the skill trigger?" a pass condition.
-2. **Build the ablation arms.** When adding, rewriting, or supplying a skill, compare the variant with that skill absent. Use the prior version when absence would make the task artificial. Give both arms the same organic prompt and project skeleton. The control is just another sanitized label. Promote only if the variant-on arm beats the variant-off arm on the rubric.
-3. **Author an organic prompt set.** Include at least one task where the behavior should apply. If the variant changes a description, routing, sticky behavior, or when-to-apply rule, include at least one task where it should not engage and add false-positive cost to the rubric. Write what a user would type, with no leakage of what is measured.
-4. **Set up isolated trials.** Create a fresh per-trial workspace with only the arm's variant and context an organic task would have. Keep the project skeleton identical across arms. A cheap deterministic preflight may confirm setup facts, but it only aids synthesis. Regex or `should_trigger` checks do not judge task success or decide promotion.
-5. **Run 2-3 one-shot trials per prompt and arm** through the **arena** skill's Phase B. Match the model mix across arms and run candidates in parallel. No retries, coaching, or repair. When budget binds, prefer 2 trials across fewer models over 1 trial across more models.
-6. **Spawn one blinded judge** on a different model family per the **arena** skill's Phase C. In one pass, it scores every output by randomized sanitized label against the same rubric and marks each criterion and output pass or fail.
-7. **Inspect transcripts after scoring to explain how, not to decide pass or fail.** Read each candidate's local transcript under the active workspace's `agent-transcripts/` directory (the system prompt names this path). Do not glob across `~/.cursor/projects/*/`; that crosses workspace boundaries and reads private chats from unrelated projects. Use the transcript to verify isolation, see what the candidate read, and explain the output. It is synthesis evidence, not a pass gate.
-8. **Read every output yourself** end to end. Report pass rates by arm and prompt, then compare your read with the judge's verdict. Promote only when the variant-on arm beats the variant-off arm overall without adding false positives. Otherwise reject it. Explain disagreements as judge bias, contamination, or rubric ambiguity.
+2. **Author an organic prompt set.** Include at least one task where the behavior should apply. If the variant changes a description, routing, sticky behavior, or when-to-apply rule, include at least one task where it should not engage and add false-positive cost to the rubric. Write what a user would type, with no leakage of what is measured. If the task prompt itself is the target, write matched current and proposed versions here; otherwise every arm gets the same prompt.
+3. **Build comparison arms for every target.** The variant arm gets the proposed skill, structure, or prompt; the control gets the current version, or the skill absent only when absence is the realistic baseline. Use the prior skill when absence would make the task artificial. Hold the project skeleton, model mix, and every non-target input constant; when the task prompt is the target, its two authored versions are the sole difference. The control is just another sanitized label. Promotion always compares the variant arm with the control.
+4. **Set up isolated trials.** Create a fresh per-trial workspace containing only that arm's variant and the context an organic task would have. Keep the project skeleton identical across arms. A fresh workspace alone does not remove skills installed through workspace `.cursor/skills/`, user `~/.cursor/skills/`, or plugin `~/.cursor/plugins/` paths: for a skill-absent arm, use an isolated profile or disable every candidate-visible copy, then preflight the resolved skill sources and fail setup if the skill remains visible. Record each trial's exact workspace path and session or transcript ID as orchestrator-only metadata. A cheap deterministic preflight may confirm setup facts, but it only aids synthesis. Regex or `should_trigger` checks do not judge task success or decide promotion.
+5. **Run 2-3 one-shot trials per prompt and arm.** Launch every runner directly in its recorded workspace with that arm's isolated context. Fan out the independent launches in parallel, but never give runners shared grounding or candidate-visible files across workspaces. Match model and trial pairings across arms. Ask only for the organic task's output, not a graft-oriented rationale. Missing output is a failed trial; there are no retries, coaching, or repairs. When budget binds, prefer 2 trials across fewer models over 1 trial across more models.
+6. **Spawn one blinded judge** on a different model family after every trial finishes. In one pass, it scores every output by randomized sanitized label against the same rubric and marks each criterion and output pass or fail. Do not invoke the arena workflow, recommend a base, graft, or combine outputs; this bakeoff ends with arm-level scoring.
+7. **Inspect transcripts after scoring to explain how, not to decide pass or fail.** Read only each recorded trial transcript from its exact workspace's transcript directory, normally `~/.cursor/projects/<trial-workspace-slug>/agent-transcripts/`, using the session or transcript ID recorded during setup. Derive `<trial-workspace-slug>` from that workspace path; do not glob across `~/.cursor/projects/*/` or inspect any unregistered workspace. Use the transcript to verify isolation, see what the candidate read, and explain the output. It is synthesis evidence, not a pass gate.
+8. **Read every output yourself** end to end. Report pass rates by arm and prompt, then compare your read with the judge's verdict. Promote only when the variant arm beats the control overall without adding false positives. Otherwise reject it. Explain disagreements as judge bias, contamination, or rubric ambiguity.
 
 **Related:** Shipped skills may keep a separate standing regression pack of 5-20 cases. It is distinct from this bakeoff.
 
