Repository navigation
Robustness for the claude/codex backends on Windows: argv overflow, subprocess encoding, tolerant JSON, test-eval dirs - #77
Conversation
…ubprocess encoding, tolerant JSON, test-eval dirs
Fixes surfaced running SkillOpt end-to-end on the bundled `claude` backend
(local Claude CLI) on Windows. None changes the OpenAI/GPT happy path.
1. skillopt/engine/trainer.py — the final test-eval directory
(test_eval_final/) is written to before being created; add
os.makedirs(..., exist_ok=True), matching the two sibling test-eval dirs.
Without it, summary.json raises FileNotFoundError when a rollout yields
zero predictions.
2. skillopt/model/claude_backend.py
a. Pass the prompt via stdin (not argv): on Windows the whole command line
is capped at ~32 KB and a large optimizer prompt (the success-analyst
minibatch carrying several report trajectories) overflows it with
[WinError 206], killing the run after retries.
b. Pass the system prompt via --append-system-prompt-file (a temp file),
not argv. The system prompt here is the skill being optimized, which
SkillOpt grows over training; since the ~32 KB cap applies to the SUM of
all argv, a grown skill would re-hit [WinError 206] even with the prompt
on stdin.
c. Pin the subprocess encoding to utf-8 (errors="replace"). With text=True
and no encoding=, stdin is encoded with the system codepage; on a zh-CN
box (cp936/GBK) a prompt containing an emoji or some Latin-1 characters
raises UnicodeEncodeError before the CLI even starts, failing every retry.
3. skillopt/model/codex_backend.py — the same utf-8 encoding pin on its
subprocess.run(input=...) call (identical unpinned-encoding pattern).
4. skillopt/utils/json_utils.py — extract_json() returned None for valid-
looking JSON that strict json.loads rejects (unescaped ASCII quotes inside
CJK string values, trailing commas), silently dropping the analyst's edits
on non-schema backends (Claude/Qwen): reflect produces N edits, 0 applied.
Add a json_repair fallback, but only on a single unambiguous object — a
balanced-brace extractor plus a refuse-on-multiple-objects guard — so a
chain-of-thought "scratch + final" response can't make repair silently
return the wrong (discarded) object, which would be worse than None (None is
detectable and retryable; a wrong-but-valid edit is applied blind). Declare
json_repair in requirements.txt and the claude/qwen optional extras so the
fallback is actually present (it otherwise no-ops, dropping edits silently).
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
There was a problem hiding this comment.
Pull request overview
This PR improves Windows robustness for the non-OpenAI backends (Claude CLI / Codex exec) and makes JSON parsing more tolerant to slightly malformed model outputs, while aiming to preserve behavior on the OpenAI/GPT happy path.
Changes:
- Ensure
test_eval*output directories exist before writing results in the trainer loop. - Avoid Windows command-line length limits and encoding issues by sending prompts via stdin and pinning subprocess text encoding to UTF-8 (Claude + Codex).
- Add a guarded
json_repair-based fallback toextract_json()and includejson_repairas a dependency / extra.
Reviewed changes
Copilot reviewed 5 out of 6 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| skillopt/engine/trainer.py | Creates test_eval* directories before writing/rolling out to avoid Windows FileNotFoundError. |
| skillopt/model/claude_backend.py | Moves prompt off argv (stdin), moves system prompt to a temp file, and pins subprocess encoding to UTF-8 for Windows robustness. |
| skillopt/model/codex_backend.py | Pins subprocess encoding to UTF-8 to avoid locale/codepage encoding failures. |
| skillopt/utils/json_utils.py | Adds a “single-unambiguous-object only” tolerant JSON repair fallback. |
| requirements.txt | Adds json_repair dependency. |
| pyproject.toml | Adds json_repair to claude/qwen optional dependency sets. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| i, n = 0, len(text) | ||
| while i < n: | ||
| if text[i] != "{": | ||
| i += 1 | ||
| continue |
…k) (#79) * Robustness for the claude/codex backends on Windows: argv overflow, subprocess encoding, tolerant JSON, test-eval dirs Fixes surfaced running SkillOpt end-to-end on the bundled `claude` backend (local Claude CLI) on Windows. None changes the OpenAI/GPT happy path. 1. skillopt/engine/trainer.py — the final test-eval directory (test_eval_final/) is written to before being created; add os.makedirs(..., exist_ok=True), matching the two sibling test-eval dirs. Without it, summary.json raises FileNotFoundError when a rollout yields zero predictions. 2. skillopt/model/claude_backend.py a. Pass the prompt via stdin (not argv): on Windows the whole command line is capped at ~32 KB and a large optimizer prompt (the success-analyst minibatch carrying several report trajectories) overflows it with [WinError 206], killing the run after retries. b. Pass the system prompt via --append-system-prompt-file (a temp file), not argv. The system prompt here is the skill being optimized, which SkillOpt grows over training; since the ~32 KB cap applies to the SUM of all argv, a grown skill would re-hit [WinError 206] even with the prompt on stdin. c. Pin the subprocess encoding to utf-8 (errors="replace"). With text=True and no encoding=, stdin is encoded with the system codepage; on a zh-CN box (cp936/GBK) a prompt containing an emoji or some Latin-1 characters raises UnicodeEncodeError before the CLI even starts, failing every retry. 3. skillopt/model/codex_backend.py — the same utf-8 encoding pin on its subprocess.run(input=...) call (identical unpinned-encoding pattern). 4. skillopt/utils/json_utils.py — extract_json() returned None for valid- looking JSON that strict json.loads rejects (unescaped ASCII quotes inside CJK string values, trailing commas), silently dropping the analyst's edits on non-schema backends (Claude/Qwen): reflect produces N edits, 0 applied. Add a json_repair fallback, but only on a single unambiguous object — a balanced-brace extractor plus a refuse-on-multiple-objects guard — so a chain-of-thought "scratch + final" response can't make repair silently return the wrong (discarded) object, which would be worse than None (None is detectable and retryable; a wrong-but-valid edit is applied blind). Declare json_repair in requirements.txt and the claude/qwen optional extras so the fallback is actually present (it otherwise no-ops, dropping edits silently). Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> (cherry picked from commit dca74a6) * fix(json_utils): harden tolerant JSON fallback from PR #77 Follow-up fixes on top of the cherry-picked Windows-robustness change: 1. Make _top_level_brace_objects() fully string-aware in its OUTER scan, not just inside an object. A '{' inside quoted prose (e.g. '"set it to {x}"') no longer starts a candidate object, so extract_json() returns None for prose pseudo-JSON instead of repairing it into a bogus dict — which would be strictly worse than dropping the edit, since extract_json feeds the optimizer's skill edits. 2. Pick the repair candidate BEFORE importing json_repair, so the missing- dependency RuntimeWarning only fires when there is genuinely a single malformed object that could have been repaired. Ordinary no-JSON / prose replies (the common case) now return None silently instead of warning on every call. 3. Resolve dependency-metadata inconsistency: json_repair is optional, so add it to the `all` extra (it was already in `claude`/`qwen`) and demote it from a hard requirement to an optional/commented entry in requirements.txt, matching the project's convention for backend-specific deps. Adds regression tests for prose-with-braces (-> None), no-warning-on-plain- text, single-object repair, and multi-object ambiguity. Existing 22 json tests still pass with and without json_repair installed. Co-Authored-By: Claude <[email protected]> --------- Co-authored-by: samuelgoofus-boop <[email protected]> Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>
Add an Acknowledgements section crediting @samuelgoofus-boop for the Windows-robustness work on the Claude/Codex backends (originally #77, merged via #79). Co-authored-by: Claude <[email protected]>
|
Superseded by #79, which cherry-picked this work (authorship preserved via |
PR: Robustness for the claude/codex backends on Windows
What
Independent robustness fixes surfaced running SkillOpt end-to-end on the bundled
claudebackend (local Claude CLI,claude_chat) on Windows. Six files, ~100lines added / 4 changed; no behavior change on the OpenAI/GPT happy path.
1.
skillopt/engine/trainer.py— create the test-eval dirs before writing.test_eval_baseline/,test_eval/, andtest_eval_final/are each written to(
summary.json) before being created. When a rollout yields zero predictions thedir never gets made by the rollout side, so the write raises
FileNotFoundErroron Windows. Adds
os.makedirs(..., exist_ok=True)at all three sites (the third,the
test_eval_finalelse-branch, mirrors the if-branch that already had it).2.
skillopt/model/claude_backend.py— keep large content off argv, pin UTF-8.(
cmd + [prompt_for_cli]). Windows caps a command line at ~32 KB, so a largeoptimizer prompt — the success-analyst minibatch carrying several report
trajectories, i.e. as soon as
use_meta_skill/use_slow_updateis on orminibatch_sizeis raised — overflows with[WinError 206] The filename or extension is too longand kills the run afterretries.
claude -preads the prompt from stdin when no positional arg is given.--append-system-prompt-file, not argv. The ~32 KBcap applies to the sum of all argv, and the system prompt here is the skill
being optimized — which SkillOpt grows over training. Once it passes ~30 KB it
re-hits WinError 206 even with the prompt on stdin. Writing it to a temp file
removes the last unbounded term from argv.
utf-8(errors="replace"). Withtext=Trueand noencoding=, stdin is encoded with the system codepage; on azh-CN box (cp936/GBK) a prompt containing an emoji or some Latin-1 chars raises
UnicodeEncodeErrorbefore the CLI even starts, failing every retry.3.
skillopt/model/codex_backend.py— same UTF-8 encoding pin on itssubprocess.run(input=...)call (identical unpinned-encoding pattern).4.
skillopt/utils/json_utils.py— tolerant, unambiguous-only JSON repair.Strict
json.loadsrejects valid-looking JSON that non-OpenAI backends emit(unescaped ASCII quotes inside CJK string values, trailing commas);
extract_jsonthen returns
Noneand the reflect step silently treats it as "0 edits" — theskill never improves, with no error logged. Adds a
json_repairfallback, butonly on a single unambiguous object: a balanced-brace extractor plus a
refuse-on-multiple-objects guard. This is deliberate — feeding the old greedy
{.*}span (or the raw text) tojson_repairmakes it silently return one ofseveral objects (empirically the wrong/discarded one from a chain-of-thought
"scratch + final" response), which is worse than
None(None is detectable andretryable; a wrong-but-valid edit is applied blind).
json_repairis added torequirements.txtand theclaude/qwenextras (it otherwise no-ops on a freshinstall — dropping the very edits it exists to save — now with a one-time warning).
Why
All surfaced running the loop on Windows +
claude_chat. Fix 2a/2b hard-kills anynon-trivial run (
[WinError 206]) the moment prompts/skills grow. Fix 4 is asilent failure: the analyst returns good edits, the run logs
→ 0 edits/reflect_s=0.0, and the skill is frozen across every step. After the fixes a smallrun optimizes normally (seed skill
hard 0.0 → 1.0, generalizing to held-outitems) and larger runs (
use_meta_skill+use_slow_update, multi-epoch) nolonger crash.
Testing
[WinError 206]on argv completes viastdin; confirmed
claude -preads the piped prompt and returns JSON.--append-system-prompt-fileconfirmed supported by the installedCLI; the skill-as-system-prompt path no longer occupies argv.
PYTHONUTF8=0(cp936), a prompt with an emoji / Latin-1char raised
UnicodeEncodeError; with the pin it succeeds.unescaped CJK quotes, nested, fenced) still repair to the right dict; a
{KEEP} (scratch: {DISCARD})/first {KEEP} final {REWRITE}multi-objectresponse returns
None(never the discarded object), where the prior fallbackreturned the wrong one.
fc1f827(git apply --checkand
git amboth OK); patched tree byte-compiles; OpenAI path untouched(json_repair only reached after strict parse fails; makedirs idempotent; stdin
and encoding are backend-agnostic).
Notes
If you'd prefer
json_repairnot be a dependency, the fallback degrades gracefully(
try/except ModuleNotFoundError→ returnsNonewith a one-time warning), so itcan stay extras-only.