Skip to content

Robustness for the claude/codex backends on Windows: argv overflow, subprocess encoding, tolerant JSON, test-eval dirs - #77

Closed
samuelgoofus-boop wants to merge 1 commit into
microsoft:mainfrom
samuelgoofus-boop:fix/claude-codex-windows-robustness
Closed

samuelgoofus-boop wants to merge 1 commit into
microsoft:mainfrom
samuelgoofus-boop:fix/claude-codex-windows-robustness

Conversation

@samuelgoofus-boop

Copy link
Copy Markdown
Contributor

PR: Robustness for the claude/codex backends on Windows

What

Independent robustness fixes surfaced running SkillOpt end-to-end on the bundled
claude backend (local Claude CLI, claude_chat) on Windows. Six files, ~100
lines added / 4 changed; no behavior change on the OpenAI/GPT happy path.

1. skillopt/engine/trainer.py — create the test-eval dirs before writing.
test_eval_baseline/, test_eval/, and test_eval_final/ are each written to
(summary.json) before being created. When a rollout yields zero predictions the
dir never gets made by the rollout side, so the write raises FileNotFoundError
on Windows. Adds os.makedirs(..., exist_ok=True) at all three sites (the third,
the test_eval_final else-branch, mirrors the if-branch that already had it).

2. skillopt/model/claude_backend.py — keep large content off argv, pin UTF-8.

  • (a) Prompt via stdin, not argv. The prompt rode on the command line
    (cmd + [prompt_for_cli]). Windows caps a command line at ~32 KB, so a large
    optimizer prompt — the success-analyst minibatch carrying several report
    trajectories, i.e. as soon as use_meta_skill / use_slow_update is on or
    minibatch_size is raised — overflows with
    [WinError 206] The filename or extension is too long and kills the run after
    retries. claude -p reads the prompt from stdin when no positional arg is given.
  • (b) System prompt via --append-system-prompt-file, not argv. The ~32 KB
    cap applies to the sum of all argv, and the system prompt here is the skill
    being optimized — which SkillOpt grows over training. Once it passes ~30 KB it
    re-hits WinError 206 even with the prompt on stdin. Writing it to a temp file
    removes the last unbounded term from argv.
  • (c) Pin subprocess encoding to utf-8 (errors="replace"). With
    text=True and no encoding=, stdin is encoded with the system codepage; on a
    zh-CN box (cp936/GBK) a prompt containing an emoji or some Latin-1 chars raises
    UnicodeEncodeError before the CLI even starts, failing every retry.

3. skillopt/model/codex_backend.py — same UTF-8 encoding pin on its
subprocess.run(input=...) call (identical unpinned-encoding pattern).

4. skillopt/utils/json_utils.py — tolerant, unambiguous-only JSON repair.
Strict json.loads rejects valid-looking JSON that non-OpenAI backends emit
(unescaped ASCII quotes inside CJK string values, trailing commas); extract_json
then returns None and the reflect step silently treats it as "0 edits" — the
skill never improves, with no error logged. Adds a json_repair fallback, but
only on a single unambiguous object
: a balanced-brace extractor plus a
refuse-on-multiple-objects guard. This is deliberate — feeding the old greedy
{.*} span (or the raw text) to json_repair makes it silently return one of
several objects (empirically the wrong/discarded one from a chain-of-thought
"scratch + final" response), which is worse than None (None is detectable and
retryable; a wrong-but-valid edit is applied blind). json_repair is added to
requirements.txt and the claude/qwen extras (it otherwise no-ops on a fresh
install — dropping the very edits it exists to save — now with a one-time warning).

Why

All surfaced running the loop on Windows + claude_chat. Fix 2a/2b hard-kills any
non-trivial run ([WinError 206]) the moment prompts/skills grow. Fix 4 is a
silent failure: the analyst returns good edits, the run logs → 0 edits /
reflect_s=0.0, and the skill is frozen across every step. After the fixes a small
run optimizes normally (seed skill hard 0.0 → 1.0, generalizing to held-out
items) and larger runs (use_meta_skill + use_slow_update, multi-epoch) no
longer crash.

Testing

  • Fix 2a: a >32 KB prompt that raised [WinError 206] on argv completes via
    stdin; confirmed claude -p reads the piped prompt and returns JSON.
  • Fix 2b: --append-system-prompt-file confirmed supported by the installed
    CLI; the skill-as-system-prompt path no longer occupies argv.
  • Fix 2c / 3: under PYTHONUTF8=0 (cp936), a prompt with an emoji / Latin-1
    char raised UnicodeEncodeError; with the pin it succeeds.
  • Fix 4: 9-case unit check — genuine single malformed objects (trailing comma,
    unescaped CJK quotes, nested, fenced) still repair to the right dict; a
    {KEEP} (scratch: {DISCARD}) / first {KEEP} final {REWRITE} multi-object
    response returns None (never the discarded object), where the prior fallback
    returned the wrong one.
  • Patch hygiene: applies cleanly to pristine fc1f827 (git apply --check
    and git am both OK); patched tree byte-compiles; OpenAI path untouched
    (json_repair only reached after strict parse fails; makedirs idempotent; stdin
    and encoding are backend-agnostic).

Notes

If you'd prefer json_repair not be a dependency, the fallback degrades gracefully
(try/except ModuleNotFoundError → returns None with a one-time warning), so it
can stay extras-only.

…ubprocess encoding, tolerant JSON, test-eval dirs

Fixes surfaced running SkillOpt end-to-end on the bundled `claude` backend
(local Claude CLI) on Windows. None changes the OpenAI/GPT happy path.

1. skillopt/engine/trainer.py — the final test-eval directory
   (test_eval_final/) is written to before being created; add
   os.makedirs(..., exist_ok=True), matching the two sibling test-eval dirs.
   Without it, summary.json raises FileNotFoundError when a rollout yields
   zero predictions.

2. skillopt/model/claude_backend.py
   a. Pass the prompt via stdin (not argv): on Windows the whole command line
      is capped at ~32 KB and a large optimizer prompt (the success-analyst
      minibatch carrying several report trajectories) overflows it with
      [WinError 206], killing the run after retries.
   b. Pass the system prompt via --append-system-prompt-file (a temp file),
      not argv. The system prompt here is the skill being optimized, which
      SkillOpt grows over training; since the ~32 KB cap applies to the SUM of
      all argv, a grown skill would re-hit [WinError 206] even with the prompt
      on stdin.
   c. Pin the subprocess encoding to utf-8 (errors="replace"). With text=True
      and no encoding=, stdin is encoded with the system codepage; on a zh-CN
      box (cp936/GBK) a prompt containing an emoji or some Latin-1 characters
      raises UnicodeEncodeError before the CLI even starts, failing every retry.

3. skillopt/model/codex_backend.py — the same utf-8 encoding pin on its
   subprocess.run(input=...) call (identical unpinned-encoding pattern).

4. skillopt/utils/json_utils.py — extract_json() returned None for valid-
   looking JSON that strict json.loads rejects (unescaped ASCII quotes inside
   CJK string values, trailing commas), silently dropping the analyst's edits
   on non-schema backends (Claude/Qwen): reflect produces N edits, 0 applied.
   Add a json_repair fallback, but only on a single unambiguous object — a
   balanced-brace extractor plus a refuse-on-multiple-objects guard — so a
   chain-of-thought "scratch + final" response can't make repair silently
   return the wrong (discarded) object, which would be worse than None (None is
   detectable and retryable; a wrong-but-valid edit is applied blind). Declare
   json_repair in requirements.txt and the claude/qwen optional extras so the
   fallback is actually present (it otherwise no-ops, dropping edits silently).

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR improves Windows robustness for the non-OpenAI backends (Claude CLI / Codex exec) and makes JSON parsing more tolerant to slightly malformed model outputs, while aiming to preserve behavior on the OpenAI/GPT happy path.

Changes:

  • Ensure test_eval* output directories exist before writing results in the trainer loop.
  • Avoid Windows command-line length limits and encoding issues by sending prompts via stdin and pinning subprocess text encoding to UTF-8 (Claude + Codex).
  • Add a guarded json_repair-based fallback to extract_json() and include json_repair as a dependency / extra.

Reviewed changes

Copilot reviewed 5 out of 6 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
skillopt/engine/trainer.py Creates test_eval* directories before writing/rolling out to avoid Windows FileNotFoundError.
skillopt/model/claude_backend.py Moves prompt off argv (stdin), moves system prompt to a temp file, and pins subprocess encoding to UTF-8 for Windows robustness.
skillopt/model/codex_backend.py Pins subprocess encoding to UTF-8 to avoid locale/codepage encoding failures.
skillopt/utils/json_utils.py Adds a “single-unambiguous-object only” tolerant JSON repair fallback.
requirements.txt Adds json_repair dependency.
pyproject.toml Adds json_repair to claude/qwen optional dependency sets.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment on lines +18 to +22
i, n = 0, len(text)
while i < n:
if text[i] != "{":
i += 1
continue
Yifan Yang (Yif-Yang) added a commit that referenced this pull request Jun 23, 2026
…k) (#79)

* Robustness for the claude/codex backends on Windows: argv overflow, subprocess encoding, tolerant JSON, test-eval dirs

Fixes surfaced running SkillOpt end-to-end on the bundled `claude` backend
(local Claude CLI) on Windows. None changes the OpenAI/GPT happy path.

1. skillopt/engine/trainer.py — the final test-eval directory
   (test_eval_final/) is written to before being created; add
   os.makedirs(..., exist_ok=True), matching the two sibling test-eval dirs.
   Without it, summary.json raises FileNotFoundError when a rollout yields
   zero predictions.

2. skillopt/model/claude_backend.py
   a. Pass the prompt via stdin (not argv): on Windows the whole command line
      is capped at ~32 KB and a large optimizer prompt (the success-analyst
      minibatch carrying several report trajectories) overflows it with
      [WinError 206], killing the run after retries.
   b. Pass the system prompt via --append-system-prompt-file (a temp file),
      not argv. The system prompt here is the skill being optimized, which
      SkillOpt grows over training; since the ~32 KB cap applies to the SUM of
      all argv, a grown skill would re-hit [WinError 206] even with the prompt
      on stdin.
   c. Pin the subprocess encoding to utf-8 (errors="replace"). With text=True
      and no encoding=, stdin is encoded with the system codepage; on a zh-CN
      box (cp936/GBK) a prompt containing an emoji or some Latin-1 characters
      raises UnicodeEncodeError before the CLI even starts, failing every retry.

3. skillopt/model/codex_backend.py — the same utf-8 encoding pin on its
   subprocess.run(input=...) call (identical unpinned-encoding pattern).

4. skillopt/utils/json_utils.py — extract_json() returned None for valid-
   looking JSON that strict json.loads rejects (unescaped ASCII quotes inside
   CJK string values, trailing commas), silently dropping the analyst's edits
   on non-schema backends (Claude/Qwen): reflect produces N edits, 0 applied.
   Add a json_repair fallback, but only on a single unambiguous object — a
   balanced-brace extractor plus a refuse-on-multiple-objects guard — so a
   chain-of-thought "scratch + final" response can't make repair silently
   return the wrong (discarded) object, which would be worse than None (None is
   detectable and retryable; a wrong-but-valid edit is applied blind). Declare
   json_repair in requirements.txt and the claude/qwen optional extras so the
   fallback is actually present (it otherwise no-ops, dropping edits silently).

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
(cherry picked from commit dca74a6)

* fix(json_utils): harden tolerant JSON fallback from PR #77

Follow-up fixes on top of the cherry-picked Windows-robustness change:

1. Make _top_level_brace_objects() fully string-aware in its OUTER scan, not
   just inside an object. A '{' inside quoted prose (e.g. '"set it to {x}"')
   no longer starts a candidate object, so extract_json() returns None for
   prose pseudo-JSON instead of repairing it into a bogus dict — which would
   be strictly worse than dropping the edit, since extract_json feeds the
   optimizer's skill edits.

2. Pick the repair candidate BEFORE importing json_repair, so the missing-
   dependency RuntimeWarning only fires when there is genuinely a single
   malformed object that could have been repaired. Ordinary no-JSON / prose
   replies (the common case) now return None silently instead of warning on
   every call.

3. Resolve dependency-metadata inconsistency: json_repair is optional, so add
   it to the `all` extra (it was already in `claude`/`qwen`) and demote it
   from a hard requirement to an optional/commented entry in requirements.txt,
   matching the project's convention for backend-specific deps.

Adds regression tests for prose-with-braces (-> None), no-warning-on-plain-
text, single-object repair, and multi-object ambiguity. Existing 22 json
tests still pass with and without json_repair installed.

Co-Authored-By: Claude <[email protected]>

---------

Co-authored-by: samuelgoofus-boop <[email protected]>
Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>
Yifan Yang (Yif-Yang) added a commit that referenced this pull request Jun 23, 2026
Add an Acknowledgements section crediting @samuelgoofus-boop for the
Windows-robustness work on the Claude/Codex backends (originally #77,
merged via #79).

Co-authored-by: Claude <[email protected]>
@Yif-Yang

Copy link
Copy Markdown
Contributor

Superseded by #79, which cherry-picked this work (authorship preserved via Co-authored-by) and added follow-up hardening to the tolerant-JSON fallback. Now merged into main. Thank you samuelgoofus-boop — credited in the README Acknowledgements (#80). Closing this PR as its changes are fully incorporated.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants