Skip to content

Remove strict plan verification and make executor-owned completion the only mode - #89

Merged
DevMando merged 1 commit into
mainfrom
codex/remove-strict-plan-verification
Sep 6, 2026
Merged

DevMando merged 1 commit into
mainfrom
codex/remove-strict-plan-verification

Conversation

@DevMando

@DevMando DevMando commented Sep 6, 2026

Copy link
Copy Markdown
Owner

Why this was done

PR #88 changed plan execution so a step completes on the executor's own evidence, and kept the old per-step model verifier alive behind strictPlanVerification as an escape hatch. That left two execution modes in the tree: two sets of failure semantics, two resume paths, and a whole pause-and-retry UI in both hosts, all maintained for a flag that was off by default and not expected to be turned on.

Two modes is the expensive part. The strict path was also the complicated one — three verdict response formats, a sandboxed read-only evidence follow-up, retry counters, and a "verification unavailable" pause state that only it could produce. Every future change to the planner had to be correct under both.

This removes the strict path and makes executor-owned completion the only mode.

What changed

Deleted: PlanStepVerifier (the three-format verdict reader), PlanQualityReview (the pre-approval proposal review), PlanEvidenceFollowup and its tool sandbox in AgentFunctionMiddleware/InvocationScope, the strictPlanVerification config key, PlanVerificationResult/PlanVerificationStatus, and TaskStep.EvidenceFollowupUsed with its checkpoint mirrors.

Also removed: the verification-unavailable pause path end to end — PlanStepOutcomeKind.VerificationUnavailable, VerificationRetryCounts, the branch in PlanTriageExecutor, TaskProgressType.StepVerificationUnavailable, and the prompt it drove in the CLI. (The Desktop half is in the paired MandoCode.Desktop PR.) That machinery existed to recover from a verifier that no longer runs.

Net: 378 insertions, 1100 deletions across 28 files.

What can still fail a step

Removing the verifier does not mean a step passes just by claiming it did. Completion is now decided entirely by signals that come from the executor's own report or from observed tool history — never from a judgement about whether the work was good:

  1. An explicit FAILED report, with its stated blocker.
  2. Stale checks — tool ordering shows the last relevant edit landed after the passing check, so the recorded pass does not describe the current files.
  3. No tool evidence at all — a step that touched nothing produced no deliverable.
  4. On the final quality phase only: a recognized test command whose last recorded exit code is nonzero.

Signals 2 and 3 were previously consulted only on the strict path. Promoting them rather than deleting them is the substantive decision in this PR: they are mechanical, cost nothing, and are the two things a confident-but-wrong executor cannot talk its way past. Without them, AssessFreshness would have become decorative code.

When a step both reports a failure and has stale checks, the repair instruction now carries both, so a rerun-after-edit instruction is not masked by the more specific blocker.

Impact

  • One execution model to reason about, document, and test.
  • No user-visible feature is lost that was reachable by default: the flag shipped off, and the pause prompt only appeared when the verifier itself malfunctioned.
  • Existing config files that still contain strictPlanVerification load fine and ignore it — covered by a test.
  • Saved plan checkpoints from before this change still resume.

Risk

Rollback is now a revert rather than a flag flip. That is the trade-off being accepted deliberately: the escape hatch is what this PR removes. Mitigating it is the point of promoting the two mechanical guards above, so the default path keeps the checks that were actually load-bearing.

Testing

Full suite green: 671 passed, 0 failed, on net8.0 and net10.0.

Test changes track the deletion rather than working around it:

  • PlanVerificationRecoveryTests → PlanStepRecoveryTests, rewritten around the four failure modes above.
  • StraightforwardPlanTests folded in; its config assertions replaced by one proving a config carrying the retired flag still loads.
  • PlanQualityTests → PlanRepositoryContextTests, which is all it covered once the review tests went.

Docs

docs/PlannerExecution.md deleted (it existed to compare the two modes). docs/TaskPlanner.md gains an explicit What can fail a step section, which the two-mode split had left implicit.

…e only mode

The second-model verifier gated every plan step behind an extra call that was
slow, expensive, and a frequent source of false blocks. Straightforward
execution shipped as the default; keeping the strict path as an opt-in kept two
execution modes, two sets of failure semantics, and a pause/resume UI alive for
a flag nobody was expected to flip.

Deletes the verifier, the proposal-review pass, the read-only evidence follow-up
and its middleware sandbox, the strictPlanVerification config key, and the
verification-unavailable pause path through the workflow runner and both hosts.

Stale checks now fail a step directly. That signal is read off tool ordering
rather than the model's own account of itself, so it survives the verifier's
removal as a real guard rather than becoming decorative. A step that captured no
tool evidence at all also cannot pass on its claim alone.

Configs that still carry strictPlanVerification load and ignore it.
@DevMando
DevMando merged commit cdb18ee into main Sep 6, 2026
@DevMando
DevMando deleted the codex/remove-strict-plan-verification branch September 6, 2026 04:11
DevMando added a commit that referenced this pull request Sep 11, 2026
A second pass over the 0.15.0 section against the merged pull requests found
three more gaps, all in the planner.

The biggest is that per-step verification by a second model was removed: #88
moved completion onto the executor's own evidence with acceptance criteria per
step and a closing test-and-repair phase, and #89 deleted the strict path
instead of leaving it behind a flag. That halves the model calls in a plan run
and changes how a step is judged done, and the changelog said neither.

#87's fixes were also unrecorded: the stall when a host stops reading progress,
and retries reusing gathered evidence rather than rerunning implementation.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant