agents: first-class DeepSeek Harness (dsh) for deepseek models - #79
agents: first-class DeepSeek Harness (dsh) for deepseek models#79L4Ph wants to merge 14 commits into
Conversation
Headless dsh --profile headless runner with a per-run DSH_HOME and a generated cordis.yml overlay: agent-default-model + llm-deepseek (direct) or llm-pi-ai OpenRouter custom provider (proxy), mcp-client via insert: (streamable-http to the pullfrog MCP server, 660s tool timeout, fail-on-startup-error), plain-JSONL sessions, effort mapping low to off. Security: every native tool is disabled in the overlay (bash/pwsh, fs, web/search, skills, subagent/workflow/ralph fan-out) — the same posture as the codex harness for seams dsh cannot gate itself. All exec and file I/O flows through the sandboxed pullfrog MCP tools; workspace-write + approval ask fails closed headless. Ops: activity watchdog on session-JSONL growth (30min, vs the shared 15min budget), post-run gate retries respawn fresh headless processes, usage reported tokens-only parsed from the session JSONL.
…patibility gate auto: deepseek models (direct and OpenRouter routes, detected through the provider registry's dshCapable flag with the openrouter/ prefix unwrapped) route to dsh when RepoSettings.dshEnabled (the server-side kill-switch verdict) is set; everything else keeps its current harness. explicit: RepoSettings.agent (auto|claude|codex|opencode|dsh) is validated against the resolved model via assertAgentModelCompatible — incompatible pairs hard-fail before the agent starts. PULLFROG_AGENT stays the ungated top-priority escape hatch. main.ts sizes the outer activity watchdog per harness (30min for dsh).
The dsh harness has no event stream to watch (headless emits nothing until exit), so its activity clock is fed from MCP tool calls plus its own session-JSONL watcher.
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 3 potential issues.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Want higher recall? High effort reviews run extra passes and find more bugs. A team admin can switch effort levels in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 4599c77. Configure here.
| - **verify the finding yourself** against the actual code before deciding whether to apply — every comment (human or agent) is a hypothesis, not a directive. agent reviewers especially are fallible. | ||
| - you are searching for a solution that is **complete, minimal, and elegant** — you may need to think hard to find it. do not over-engineer, do not be over-defensive, **do not write AI slop**. reviewers bias toward *recommending additions*, and that bias has a recognizable slop texture: defensive checks for impossible cases, extra abstractions used once, comments restating obvious code, tests asserting tautologies, "just-in-case" guards, error handlers for cases the type system already rules out. reject those. evaluate whether applying the finding would leave the code more **sound, correct, AND elegant**; two-out-of-three is a signal to look harder for a fix that gets all three. if a request would add bloat — ceremony without commensurate correctness benefit — push back in your reply rather than mechanically applying it. | ||
| - if the request stands, make the code change using your native tools; otherwise reply explaining why | ||
| - if the request stands, make the code change via the pullfrog MCP tools (\`${t("shell")}\` etc. — this harness has no native tools); otherwise reply explaining why |
There was a problem hiding this comment.
AddressReviews misleads all agents
Medium Severity
The AddressReviews checklist now tells every harness it has no native tools and must use pullfrog MCP. That guidance was only meant for dsh; Build mode correctly gates on agentId === "dsh", but this line does not, so Claude/Codex/OpenCode get incorrect instructions.
Reviewed by Cursor Bugbot for commit 4599c77. Configure here.
![pullfrog[bot]](https://avatars.githubusercontent.com/in/1768019?s=60&v=4){kind=small}
There was a problem hiding this comment.
Caution
Two routing defects mean the feature does not work as described once the kill switch flips: every direct-auth dsh run dies with a misleading Anthropic-key error before the agent starts, and the OpenRouter DeepSeek Flash route never reaches dsh at all.
Reviewed changes
Read the full diff across all 15 files, then verified the dsh CLI and plugin-composition contract against the published @deepseek-ai/[email protected] tarball and its dsh-base / dsh-headless bundles rather than trusting the PR description of them.
- New
dshharness —agents/dsh.tsspawnsdsh --profile headlesswith a per-runDSH_HOME, a generated cordis--patchoverlay, and the pullfrog MCP server over streamable-http. - Capability metadata —
models.tsaddsdshCapable,realProvider(),isDshCompatibleModel(),dshProviderForModel(); deepseek is marked dsh-capable. - Routing —
utils/agent.tsgains an explicit-agentbranch,assertAgentModelCompatible(),hasDeepSeekAuth(), and adshEnabledkill switch (utils/runContext.ts, defaultfalse). - Prompt + activity —
modes.tsadds dsh-aware Build guidance;mcp/shared.tsmarks every MCP tool call as activity. - Tests + docs — 25 unit tests,
test/providers.tsflips the deepseek flagship to dsh, README documents harness selection.
Confirmed correct against the real package, so no action needed there: the CLI contract (--profile headless, positional prompt, repeatable --patch, $DSH_HOME), the insert:-vs-- id: distinction, and the claim that --patch is the highest-precedence layer so repo-local config cannot re-enable a disabled plugin. pnpm typecheck is clean and all 25 new tests pass.
⚠️ Editing agents/dsh.ts will not trigger the crossagent security suites
The six crossagent suites that pin the no-native-tools invariant — nobash, tokenExfil, fsExfil, gitNativeWrite, restricted, mcpmerge — all declare coverage: ["action/agents/{claude,codex,opencode}.ts", ...]. dsh is absent from every one of those globs, so a future PR touching only agents/dsh.ts skips the entire security suite on a harness whose whole safety story is a deny-list.
This is easy to miss because test/matrix.ts's loadAgents() regex-parses agents/index.ts and does auto-add dsh to the fan-out set — so FULL=1 and main pushes execute these tests against dsh today. The gap is the change-trigger, not the agent list.
Technical details
# Add dsh to crossagent coverage globs
## Affected sites
- `test/crossagent/nobash.ts:48` — `action/agents/{claude,codex,opencode}.ts`
- `test/crossagent/tokenExfil.ts:71` — same
- `test/crossagent/fsExfil.ts:189` — same
- `test/crossagent/gitNativeWrite.ts:105` — same
- `test/crossagent/restricted.ts:58` — same
- `test/crossagent/mcpmerge.ts:47` — same
## Required outcome
- A PR that modifies only `agents/dsh.ts` must schedule the crossagent security suites, not skip them.
## Suggested approach (optional)
- Extend each glob to `action/agents/{claude,codex,opencode,dsh}.ts`.
- Consider deriving the brace list from `loadAgents()` so the two never drift again — the matrix builder already knows the full agent set.ℹ️ providers-live will exercise opencode while labelling the run dsh
test/providers.ts flips the deepseek flagship to agent: "dsh", but dshEnabled defaults to false (utils/runContext.ts:142) and nothing in this repo sets it. Until the server ships the kill switch, the deepseek providers-live job resolves to opencode while the matrix reports it as dsh — a green run that never touched the new harness.
The same entry also drops action/agents/opencodePlugin.ts from deepseek's coverage globs, which is still the code path that actually executes on that job today.
Technical details
# Reconcile the providers-live deepseek entry with the kill switch
## Affected sites
- `test/providers.ts` — deepseek entry: `agent: "dsh"`, `coverage: ["action/models.ts", "action/agents/dsh.ts"]`
- `utils/runContext.ts:142` — `dshEnabled: false`
## Required outcome
- The deepseek `providers-live` job must either genuinely run dsh, or not claim to.
- Deepseek coverage must still include the harness that actually runs.
## Open questions for the human
- Should the live job force `dshEnabled` (e.g. via `PULLFROG_AGENT=dsh`, which the README documents as the ungated escape hatch) so it exercises the real harness pre-rollout?
- Or should the entry stay on opencode until the server enables the switch, and flip in a follow-up?ℹ️ The rollout leaves an undocumented dual track
Both agent and dshEnabled are server-delivered and nothing in this repo sets either, so the feature is entirely dark on merge — which is a good safety property, but it also means none of this code is exercised in production until a change in another repo lands. The PR doesn't state whether the opencode deepseek path is a permanent fallback or gets deleted once dsh is proven, nor what happens to an in-flight run if the switch flips mid-deploy.
Worth pinning down now, while the dual track is fresh, rather than discovering in six months that both paths are load-bearing.
Technical details
# Clarify dsh rollout and opencode deprecation
## Open questions for the human
- Is opencode-for-deepseek a permanent fallback, or is there a deletion milestone once dsh is proven?
- What is the intended sequencing between this merge and the server-side `isDshAgentEnabled()` change?
- If `dshEnabled` flips off mid-run (retry loop, post-run gate), does an already-spawned dsh process keep running, or should the harness re-check?ℹ️ Nitpicks
agents/dsh.ts:97—OPENROUTER_PROVIDER_IDis declared but never referenced;dshProviderForModel()inmodels.tsreturns the literal"openrouter"instead. Either use the constant or drop it.README.mddocuments dsh routing in the present tense, butdshEnableddefaults tofalse— the docs describe a state that isn't reachable yet on merge.agents/dsh.ts:473-478spawns with{ ...process.env, ... }. The headless bundle mountscode-runtimeand readstools.modefromprocess.env.DSH_TOOLS_MODE, so an inheritedDSH_TOOLS_MODEwould flip the run into Code Mode. PinningDSH_TOOLS_MODE: ""in the spawn env costs one line and removes the ambiguity. (Compareagents/codex.ts:809, which appliesfilterEnv()as defense-in-depth.)package.jsonlists@deepseek-ai/dshbefore@anthropic-ai/claude-code, breaking alphabetical order in the dev-dependency block.
Claude Opus (free via Pullfrog for OSS) ｜ 𝕏
| * - a provider that accepts the request but never streams trips dsh's own | ||
| * streamIdleTimeoutMs and ends the turn with an error (exit 1), so a | ||
| * first-event watchdog is redundant; missing keys are caught before the | ||
| * agent starts by validateAgentApiKey in main.ts. |
There was a problem hiding this comment.
This comment claims validateAgentApiKey gates the dsh run, but that function has no dsh branch. main.ts calls it whenever !payload.proxyModel, and for agent.name === "dsh" control falls through to hasSingleProviderAuth("dsh") (utils/apiKeys.ts:348-353), whose default arm checks ANTHROPIC_API_KEY / CLAUDE_CODE_OAUTH_TOKEN.
A DeepSeek-only user therefore gets a misleading "missing Anthropic key" error and the harness never starts — every direct-auth dsh run fails.
Technical details
# validateAgentApiKey has no dsh branch
## Affected sites
- `utils/apiKeys.ts:348-353` — `hasSingleProviderAuth()` only special-cases `codex`; every other agent falls through to the Anthropic env vars.
- `utils/apiKeys.ts:445` and `utils/apiKeys.ts:481` — both call sites `return` on that check, else throw `buildKeyError`.
- `agents/dsh.ts:40` — comment asserts the opposite.
## Required outcome
- A non-proxy run with `agent.name === "dsh"` and a valid DeepSeek credential must pass validation.
- A dsh run with no DeepSeek credential must still fail, with an error naming DeepSeek rather than Anthropic.
## Suggested approach (optional)
- Add a `dsh` arm to `hasSingleProviderAuth()` checking the same env vars `hasDeepSeekAuth()` uses in `utils/agent.ts`, so the two cannot drift.
- The proxy path is unaffected — `validateAgentApiKey` is skipped when `payload.proxyModel` is set.
## Open questions for the human
- Was this path covered by the local e2e run described in the PR body, or did that run go through the proxy route?| * DeepSeek; openrouter/anthropic/... by Anthropic. needed because the | ||
| * OpenRouter route must be routed by the INNER provider's capability, not by | ||
| * the transport vendor. */ | ||
| export function realProvider(specifier: string): string { |
There was a problem hiding this comment.
realProvider splits on / and returns the raw second segment, so openrouter/~deepseek/deepseek-v4-flash-latest yields "~deepseek" — not a key in providers, so isDshCompatibleModel() returns false and Flash silently routes away from dsh. The ~ rolling-pointer form is the real OpenRouter specifier for Flash (models.ts:373, :794, :796) and nothing in the repo strips it.
Pro (openrouter/deepseek/deepseek-v4-pro-0813) and v3.2 unwrap fine, so this fails on exactly one model — the default efficient pick — with no error, contradicting the PR's "direct API and OpenRouter route" claim.
Technical details
# realProvider does not strip the OpenRouter rolling-pointer prefix
## Affected sites
- `models.ts:853-860` — `realProvider()` returns the raw inner segment.
- `models.ts:373` — `openRouterResolve: "openrouter/~deepseek/deepseek-v4-flash-latest"`.
- `models.ts:794`, `models.ts:796` — same specifier on the `openrouter/deepseek-flash` alias.
- `models.ts:875-878` — `dshProviderForModel()` inherits the miss.
## Required outcome
- `isDshCompatibleModel("openrouter/~deepseek/deepseek-v4-flash-latest")` must be `true`.
- Non-tilde specifiers must keep resolving exactly as they do today.
## Suggested approach (optional)
- Strip a leading `~` from the inner segment before the registry lookup. The `~` is an OpenRouter routing sigil, not part of the provider identity — the same normalization applies to `~anthropic` and `~google`, which use the form at `models.ts:155`, `:664`, `:761`.
- Add a unit test covering the tilde form; the existing dsh routing tests only use non-tilde specifiers, which is why this passes CI.| await exited; | ||
| clearInterval(watchdog); | ||
|
|
||
| const usage = parseSessionUsage(params.sessionsRoot); |
There was a problem hiding this comment.
parseSessionUsage walks the whole sessionsRoot, but each post-run retry respawns a fresh headless process into that same root (headless has no --resume, as the PR notes). So attempt 2's parse returns attempt 1 + attempt 2 combined.
runPostRunRetryLoop then sums per-attempt usage via mergeAgentUsage on top of initialUsage, so a single retry reports 2×T1 + T2 tokens. That flows straight into server-side cost analytics.
Technical details
# Session usage is double-counted across post-run retries
## Affected sites
- `agents/dsh.ts:407` — `parseSessionUsage(params.sessionsRoot)` aggregates every session under the root.
- `agents/dsh.ts:483-491` — `runPostRunRetryLoop({ initialUsage: initial.usage, resume: async (c) => runOnce(c.prompt) })`.
- `agents/postRun.ts:545` and `agents/postRun.ts:494` — `aggregatedUsage = mergeAgentUsage(aggregatedUsage, result.usage)`.
- `agents/shared.ts:208-228` — `mergeAgentUsage` sums input/output/cache buckets.
## Required outcome
- Total reported usage across N attempts must equal the sum of each attempt's real usage, counted once.
## Suggested approach (optional)
- Either give each attempt its own sessions subdirectory (the `--patch` overlay already sets `session-persistence-jsonl.root` per run, so this is a path change), or have `parseSessionUsage` accept a set of already-counted session files and skip them.
- The per-attempt option composes better with `mergeAgentUsage`, which is built to sum disjoint attempts.| - **verify the finding yourself** against the actual code before deciding whether to apply — every comment (human or agent) is a hypothesis, not a directive. agent reviewers especially are fallible. | ||
| - you are searching for a solution that is **complete, minimal, and elegant** — you may need to think hard to find it. do not over-engineer, do not be over-defensive, **do not write AI slop**. reviewers bias toward *recommending additions*, and that bias has a recognizable slop texture: defensive checks for impossible cases, extra abstractions used once, comments restating obvious code, tests asserting tautologies, "just-in-case" guards, error handlers for cases the type system already rules out. reject those. evaluate whether applying the finding would leave the code more **sound, correct, AND elegant**; two-out-of-three is a signal to look harder for a fix that gets all three. if a request would add bloat — ceremony without commensurate correctness benefit — push back in your reply rather than mechanically applying it. | ||
| - if the request stands, make the code change using your native tools; otherwise reply explaining why | ||
| - if the request stands, make the code change via the pullfrog MCP tools (\`${t("shell")}\` etc. — this harness has no native tools); otherwise reply explaining why |
There was a problem hiding this comment.
This AddressReviews step now tells every harness "this harness has no native tools", but that's only true for dsh — claude, codex, and opencode all have native tool access. The sibling Build-mode change in this same commit is correctly gated on agentId === "dsh"; this one wasn't.
The result is a prompt regression for the three existing harnesses on every AddressReviews run.
| - if the request stands, make the code change via the pullfrog MCP tools (\`${t("shell")}\` etc. — this harness has no native tools); otherwise reply explaining why | |
| - if the request stands, make the code change${agentId === "dsh" ? ` via the pullfrog MCP tools (\`${t("shell")}\` etc. — this harness has no native tools)` : ""}; otherwise reply explaining why |
| } else if (agentName === "codex") { | ||
| allowed = provider === "openai"; | ||
| } else if (agentName === "dsh") { | ||
| allowed = isDshCompatibleModel(model); |
There was a problem hiding this comment.
This gate resolves the provider with getModelProvider(model) (raw first segment) while the dsh path uses realProvider() (OpenRouter-unwrapped). On a Router run, getModelProvider("openrouter/anthropic/claude-opus-5") returns "openrouter", which matches no vendor agent — so an explicit agent: "claude" console pick now hard-fails on runs that worked before this PR.
The two functions exist precisely because raw prefixes lie about the serving provider; the compatibility gate should use the same unwrapping the routing does.
Technical details
# Compatibility gate and dsh routing disagree on what "provider" means
## Affected sites
- `utils/agent.ts:83-89` — `assertAgentModelCompatible()` uses `getModelProvider(model)`, then `isDshCompatibleModel(model)` for the dsh arm only.
- `utils/agent.ts:55` — `compatibleAgentsFor()` has the same split.
- `models.ts:853-860` — `realProvider()` is the unwrapping helper this PR adds for exactly this reason.
## Required outcome
- An explicit `agent` selection must be judged against the provider that actually serves the model, on both direct and OpenRouter routes.
- `agent: "claude"` + `openrouter/anthropic/...` must remain allowed.
## Suggested approach (optional)
- Use `realProvider()` for the vendor-agent arms too, so all branches agree.
- Note this interacts with the tilde bug flagged on `models.ts:853` — fix that first, or `openrouter/~anthropic/claude-opus-latest` will fail the gate for the same reason.
## Open questions for the human
- Is the explicit-`agent` setting reachable on Router runs today, or does the console only offer it for BYOK? That determines whether this is live breakage or latent.openrouter/~deepseek/deepseek-v4-flash-latest resolved its inner segment as ~deepseek, which is not a provider key — so the default efficient Flash pick silently missed the dsh route on proxy runs. the ~ is an OpenRouter routing sigil, not part of the provider identity.
assertAgentModelCompatible/compatibleAgentsFor used getModelProvider (raw first segment) while dsh routing uses realProvider (OpenRouter-unwrapped). on Router runs, getModelProvider(openrouter/anthropic/...) returns openrouter, so an explicit claude pick hard-failed on runs that worked before. both now use realProvider, matching the routing.
hasSingleProviderAuth had no dsh arm, so a direct-auth dsh run fell through to the Anthropic env vars and a DeepSeek-only user got a misleading missing ANTHROPIC key error before the harness ever started. add the DEEPSEEK_API_KEY check; buildKeyError already names the model's own env var.
…erlay - idle watchdog subtracted the filesystem's epoch mtime from a monotonic performance.now, so idleMs went ~ -1.8e12 and the 30min kill never fired. track lastProgressAt on one monotonic clock. - each post-run retry respawns a fresh headless process; dsh persists the boot patch into DSH_HOME/profiles, so a second --patch on the same home double-applied the mcp-client insert and failed the boot. give every attempt its own DSH_HOME, which also keeps usage parsing disjoint (the mergeAgentUsage sum was double-counting earlier attempts). - correct the disable list id (tool-subagent-control/list-agents -> the mounted tool-subagent-list-agents), add tool-jobs, and fail CLOSED when a disable id matches nothing on stderr instead of warning and skipping.
1 participant
Summary
Integrates DeepSeek Harness (
@deepseek-ai/dsh0.1.0-rc.6) as a fourth agent harness. DeepSeek V4 Flash/Pro models (direct API and OpenRouter route) run ondsh --profile headlesswhen the server-side kill switch is enabled; all other providers keep their current harnesses.Decomposed into 8 reviewable commits (models capability metadata → dep pin → harness → routing/gate → modes → mcp activity → tests → docs).
Selection logic (
utils/agent.ts)realProvider()unwraps theopenrouter/prefix, and models whose provider is markeddshCapablein the models.ts registry (today: deepseek) route to dsh. Gated byRepoSettings.dshEnabled(server-sideisDshAgentEnabled()kill-switch verdict — safe rollout: default false until the server enables it).settings.agent(console radio, server-delivered):auto|claude|codex|opencode|dsh— compatibility-gated viaassertAgentModelCompatible(); incompatible pairs hard-fail before the agent starts (e.g.claude× deepseek). Explicitdshpicks also require the kill switch.PULLFROG_AGENTenv stays the ungated top-priority escape hatch.Harness (
agents/dsh.ts)dsh --profile headlessvia the CLI JS directly, per-runDSH_HOMEtmpdir, generated cordis.yml overlay:agent-default-model+llm-deepseek(direct) orllm-pi-aiOpenRouter custom provider (proxy),mcp-client(insert:form — a plain- id:entry only patches existing entries, measured), streamable-http → pullfrog MCP with 660s tool timeout + fail-on-startup-error, plain-JSONL sessions, effort mapping low→off / high→high / max→max.DSH_PERMISSION_MODE=workspace-write+ approval ask fails closed headless.--resume), usage reported tokens-only parsed from the session JSONL (server-side analytics computes cost; a usage-report plugin is the documented fallback if analytics turns outcostUsd-dependent).mcp__pullfrog__<tool>(claude-same naming);AgentIdextended.Testing
test/providers.ts: deepseek flagship now uses the dsh harness (drivesproviders-livereal-key smoke) with dsh-specific coverage globs.mcp__pullfrog__*), disabled-tool enforcement, session usage parse.Server/console follow-ups (outside this repo)
isDshAgentEnabled()kill switch +RepoSettings.agent/dshEnableddelivery + console radio.agents/dsh.ts).providers-livedeepseek) and crossagent smoke wiring.Note
Cursor Bugbot is generating a summary for commit c5c6424. Configure here.