Skip to content

feat(codex): env-mcp subcommand of codex-app-gateway - #78

Merged
imryao merged 11 commits into
mainfrom
worktree-codex-env-mcp
May 11, 2026
Merged

imryao merged 11 commits into
mainfrom
worktree-codex-env-mcp

Conversation

@imryao

@imryao imryao commented May 10, 2026

Copy link
Copy Markdown
Member

Summary

First subsystem of the new codex-gateway-mcp-rewrite track. Adds codex-app-gateway env-mcp ..., a stdio MCP server that bridges MCP tools/call traffic from a spawned codex subprocess to a single executor's exec-server endpoint via WebSocket. This is Subsystem 4 of the new spec — the runtime/foundations/exec-gateway subsystems are tracked separately.

  • New binary cmd/codex-app-gateway/ with a multi-subcommand entry (env-mcp implemented; serve placeholder).
  • New package internal/codexappgateway/envmcp/:
    • bridge.go — JSON-RPC client over one WebSocket to /bridge/{exe_id} (race-tested concurrency).
    • translator.go — MCP shell tool call → exec-server process/start + polling process/read loop.
    • mcp_server.go — newline-delimited JSON-RPC stdio MCP server exposing one shell tool.
    • envmcp.go — top-level Run() wiring.
    • 22 tests (21 unit + 1 integration against real codex exec-server).
  • Dockerfile.codex-app-gateway — multi-stage build (golang:1.26-trixie → debian:trixie-slim).

Spec / Plan

  • Spec: docs/superpowers/specs/2026-05-10-codex-gateway-mcp-rewrite.md § Subsystem 4 (PoC-validated 2026-05-10).
  • Plan: docs/superpowers/plans/2026-05-10-codex-env-mcp.md — 7 tasks, all complete, executed via subagent-driven-development with per-task spec + code-quality reviews.

This branch supersedes the env-mcp slice that was previously planned inside the 2026-05-05 codex-app-gateway runtime plan (now marked OBSOLETE).

Test Plan

  • go test ./internal/codexappgateway/envmcp/ ./cmd/codex-app-gateway/ — 21/21 PASS
  • go test -race ./internal/codexappgateway/envmcp/ — clean
  • go test -tags integration ./internal/codexappgateway/envmcp/ -run TestRun_AgainstRealCodexExecServer — PASS against real codex 0.128.0
  • go vet ./cmd/codex-app-gateway/ ./internal/codexappgateway/envmcp/ — clean
  • Smoke: `codex-app-gateway env-mcp --help` → exit 0 with usage block; missing required flags → exit 2 with single-line error
  • Reviewer: confirm the cap-token blast radius (one HMAC token per executor per turn, env-passed not argv-passed) is correctly enforced end-to-end once the runtime plan lands.

Notes for the reviewer

  • Two protocol gotchas (codex exec-server closes `permessage-deflate` peers; all exec-server fields are camelCase) are documented in the spec PoC log and respected in code (`bridge.go` doc comment + every JSON tag in `types.go`).
  • The full 7-task TDD trace including two reviewer-caught fix cycles (CLI hygiene in Task 1, data race in Task 3) is preserved as 11 separate commits on the branch — squash or rebase at your discretion before merge.

imryao pushed a commit that referenced this pull request May 10, 2026
…ead codex app-server subprocess

Adds 2026-05-10-codex-app-gateway-subprocess.md, which replaces
Subsystem 2 of the MCP-rewrite spec with a design that spawns one
upstream `codex app-server --listen ws://...` subprocess per thread
and proxies ws frames between the user's TUI and the subprocess. Cuts
~1100 LOC of Go (no 17-RPC handler reimplementation, no Postgres
schema, no event mapper) at the cost of a per-thread subprocess
supervisor + S3 CODEX_HOME round-trip.

Validated against codex-cli 0.130.0:
- PoC #3: full initialize → thread/start → turn/start → ServerNotification
  stream end-to-end on a loopback ws.
- PoC #3b: two concurrent subprocesses with isolated CODEX_HOMEs run
  in independent threads with no sqlite contention.

Subsystems 1 (no fork patches), 3 (codex-exec-gateway), and 4 (env-mcp,
PR #78) are unchanged. Cross-references added to the MCP-rewrite spec
header.
imryao pushed a commit that referenced this pull request May 10, 2026
…tion plan

Marks the 2026-05-05 codex-app-gateway-and-exec-gateway design spec
SUPERSEDED by the 2026-05-10 MCP-rewrite spec (and now further refined
by the codex-app-gateway-subprocess spec). Marks the four 2026-05-05
plan files OBSOLETE with pointers to their successors.

Adds 2026-05-10-codex-env-mcp.md, the implementation plan executed
in PR #78 that delivered Subsystem 4 of the MCP-rewrite spec
(env-mcp subcommand of codex-app-gateway).
@imryao
imryao merged commit 11db84b into main May 11, 2026
6 checks passed
imryao added a commit that referenced this pull request May 11, 2026
… app-server (Subsystem 2)

Implements Subsystem 2 of the codex-gateway design as a thin auth-proxy + per-thread \`codex app-server\` subprocess manager + transparent ws frame proxy. Replaces the originally-planned 17-RPC handwritten Go RPC layer.

- New: \`internal/codexappgateway/{auth,codexhome,supervisor,proxy}/\` packages + \`server.go\` + \`s3_store.go\`.
- Modified: \`cmd/codex-app-gateway/main.go\` (wires the \`serve\` subcommand previously stubbed in #78).
- 63 tests + 1 integration test against real codex 0.130.0 — all PASS, race-clean.
- Spec: \`docs/superpowers/specs/2026-05-10-codex-app-gateway-subprocess.md\`
- Plan: \`docs/superpowers/plans/2026-05-11-codex-app-gateway-subprocess.md\`

**Final round of post-review fixes (after holistic 5-reviewer audit):**

- Runtime correctness: Touch-on-frame, race-loser unlock, stale-handle health-check on \`EnsureSubprocess\`, \`ChildHandle.Cmd\` → unexported.
- Hardening: \`os.Environ()\` inheritance for subprocess, \`filepath.Rel\` tmpdir guard, \`ConfigInput\` cross-field validation, \`*slog.Logger\` fields on Supervisor/IdleReaper/BridgeClient/MCPServer, S3 endpoint scheme validation.
imryao pushed a commit that referenced this pull request May 11, 2026
Near-verbatim refresh of the 2026-05-05 plan with three contextual
updates aligned to the 2026-05-10 MCP-rewrite spec:
- Spec reference now points to 2026-05-10-codex-gateway-mcp-rewrite.md § Subsystem 3
- /bridge/{exe_id} consumer is the env-mcp child binary (PR #78), not
  the spawned codex itself (wire protocol + auth model unchanged)
- Cap-token allow-list is single-exe-id per the 2026-05-10 refinement
  (verification logic unchanged; minting is stricter, happens in
  codex-app-gateway)

Old 2026-05-05 plan file's banner updated to point at this reissue.
imryao pushed a commit that referenced this pull request May 12, 2026
…uracy

Six interlinked fixes from the holistic PR self-review:
- C1: Config.Validate() rejects empty HMAC/internal secrets; NewServer now
  returns (*Server, error) and refuses zero-value config. Closes a path
  where empty CapTokenHMACSecret silently accepted forged tokens.
- C2: inbound + bridge ws now SetReadLimit(-1). nhooyr's 32KB default
  would silently close large process/read responses — codex exec-server
  streams them. (Same fix env-mcp PR #78 already had.)
- C3: RevokedSet.Add returns evictedLive bool; RevokeTurn handler logs
  at Warn when a still-live revocation is evicted at capacity. Closes a
  silent security regression where evicted-but-live tokens worked again.
- C4: bridge pump context now derived from r.Context() so graceful
  shutdown drains active sessions instead of leaking pump goroutines.
- I7: migrate() wraps each error with the migration name so operators
  can tell which migration failed.
- I8: corrects 3 wrong comments (bridge.go step-3 rationale, server.go
  nil-fields claim, forwarder.go shutdown semantics).

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
imryao added a commit that referenced this pull request May 12, 2026
…egistry (Subsystem 3)

Implements **Subsystem 3** of the codex-gateway design — completes the gateway service trio (env-mcp #78, codex-app-gateway #79, codex-exec-gateway here).

- New service \`cmd/codex-exec-gateway\` + \`internal/codexexecgateway/\`:
  - chi router + lifecycle, embedded Postgres migrations.
  - Inbound \`/codex-exec/{exe_id}\` ws acceptor (bcrypt tunnel-token verify, single-conn-per-exe eviction).
  - Bridge \`/bridge/{exe_id}\` ws acceptor (HMAC cap-token verify with single-exe-id allow-list, revocation check, per-exe-id bridge mutex via \`AcquireBridge\`/\`ReleaseBridge\` to prevent concurrent-pump data races).
  - In-memory \`ConnRegistry\` (concurrent-safe, identity-guarded \`Unregister\`).
  - Bounded \`RevokedSet\` (cap 10000, FIFO eviction, periodic exp pruning, \`Add\` returns \`evictedLive bool\` for security alerting).
  - HTTP endpoints: \`POST /api/codex-exec/register\`, \`POST/GET/DELETE /api/codex-exec/workspaces/{wid}/executors\`, internal \`GET /api/exec-gateway/connected\`, \`POST /api/exec-gateway/revoke-turn\` (shared-secret-protected).
- Cross-package factor-out: \`internal/codexappgateway/proxy/\` (from PR #79) moved to shared \`internal/wsbridge/\`, consumed by both gateways.
- Shared DTOs in \`internal/codexexecgateway/execmodel\` (eliminates parent↔handlers import-cycle adapters).

**Self-review** (5 specialized reviewers in parallel) flagged 4 Critical + 11 Important + 7 Minor; 3 fix batches landed (\`10fd4a1\`, \`f1c7f3e\`, \`1a52358\`):
- C1: \`Config.Validate()\` rejects empty HMAC; \`NewServer\` returns error on zero-value config.
- C2: inbound + bridge \`SetReadLimit(-1)\` (closes silent 32KB-frame truncation).
- C3: revoked-set eviction logs Warn when dropping a still-live revocation.
- C4: bridge pump derives ctx from \`r.Context()\` (graceful shutdown drains active sessions).
- I3: per-exe-id bridge mutex prevents concurrent-pump data race.
- Plus observability (auth failure logs, pump-error log levels, close-error logs), comment accuracy, dead-config removal, \`Store\` encapsulation tightening.

Final: 35 PASS / 15 SKIP (DB-gated) / 0 FAIL, \`-race\` clean, vet clean, build clean.

**Spec / Plan**
- Spec: \`docs/superpowers/specs/2026-05-10-codex-gateway-mcp-rewrite.md\` § Subsystem 3
- Plan: \`docs/superpowers/plans/2026-05-12-codex-exec-gateway.md\` (14 TDD tasks, all complete)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant