Make dev mode robust: daemon output, Lambda log reads and levels, no dev alarms, MCP take-back, stop/start - #343

This fixes findings 3, 4, 5, 7, 9 and 10 of the third dev-mode dogfood run (r2p) and the misleading dev network delete message.

Daemon output (finding 4). The dev agent daemon wrote to pipes whose reader, the process that started it, had exited. A Python rebuild pipes Docker's output into the daemon's stdout, so it failed with EPIPE. The daemon now gets two files in the user-only token directory as stdout and stderr; the starting process tails them until AGENT_READY. After ready, the daemon drops its own console output (the agent log has it), so the files stay small, while child processes still write to the files. A rebuild that fails with something other than a Stacktape error (CliError), or with a failed AWS request, is tried once more. A new process test runs a stand-in daemon that pipes a child's output into its stdout after ready; it records EPIPE on the old code and passes on the new.

Log levels (finding 3). A new parser (utils/lambda-log-lines.ts) reads Node and .NET tab lines, Python's logging format and unhandled errors ([ERROR] ..., tracebacks, LAMBDA_WARNING: Unhandled exception), JVM exceptions, the JSON log format, and Status: error|timeout on REPORT and INIT_REPORT. The table test uses real lines I captured from throwaway nodejs22.x, python3.12, java21 and dotnet8 functions. Java's stack traces contain tabs and were misread as Node lines before.

Log polling (finding 5). One scheduler per process reads every function's log group: a group with new events about every second, a quiet one every 2 s, never more than 2 requests per second in total, with backoff and jitter on throttling. Throttling is reported once as a warning, never among the app's error logs. Each read is one FilterLogEvents over the whole group; the old printer followed only the first log stream, so it missed concurrent execution environments. stacktape logs follow mode uses the same scheduler and no longer gives up on throttling. I did not use StartLiveTail: it costs $0.01 per session minute after 1,800 free minutes a month, which a dev session left running all day turns into real money.

Dev stacks (finding 7). Dev stacks no longer get alarms, their EventBridge notification rules or alarm-triggered function events. A dev schedule rule names its function by ARN instead of GetAtt, so CloudFormation creates it while it creates the function. EventBridge accepts a target that does not exist yet (probed). Production stacks are unchanged. The dev fingerprint version moves to 6, so each existing dev stage updates once and loses its alarms.

MCP (finding 9). A replaced session's take-over answer no longer shadows a session this server started later in the same directory, and args.stage picks among one directory's sessions. A newer session found by discovery still does not override the answer (the existing test for a holder in the same directory still passes).

Stop and start (finding 10). DevBoxConnection.stop closes the relay presence and waits up to 5 s until the relay no longer lists the session before it ends the SSM session. A start replaces, instead of refusing, a holder from the same user and directory whose process has exited; the relay session ID now starts with the process ID. MCP stop waits until the agent process has exited.

dev network delete now says it can take up to about 20 minutes when dev functions were just deleted, reports progress once a minute, and waits up to 45 minutes instead of 20.

Live run in us-west-2 (stacktape-e2e, STACKTAPE_DEV_NETWORK_SUFFIX=r2r, copy of the r2p app). Cold start with network creation took 296 s; the dev stack has no alarms and its schedule rule ran in parallel with the functions. A new stage on the existing network took 134 s (r2p: 203 s), with a dev stack update of 73 s (r2p: 167 s). Three Python rebuilds and a rebuild_all after the MCP parent had exited returned OK with no EPIPE. A Python edit right after "ready" under --watch was live in 8.6 s. A planted Python error came back from MCP logs {level: "error"} as two error entries (r2p: none). Account-wide FilterLogEvents calls for three functions went from about 360/min (old code, computed) to about 65–117/min (measured with the AWS/Usage metric). Floods of 25 and 100 requests per second (298 and 1,750 throttled) produced no warning or error lines in the session, and logs kept arriving. Across two MCP servers, the r2p take-over sequence now reaches the newest session without an ID. Stop followed by an immediate start succeeded (stop 1.3 s, start 19.3 s). The network deletion took 92 s. Everything I created was deleted and verified.

Checks: test:local-dev 210/210, test:mcp-docs 81/81, test:src 3,058 pass and 0 fail, test:characterization 371/371, test:synthesis-families 32/32, test:operations:rules 65/65, test:tools 85/85, CLI typecheck, oxlint and oxfmt on the touched files, and generate:check after regenerating the LLM docs for the two doc edits.

What remains uncertain:

• Logs still arrive 9–18 s after a request. CloudWatch itself made the lines visible after 11–14 s in us-west-2, so polling adds at most a few seconds.
• I could not make the throttling warning appear live: the SDK's own retries absorbed even the 100/s flood. The warning path is covered by unit tests only.
• An alarm-only config change still counts as a config change and needs a dev restart, although the dev stack no longer contains alarms.
• The daemon output files stay in ~/.stacktape/dev-agent-tokens/ after the agent stops; the next agent on the same port overwrites them.