Skip to content

Bound the stacktape logs backlog, dedupe by time, cap dev daemon output [skip ci] - #346

Merged
matuscongrady merged 3 commits into
mainfrom
fix/logs-followups
Oct 27, 2026
Merged

matuscongrady merged 3 commits into
mainfrom
fix/logs-followups

Conversation

@matuscongrady

@matuscongrady matuscongrady commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

This fixes the seven findings the #343 review left open, and the two P1s from the review of this PR.

stacktape logs in a terminal read the whole --since window (an hour by default), every page, before it followed new lines. Now it shows the newest --limit lines (100 by default), which is what the one-time read and the docs promise, and then follows. The search starts with the last minute and reads each older window up to ten times wider. A window that holds more events than needed is narrowed to its newest part, and a window of one second or less is read whole and its newest events kept. Every poll sends one FilterLogEvents request, and further pages come on later polls within the 2 req/s budget. Following now also applies --filter, which it ignored before. The docs page describes follow mode, and the generated LLM docs corpus is regenerated for that paragraph.

The first version of this PR started the search with the whole window. With --filter, CloudWatch can return empty pages while it scans, so the 12-request budget could run out on the oldest part, and the recent matches were never shown (review P1). Starting from the last minute shows recent matches first. When the budget runs out, the backlog shows only the windows it read to the end and prints which lines it did not read, suggesting --query to search the whole window. Lines the backlog read but cut by --limit now count as handled, so following no longer prints one that has the same millisecond as the oldest line shown (review P2).

The dedupe set was trimmed to the last 5,000 event IDs, so a group writing more than 500 lines a second printed duplicates. It now keeps an ID only while a later read can still return that event (the 10 s look-back).

The dev daemon used to mute process.stdout.write after it reported ready, but Bun's console.* bypasses that. Now nothing is muted. The output files are opened for appending, and the daemon keeps each below 5 MB, moving the previous part to <file>.1. Append mode lets the daemon and its children (Docker builds) keep writing from the start of an emptied file, without a gap of zero bytes. The file is emptied through a new handle opened by its path: Windows refuses to truncate through a handle opened for appending (review P1). I reproduced that with Windows node.exe 26.3.0 and bun.exe 1.3.14 from WSL: EPERM, and 134 to 138 KB written against a 20 KB cap. Truncating by path kept the file at about the cap (at most 25 KB) with no zero bytes, on Windows with both runtimes and on Linux. The daemon gets the file paths in an environment variable instead of deriving them from its port.

Smaller fixes:

  • A failed rebuild is not retried when a newer build of the workload is already queued, and the retry waits 3 s, for example for Lambda's "function is updating" conflict.
  • MCP: only a session of the same stage clears the remembered DEV_STAGE_TAKEN_OVER answer of a replaced session.
  • Log levels: numeric pino/bunyan levels (50 error, 40 warn) are read, and a plain Error: ... line without a stack is info. The .NET start-up failure and errors with a stack trace are still errors.
  • The stale-holder check before a dev session starts compares a machine ID carried in the relay session ID: a hash of the hostname and the installation's system ID. Before, it relied only on the hostname in the label, so it could run kill(pid, 0) for a process on another machine with the same hostname. Holders from older CLIs have no machine ID and are treated as live.

Checks on the merged head: test:src 3,094 pass / 0 fail, test:local-dev 213 / 0, test:tools 85 / 0, CLI typecheck, pnpm lint and generate:check. In the first round I also ran test:operations:rules, test:operations:terminal, test:cli-smoke, test:llm-docs and test:mcp-docs, all passing. Each new test fails against the code it fixes. The three tests added in this round fail on the reviewed head 9682093. The in-memory FilterLogEvents fake now returns a token after the last page, as CloudWatch does, and can return empty scanning pages.

Live checks in us-west-2 (stacktape-e2e) with the real printer against throwaway log groups, all deleted and the deletion verified:

  • Unfiltered: a group with 30,000 lines over 10 minutes plus a burst 40 minutes earlier. The backlog showed exactly the newest 100 lines in 10 requests. Following showed new lines and a line that arrived 3 s late, each once.
  • Filtered (ERROR): the same 30,000 non-matching lines, 5 matches 50 minutes ago and 10 in the last minute. The backlog showed all 15 matches in order in 5 requests (2.0 s), starting with the last minute. Following printed the new matching line and skipped the non-matching one.

Uncertain:

  • The repository's daemon test cannot run under Windows bun.exe from WSL: Windows cannot follow pnpm's Linux symlinks in node_modules over \\wsl.localhost. The Windows evidence is a standalone probe that makes the same calls as rotateDaemonOutputFile, with an opener, a daemon and a grandchild writing through the inherited descriptor.
  • In the live filtered check, CloudWatch returned no empty scanning pages, so the budget-exhausted path is covered only by unit tests.
  • The narrowing assumes FilterLogEvents returns events oldest first within one query, across streams. The live runs agree, but the AWS docs do not state it.
  • Following a group with new events costs one extra empty-page request per poll, because of the trailing token. The old code made the same request, just in the same burst.

matuscongrady and others added 3 commits October 27, 2026 13:16
…tput, and fix #343 review leftovers [skip ci]

- `stacktape logs` in a terminal shows the newest `--limit` lines of the window first, found one request at a
  time from the newest window backwards within the 2 req/s budget, then follows; `--filter` applies to both.
  Every poll is one FilterLogEvents request; further pages come on the next polls.
- Shown event IDs are kept only as long as a later read can return them (the 10 s look-back), not the last 5,000.
- The dev daemon's output files are opened for appending and kept below 5 MB each (previous part in `<file>.1`)
  instead of muting process.stdout.write, which Bun's console.* bypassed.
- A failed rebuild is not retried when a newer build of the workload waits, and the retry waits 3 s.
- MCP: only a session of the same stage clears the remembered DEV_STAGE_TAKEN_OVER answer.
- Log levels: numeric pino/bunyan levels; a plain "Error: ..." line without a stack is info.
- The stale-holder check compares a machine ID in the relay session, not only the hostname in the label.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
…ly, and rotate daemon output on Windows [skip ci]

- The `stacktape logs` backlog starts with the last minute and widens older windows tenfold, so recent matches of
  a filter are not stuck behind CloudWatch's empty scanning pages for the oldest part of the window. When its
  12 requests run out, it shows only the fully read recent windows and prints which lines it did not read.
- Lines the backlog read but cut by `--limit` count as handled, so following does not print one with the time of
  the oldest line shown.
- Daemon output files are emptied through a new handle opened by path: Windows refuses to truncate a handle opened
  for appending (EPERM), so the files grew without bound there.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
# Conflicts:
#	apps/cli/@generated/llm-docs/lexical-index.json
@matuscongrady
matuscongrady merged commit a60909c into main Oct 27, 2026
@matuscongrady
matuscongrady deleted the fix/logs-followups branch October 27, 2026 12:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant