Skip to content

Improve realtime watch reliability and add scoped rerun - #816

Merged
epszaw merged 17 commits into
mainfrom
realtime-watch-affected-tests
Aug 20, 2026
Merged

epszaw merged 17 commits into
mainfrom
realtime-watch-affected-tests

Conversation

@todti

@todti todti commented Jul 23, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • Scoped rerun in allure run --watch (new): when a spec file changes, only that file is rerun instead of the entire suite — makes watch mode practical for large e2e suites where rerunning everything on every save was too slow.
  • Reliable shutdown for allure run --watch: Ctrl+C now reliably kills the full test process tree (SIGTERM, then SIGKILL if the process doesn't exit in time), instead of sometimes hanging or leaving orphaned processes behind.
  • Crash isolation for allure run --watch: if a single rerun fails unexpectedly (e.g. a spawn error), the watch session now logs it and keeps watching, instead of the whole process crashing.
  • Fixed EMFILE crashes in allure run --watch: source-file watching now skips node_modules, dotfiles/dot-directories, and build output by default, instead of exhausting the OS file-descriptor limit on large repos.
  • Dynamic result discovery in allure watch: allure-results directories that don't exist yet at startup (e.g. a package that hasn't run its tests yet) are now picked up automatically once they appear, instead of only ever watching the directories found at launch.
  • Interruptible shutdown for allure watch: Ctrl+C now interrupts an in-progress directory scan immediately instead of waiting for the whole scan to finish first.
  • New --new-only flag for allure watch: skips the existing backlog of results on disk at startup and only reacts to results written after the watch session started.
  • Fixed runaway terminal output in watch mode's progress display: the live failed/broken/passed/total line no longer duplicates or corrupts itself when re-rendered rapidly or when other plugins log output at the same time.
  • TestOps: stopped redundant uploads: global attachments and global errors were being re-uploaded on every single realtime update cycle; now each is uploaded only once.
  • TestOps: added a heads-up log message announcing newly found test results before they start uploading, for better visibility into what's happening during a watch session.

@github-actions

github-actions Bot commented Jul 23, 2026 •

Copy link
Copy Markdown

Allure Report Summary

Name Duration Stats New Flaky Retry Report
Allure 3 Report 42m 57s Passed tests 10037   Skipped tests 15   Unknown tests 24 75 0 34 View
My Dashboard 42m 57s Passed tests 10037   Skipped tests 15   Unknown tests 24 75 0 34 View

@github-actions

github-actions Bot commented Jul 23, 2026 •

Copy link
Copy Markdown

Allure perf metrics

Generated at: 2026-08-20T11:45:55.423Z

Phase Count Total Avg Min Max
restoreState.total 1 2935.5 ms 2935.5 ms 2935.5 ms 2935.5 ms
restoreState.dump 3 2935.0 ms 978.3 ms 906.9 ms 1084.9 ms
restoreState.attachments 3 2321.0 ms 773.7 ms 705.5 ms 853.3 ms
restoreState.storeRestore 3 199.1 ms 66.4 ms 62.1 ms 74.7 ms
generate.total 1 8494.5 ms 8494.5 ms 8494.5 ms 8494.5 ms
generate.plugins.done 1 7962.2 ms 7962.2 ms 7962.2 ms 7962.2 ms
publish.upload.total 1 165019.6 ms 165019.6 ms 165019.6 ms 165019.6 ms
summary.generate 2 11.5 ms 5.7 ms 5.0 ms 6.4 ms
generate.plugin.done.agent 1 0.2 ms 0.2 ms 0.2 ms 0.2 ms
generate.plugin.done.awesome 1 7029.3 ms 7029.3 ms 7029.3 ms 7029.3 ms
generate.plugin.done.dashboard 1 927.2 ms 927.2 ms 927.2 ms 927.2 ms
generate.plugin.done.log 1 4.6 ms 4.6 ms 4.6 ms 4.6 ms
generate.plugin.done.testops 1 0.3 ms 0.3 ms 0.3 ms 0.3 ms
publish.upload.plugin.awesome 1 132487.0 ms 132487.0 ms 132487.0 ms 132487.0 ms
publish.upload.plugin.dashboard 1 18800.8 ms 18800.8 ms 18800.8 ms 18800.8 ms

Artifacts: allure-perf-metrics

@todti
todti force-pushed the realtime-watch-affected-tests branch from 4ffb746 to 1fcad62 Compare July 23, 2026 16:22
@todti
todti requested review from epszaw and formaceft-93 and removed request for formaceft-93 July 29, 2026 09:29
@todti
todti force-pushed the realtime-watch-affected-tests branch from 6401b6d to 01cc99e Compare July 29, 2026 17:10
@todti
todti requested review from formaceft-93 and removed request for epszaw July 30, 2026 08:30
Comment thread packages/cli/src/commands/commons/run.ts Outdated
Comment thread packages/cli/src/commands/watch.ts
Comment thread packages/cli/src/commands/run.ts Outdated
Comment thread packages/directory-watcher/src/index.ts Outdated
Comment thread packages/directory-watcher/src/index.ts Outdated
Comment thread packages/cli/src/commands/watch.ts Outdated
Comment thread packages/cli/src/commands/run.ts Outdated
@formaceft-93
formaceft-93 self-requested a review July 30, 2026 12:02
@todti
todti requested a review from epszaw July 30, 2026 13:27
@todti
todti force-pushed the realtime-watch-affected-tests branch 2 times, most recently from 0e00f74 to 411010a Compare August 4, 2026 11:52
@todti
todti removed the request for review from epszaw August 5, 2026 12:56
@formaceft-93
formaceft-93 requested a review from epszaw August 13, 2026 09:32
todti added 8 commits August 19, 2026 13:28
allure run --watch: rerun only the changed spec file instead of the whole
suite; kill the full test process tree on Ctrl+C with SIGTERM/SIGKILL
escalation instead of hanging or leaving orphans; keep the watch session
alive when a run crashes (e.g. spawn errors) instead of taking down the
whole process; avoid EMFILE by skipping dotfiles/node_modules/build output
when watching source files for changes.

allure watch: discover allure-results directories dynamically instead of
only at startup, so directories that don't exist yet get picked up once
they appear; make shutdown interruptible mid-scan instead of waiting for a
full directory walk to finish; add --new-only to skip the existing result
backlog and react only to results written after the watch started.

plugin-progress: fix runaway terminal output from overlapping renders and
other plugins' log lines corrupting the in-place progress line.

plugin-testops: stop re-uploading the same global attachments/errors on
every realtime update cycle; announce newly found test results before
uploading them.
Add tests for directory-watcher's default node_modules/dotfile ignoring
and AbortSignal-based scan interruption, plugin-progress's render
mutex/throttling/foreign-write isolation, and plugin-testops's global
attachment/error upload deduplication. Shorten the multi-line comments
added earlier in the session down to single lines per the repo's
comment style.
yoctocolors wraps the bolded count in the middle of the "Found N new test
results" message, splitting it with escape codes. That's invisible locally
(no TTY, no colors), but CI runners enable color output, so the plain
stringContaining assertion never matched there.
isIgnoredByDefault was splitting on node:path's sep, which is a backslash
on Windows. chokidar normalizes paths to forward slashes regardless of
platform, so the split never found a segment boundary there and
node_modules/dot-directories went unfiltered. Split on either slash instead.
…watch

Classify TestOps upload failures into transient (5xx/429/timeout),
recoverable ("launch is closed"), or terminal (auth/validation/payload/
conflict) kinds, and retry only the retryable ones with exponential
backoff. Wired into all three upload call sites (test results, global
attachments, global errors) — previously a failed request just logged
and gave up with no retry at all.

Add a leaky-bucket pacer with independent per-window budgets for
requests, files, and bytes sent to TestOps, enabled by default with
sane defaults (20 req/s, 1000 files/s, 1 GiB/s) since the server-side
limits it approximates are real regardless of whether anyone configures
for them. Disable via a new uploadRateLimit: false plugin option, or
tune it via an explicit uploadRateLimit config.

Replace the ad-hoc SIGINT handler in run --watch, and add SIGTERM
handling (which it didn't have), with a small shared signals helper
(notifySignals/waitForAbort/boundedTerminationSignal) — conventional
exit codes (130/143), a graceful-shutdown deadline, and the same
kill-tree-with-escalation behavior as before. The `open` command's
temp-report cleanup now handles SIGTERM too.
allure run now watches source files and reruns just the changed spec by
default when it's not running in CI (checked via the same CI-provider
detection plugin-testops already uses, plus a CI=true fallback) — running
it locally feels like starting a watch session, matching how the command
is actually used day to day. In CI it still runs once and exits with a
pass/fail code, unchanged. --watch / --no-watch still override the
default explicitly either way.
Clearer than relying on clipanion's implicit --no-watch negation.
--watch and --once together is a usage error.
…only dropping newly discovered directories

terminationOf only listened for 'exit', so a failed spawn (e.g. bad command)
emitted an unhandled 'error' event and crashed the whole watch session
instead of being reported as a failed run.

allure watch --new-only was passing ignoreInitial to every dynamically
discovered allure-results directory, including ones that appear after watch
has already started — dropping their backlog even though it's new content,
not a pre-existing one to skip.
todti and others added 5 commits August 19, 2026 13:28
…tly instead of via a real spawn

On Windows, runProcess spawns through a shell by default, so a bad command
exits normally instead of emitting 'error' the way it does on POSIX with a
direct spawn — making the spawn-based regression test platform-dependent.
Emit the 'error' event on a fake process instead so the test is consistent
across platforms.
Defaulting to watch mode outside CI would silently change behavior for
every existing "allure run" workflow that isn't run in CI, since the
command is what actually executes the test suite. Back to plain --watch,
no CI auto-detection, no --once.
…ed directory's .gitignore

Blanket-ignoring anything starting with "." could silently drop test data
users keep in a dot-prefixed directory. Now only node_modules/dist/build/
out/coverage/.git are ignored unconditionally, plus whatever plain names
the watched directory's own .gitignore lists.

Also flips allure watch's --new-only to on by default (pass --no-new-only
to ingest the existing backlog instead), since starting a watch session and
getting flooded with every already-existing result is rarely what's wanted.
remove unnecessary logs from testops plugin (change log level) in realtime
simplify testops uploader logic
fix some progress plugin issues
@epszaw
epszaw force-pushed the realtime-watch-affected-tests branch from 8d1b3c1 to 1cc93ac Compare August 19, 2026 11:28
@epszaw
epszaw requested review from epszaw and formaceft-93 and removed request for epszaw August 19, 2026 11:29
@formaceft-93

Copy link
Copy Markdown
Contributor

allure run --watch no longer works - it is okay?

Comment thread packages/cli/src/commands/watch.ts
Comment thread packages/plugin-testops/src/plugin.ts Outdated
Comment thread packages/plugin-testops/src/plugin.ts
@epszaw

epszaw commented Aug 19, 2026

Copy link
Copy Markdown
Member

allure run --watch no longer works - it is okay?

It's ok because the previous implementation was unstable, and I decided to remove it due to the instability with different test runners and frameworks

@epszaw
epszaw requested a review from formaceft-93 August 19, 2026 12:49
todti added 3 commits August 19, 2026 13:50
…ast one

allGlobalErrors() is append-only, but the code only ever sent
allResults.at(-1): the same last error got re-uploaded every realtime
cycle, and if multiple new errors landed between updates, all but the
last were silently dropped. Restore the offset-based slice so every
not-yet-uploaded error is sent.
TestError has no stable id, but the store only ever appends to
allGlobalErrors() and never replaces existing entries, so the same
logical error keeps the same object reference across calls. A
Set<TestError> keyed by that reference dedupes the same way global
attachments already do (Set<string> by attachment.id), without
depending on array position staying meaningful.
Covers the exact scenario the at(-1)-based bug hit: several new errors
landing between updates, another update with nothing new, then one
more. Verified this fails against the old at(-1) logic (drops one of
two errors added in the same cycle) and passes with the current
Set-based dedup.
@epszaw
epszaw merged commit 76343c8 into main Aug 20, 2026
18 of 19 checks passed
@epszaw
epszaw deleted the realtime-watch-affected-tests branch August 20, 2026 11:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants