fix(test): confine the RLIMIT_FSIZE atomic-write repro to a child process - #227

Co-Authored-By: Claude Fable 5 <[email protected]>

… crawl, single-pass rewriters (#257)

* perf(core): add an ordered, bounded concurrency helper for API loops

`ordered_concurrent` / `map_ordered_concurrent` wrap
`stream::iter(..).map(f).buffered(limit)`: at most `limit` requests in
flight, results yielded in input order, nothing started until polled.
The serial patch-API loops can adopt it and fold results exactly as
before. `API_CONCURRENCY` (8) and `PROXY_API_CONCURRENCY` (4) carry the
per-client caps. futures-util was already in the lock; it is now a
direct dependency of core and the CLI.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* test(scan): pin API-loop ordering under reversed latencies

New subprocess suite for the three patch-API loops `scan` drives (batch
POSTs, per-package detail GETs, hosted record views). Every mock answers
later requests first, so an implementation that folds in completion
order, or lets a discarded response leak in, changes the output:

- batch: a 401 on the first chunk sends that chunk and all later ones
  to the proxy with one auth request and one warning; a 401 on chunk 3
  of 6 folds 0-2 from the auth API and replays 3-5 on the proxy; per
  batch 500 warnings print in chunk order; the all-failed error carries
  the last chunk's error.
- details: partial-failure warnings print in package order and the
  whole human preview equals a zero-latency run; the all-failed error
  names the last package.
- hosted wet run: record_fetch_failed warnings keep confirmed order and
  stdout, lockfile and ledger equal a zero-latency run.

The suite passes against the current serial loops (checked with the
baseline binary) and is the oracle for making them concurrent.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* perf(scan): fetch per-package patch details concurrently

`fetch_patch_details` awaited one `by-package` GET per package with
patches (74 on depscan, ~10 s of serial round trips). The queries now
run through `ordered_concurrent` (8 in flight, 4 on the public proxy)
and are consumed in `packages` order, so `results`, `failures`, the
warn-after loop and the all-failed rule see exactly what the serial
loop produced. `ApiClient::uses_public_proxy` picks the cap.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* perf(scan): run batch discovery concurrently with exact proxy fallback

The batch loop POSTed one chunk at a time (56 chunks on depscan). Chunks
now run through `ordered_concurrent` and are consumed strictly in chunk
order, so per-batch warnings, `batch_error_count`, `last_batch_error`
and the paid-access flag fold as before.

The authenticated-to-proxy downgrade keeps the serial loop's exact
sequence: the first chunk goes alone (a stale token still costs the
auth API one request), and at the first consumed chunk k whose error is
a fallback candidate — any index — the window is dropped, responses for
chunks past k are discarded unfolded, the same warning prints, chunk k
is retried on the proxy and the rest continue there (4 in flight).

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* perf(hosted): fetch patch record views concurrently on wet runs

A wet hosted run fetched `patches/view/{uuid}` for every confirmed
redirect one at a time (74 on depscan, ~9 s). The views now run through
`ordered_concurrent` and are consumed in `confirmed` order, so `records`
(newest wins) and the `record_fetch_failed` warnings are unchanged. The
ledger re-fetch on idempotent re-runs is deliberately kept.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* perf(scan): send telemetry off the critical path, flushed before exit

Scan awaited each telemetry POST inline (150-300 ms typical, up to the
5 s budget on a bad network) before carrying on. Its three events now
go through `spawn_patch_scanned` / `spawn_patch_scan_failed`: the event
is built and its endpoint resolved where it fires (same body, timestamp,
env reads and "Sending telemetry" debug line), and only the POST runs in
a background task. `scan::run` awaits `PendingTelemetry::flush` before
returning, so every event is still delivered, or given up on within the
same 2 s connect / 5 s request budget, before the process exits. The
inline trackers and every other command are unchanged.

Tests: core unit tests pin that a background send posts the same bytes
and headers as an inline one and that flush waits for it; telemetry_e2e
pins that each scan terminal (success, empty crawl, all batches failed)
delivers its one event and stays alive until the slow endpoint answers.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* fix(scan): deliver background telemetry before the first stdout write

The background send was only awaited after run_scan returned, so a
process killed after the event fired but before that flush lost it:
`scan | head` / `scan | true` dies of SIGPIPE on its first result write
(main restores SIG_DFL), and a Ctrl-C at a confirm prompt or a CI
SIGTERM had the same effect. The inline send it replaced had always
landed before any output.

`PendingTelemetry::flush` now drains (`&mut self`), and scan flushes at
the first output point after each event fires: right after the send on
the empty-crawl and all-batches-failed terminals (they print at once),
at the start of the human section (before the table, prompts and every
human exit), before the plain `--json` envelope, and inside
`discover_selected` right after the detail fetches (before its error
line and whatever the `--apply`, hosted and vendored `--json` arms
print next). The send still overlaps the by-package detail fetches on
those arms; the flush at the end of `run` stays as the exit backstop.

Under `--debug` this also puts the human path's "Telemetry sent" line
back ahead of the per-package detail warnings, as in the inline order.

Tests: telemetry_e2e runs each JSON terminal with stdout closed before
the child writes and requires the event delivered (red on the previous
commit: SIGPIPE, 0 events); a core unit test pins that flush drains and
that sends started after it join the next flush.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* refactor(telemetry): share the patch_scan_failed metadata builder

`track_patch_scan_failed` and `spawn_patch_scan_failed` each spelled out
the `{"fallback_to_proxy": ...}` literal; build it in one place, as
`patch_scanned_metadata` already is for the success event, so the inline
and background paths cannot drift. The inline trackers stay: they are
public API of the published core crate.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* refactor(core): keep the collecting concurrency helper test-only

No production caller used `map_ordered_concurrent`: every API loop
consumes `ordered_concurrent` directly. Move it into the tests module
so it no longer ships as unused public API.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* perf(crawl): walk npm trees on the blocking pool with parallel gather + ordered merge

`crawl_all` and the workspace roots walk made one `spawn_blocking` round
trip per readdir, stat and package.json read, strictly in sequence. Both
now run as one blocking-pool task: directory I/O is gathered in parallel
(rayon, already in the dependency graph via qbsdiff) into per-root event
trees that record the sequential visit order, and a single-threaded merge
replays them so the order-dependent `seen` dedup and the store entries'
`identity_seen` decisions see exactly the state the old walk saw — same
packages, same paths, same order.

Two probes are answered from listings the walk reads anyway, only where
that is provably the same answer:
- the roots walk skips the `is_dir(child/node_modules)` stat when the
  child's complete listing holds nothing that could alias `node_modules`
  on a case-insensitive filesystem (a listed dir still stats: a
  readable-but-unsearchable parent lists kinds while stats fail);
- a store entry's `node_modules` existence probe is the readdir the scan
  needs next; a dir that does not open falls back to the stat.

FIFO-safe package.json reads (read_regular_to_string_sync), the
NESTED_STORE depth/dir caps (kept sequential: the budget order decides
survivors), symlink-not-traversed rules and lossy-vs-raw name joins are
unchanged. The previous async implementation is kept verbatim as a
#[cfg(test)] oracle; a randomized fixture test (flat/nested/legacy stores,
scoped, live/dangling/store symlinks, duplicate identities, aliases,
broken/BOM/FIFO/dir package.json, unreadable and unsearchable dirs,
node_modules case variants) plus a kitchen-sink tree assert identical
roots, crawl output, find_by_purls results and store enumeration.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* perf(crawl): find_by_purls lists each node_modules once and probes only listed names

The resolver opened `<nm>/<target>/package.json` for every pending
target in every visited node_modules — targets × dirs failed opens, each
its own spawn_blocking hop — then listed the same dir again for the
descent. Both passes now run as one blocking-pool task: each dequeued
dir is listed once, a target is probed there only when the listing could
hold its first path component, the surviving probes run in parallel and
fold back in target order, and the same listing drives the descent
(whose per-entry stats also run in parallel, appended in listing order).

The name filter is a strict superset: it only engages for a complete,
all-ASCII listing and matches ASCII-case-insensitively (APFS/NTFS),
and components a filesystem can resolve to a differently spelled entry
(non-ASCII, `~` 8.3 aliases, trailing dot/space) are always probed. BFS
root-first order, every-copy collection, the name+version identity
check, the pass-2 fallback and the store-entry name filter are
unchanged. `.pnpm` entry names are still filtered after the
`node_modules` stat, not before: an entry without one is a nested host
whose synthesized children can match, so the stat decides the result.

The oracle equivalence suite (now also covering case-variant package and
scope dirs) asserts identical find_by_purls maps on every generated root.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* perf(crawl): run the nine ecosystem crawlers concurrently

`crawl_all_ecosystems` awaited each crawler in turn, and the crawlers
that block (maven's walkdir walk + POM reads, `gem env`, the python
site-packages probe, `composer global config home`) did so inline on the
async task. The crawlers are independent — none prints, none mutates
shared state — so they are now joined, with every blocking walk or
subprocess moved onto the blocking pool, and their results are consumed
in the fixed Npm, Pypi, Cargo, Gem, Golang, Maven, Composer, Nuget, Deno
order, so packages and counts are exactly the serial run's. The joined
futures are heap-allocated from a non-async constructor so the caller's
poll frame does not grow by their combined size (Windows main-stack
budget).

`gem env gemdir` and `gem env gempath` run concurrently but are still two
calls consumed gemdir-then-gempath (no single-call merge: platform path
separators). A polyglot `--global-prefix` test pins the joined output
against the serial sequence.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* perf(crawl): resolve find_by_purls one BFS level at a time

The previous commit fanned each visited dir's probes and descent stats
out to the rayon pool separately, one injection per dir: on a deep
pnpm tree the per-dir handoff latency outweighed the parallelism, and
`apply --dry-run` on a large monorepo ran slower than the async walk.

A visit's reads depend only on the dir and the fixed target list, never
on what earlier dirs resolved, so the walk now proceeds level by level
(exactly the FIFO queue's order: everything a dir enqueues lands behind
the rest of its level). Each level's visits — listing, filtered probes,
nested-dir discovery with the virtual store's entries returned whole —
are gathered in one parallel pass, then the order-dependent part (folding
matches into the result, the unmatched-name store filter, next-level
order) is replayed sequentially in queue order. Output is unchanged;
the oracle equivalence suite still covers it.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* test(crawl): keep the oracle fixture generator warning-free on non-Unix targets

Symlinks, FIFOs and permission stripping are generated on Unix only, so
the fields that record them are never read elsewhere.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* style(crawl): keep the test-only oracle module out of the npm crawler's use block

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* fix(crawl): walk workspace roots level by level instead of recursing

The parallel roots walk recursed once per directory level on rayon and
blocking-pool threads (2 MiB stacks), where the old async walk recursed
through boxed futures on the 8 MiB main thread. A deep enough directory
chain (reachable under Linux's 4096-byte PATH_MAX, and deeper on Windows
long paths) aborted the scan with a stack overflow the old walk
survived.

Read the tree one level at a time, each level's dirs in parallel, record
each dir's child range, then emit with an explicit stack in the same
depth-first order. Stack use no longer grows with depth; a new test runs
a 400-deep chain on 256 KiB walk threads (the recursive walk overflowed
there).

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* fix(crawl): run npm walks on a main-sized stack within the descriptor budget

Two properties of the old sequential async walk did not survive the
move to parallel sync walks on rayon's global pool:

- Stack: the recursive node_modules gather ran on 2 MiB worker threads
  instead of the 8 MiB main thread. The npm walks now run on a dedicated
  walk pool whose threads get the main thread's 8 MiB.
- Descriptors: every walker treats a failed read_dir/open, EMFILE
  included, as an absent dir, and the old crawl held one descriptor at
  a time with the nine crawlers run back to back. With one walk thread
  per CPU plus concurrent crawlers, depscan lost packages silently
  below `ulimit -n 24` (5349 of 5520 at 20) where the old crawl was
  intact down to 14. Under a soft RLIMIT_NOFILE below 128 the walk pool
  now gets one thread and the crawlers run one at a time (the old
  descriptor profile); above it the pool is capped at half of what is
  left after a 64-descriptor reserve. depscan now matches the baseline
  byte-for-byte at every limit from 16 to 256.

New tests: pool sizing, a 4 MiB frame fitting on a walk thread, and an
e2e scan under `ulimit -n 16` that must match the ample-limit JSON (the
one-thread-per-CPU pool lost most of that tree there).

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* perf(redirect): parse each pnpm lock once and splice in one pass

The hosted pnpm rewriter re-parsed every lock (entries, the early
shrinkwrap sniff, the residual gate) and rebuilt the whole lock string
once per dep: O(deps x lock) work that cost ~430 ms of critical-path CPU
on depscan's 2 MB lock with 74 redirected deps.

Each lock is now parsed and key-indexed once; a dep's instances are
found by binary search, the residual gate judges each instance on its
post-splice body, and committed splices are applied in one pass at the
end. A later dep that hits an already-spliced entry (a duplicate
name@version override) folds the pending splices in and re-indexes
first, so it re-reads the rewritten text exactly as before, and the
vendored-marker scan runs over the post-splice text the same way.

Output bytes, the FileEdit list (order and original fragments),
warnings and refusals are unchanged: the previous implementation is kept
as a test oracle and compared on a depscan-sized synthetic lock set, on
300 randomized mixes of every lock flavor, and on duplicate-override and
peer-suffixed multi-instance cases. depscan wet run: pnpm-lock.yaml and
redirect-state.json byte-identical.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* perf(hosted): probe Python locks once and fetch wheel metadata concurrently

Deciding which pypi deps need hosted wheel metadata ran a full
`rewrite_python_lock` (parse, a second parse for the source-scope check
on script locks, mutate, serialize) per dep per lock, only to test the
result for `Some`. The rewrite's refusal and not-applicable checks now
live in one planning step that `rewrite_python_lock` and a new
`PythonLockProbe` share: the probe parses each lock once and answers
exactly `matches!(rewrite_python_lock(..), Ok(Some(_)))` per dep, and
the rewrite no longer re-parses the lock for the scope check.

The qualifying wheels are then downloaded through an ordered
`buffered(8)` stream and folded in dep order, so `python_metadata`, the
withheld artifacts and the `python_metadata_unavailable` skips are
unchanged. The stream is inlined here (futures-util added with the same
workspace spec as the scan-concurrency branch); it moves onto the shared
ordered-concurrency helper once that lands.

New tests: a probe/rewrite equivalence sweep over every lock shape and
outcome, and a hosted scan whose slow first failure must still be
reported before a fast second one.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* perf(redirect): derive npm and yarn-classic entry identities once per lock

The npm package-lock rewriter re-derived every `packages` entry's
identity (the `node_modules/` key split plus the `name`/`version`
lookups) for every dep, and the classic yarn.lock rewriter re-split
every block's key patterns for every dep: O(deps x entries) work that
dominated both rewriters' CPU.

Each identity is now computed once per lock. npm entries keep theirs by
map position (a rewrite only touches `resolved`/`integrity`, never a
key, `name` or `version`); a yarn block's key and sole real package are
recomputed whenever this run rewrites that block, so later deps still
see its current text.

Output bytes, FileEdits and warnings are unchanged: both previous
implementations are kept as test oracles and compared on 400 randomized
locks each (aliases, links, bundled copies, workspaces, v1/v2
dependency trees, alias-only and fork-substitution yarn keys, CRLF and
mixed line endings, duplicate overrides). Rewrite-phase CPU on the
lockfile-only benches: npm-socket 103 -> 61 ms, yarn-strapi 49 -> 32 ms
(whole-process medians).

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* test(redirect): pin the pnpm residual boundaries on the production gate

The indexed rewriter judges residuals inline, so the boundary test over
`pnpm_unrewritten_instances` now covers only the test-only reference.
Feed the same boundary locks through `rewrite_registry_redirect`: hosted,
longer-version, scoped and snapshot keys never count, v6 nested-paren and
v5 `_` instances are repointed, and only the unparseable instance is
named in the refusal. The helper's doc comment now says what it is.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* perf(hosted): cap wheel-metadata fetches at 4 and test that they overlap

Each in-flight wheel download buffers the whole wheel under its own body
timeout and retry budget, so memory and link sharing scale with the
limit; 4 keeps the overlapped round trips while halving that. The
comment records what concurrency changes that output cannot see (status
line names the awaited dep, debug lines interleave, Retry-After pauses
one fetch). The order test now also records request arrivals and fails
if `bbb` is not requested before `aaa`'s delayed response is due, so a
regression to serial fetching is caught.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* perf(scripts): add a record/replay network benchmark harness

scan time is dominated by API round trips, so live timings are noisy and
unrepeatable. scripts/perf/replay.py stands in for api.socket.dev,
patch.socket.dev and the public proxy: `record` forwards and stores every
response, `replay` serves only from the store with a fixed or recorded
per-request latency (plus optional per-connection latency) and reports
request counts per endpoint, max in-flight, connections and network span.
Batch POSTs replay per purl, so a build that changes chunking or order
still gets identical answers. The listener skips HTTPServer's getfqdn(),
which stalls ~35 s under the macOS sandbox.

scripts/perf/bench.sh drives it: `record`, `replay`, and `ab`, which runs
BASE and NEW interleaved against one store and fails unless every run's
stdout sha256 and exit code match the first BASE run. Stores hold real
API responses (possibly paid-patch data), so bench.sh refuses a store
path inside the repository.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* test(scripts): cover the perf record/replay harness offline

Syntax-checks replay.py (py_compile) and bench.sh (bash -n), then drives
the harness against a local upstream stub: record-then-replay with batch
re-assembly across chunks and orders, miss/unknown-purl accounting,
--fill, a 502 (never stored) for an unreachable upstream, per-request
latency with max in-flight, the getfqdn-free bind, the in-repo store
refusal, and an end-to-end `bench.sh ab` pass and sha-mismatch failure
with fake CLI binaries. Picked up by the existing
`unittest discover -s scripts/tests` CI step.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* refactor(hosted): fetch wheel metadata through the shared ordered_concurrent helper

The inline stream::iter().buffered() from the wheel-metadata fan-out
predates utils::concurrent landing; route it through ordered_concurrent
with the same limit (4) and the same in-order fold.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* fix(scan): deliver background telemetry before the first stderr write too

0ef24902 flushed the scan event before the first stdout write after it
fires, but stderr raises SIGPIPE just as well (main restores SIG_DFL).
Two stderr writers could run in that window with the send still in
flight: the lenient redirect-ledger load's "Warning: <corrupt ledger>"
(non-hosted JSON and human paths, before discover_selected or the human
flush) and, on the report-only JSON arm, the GC and VEX build ahead of
the envelope. The inline send it replaced was always delivered first.

The ledger load is inlined at its scan call site so the send is flushed
right before its warning (only when it warns, so the overlap with the
detail fetches is kept), and the JSON arm flushes before the GC/VEX
step instead of just before the envelope. The --apply arm's warnings
already follow discover_selected's flush.

Test: telemetry_e2e runs a scan over a malformed redirect ledger with
stderr closed and requires the event delivered (red before: SIGPIPE,
0 events). It uses a well-shaped token so the token-shape warning does
not kill the child before the event fires.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* docs(telemetry): say why the inline scan trackers stay public

scan now sends patch_scanned / patch_scan_failed through the spawn_*
variants, which leaves the inline trackers without an in-tree caller.
They stay: socket-patch-core is published to crates.io, removing a pub
fn is a breaking change there, and every other event keeps its inline
tracker. The doc comments now say so, so a later cleanup does not read
them as leftovers.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* test(scan): pin that the mid-run fallback discards in-flight answers

batch_fallback_mid_run_replays_from_the_failing_chunk checked the folded
uuids and the proxied tail, but not that chunks 4-5 were ever sent to
the authenticated API. A serial loop (or a window of 1) would never
request them and still pass. Assert all 6 authenticated requests: chunk
0 alone, then the whole 1..6 window in flight, so the discard path
really runs.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* perf(api): share one in-flight cap across concurrent proxy batch calls

On the public proxy scan runs up to PROXY_API_CONCURRENCY (4) batch
windows at once. Each window's search_patches_batch degrades to the
legacy per-package GETs (10 at a time) when /patch/batch rejects the
chunk: a 400 from one exotic purl such as pkg:jsr, or an old proxy with
no batch route. So a polyglot project on the proxy could put 4 x 10
by-package GETs in flight where the serial loop peaked at 10. That
path swallows per-purl errors as "no patches", so extra load that
saturates the proxy could change which packages come back.

The client now holds a semaphore of PROXY_BATCH_PATH_CONCURRENCY (10)
slots, shared by clones. Every proxy /patch/batch POST and every legacy
per-package GET takes a slot, so all concurrent batch calls on one
client stay within the old peak. A single call never waits: its groups
of 10 fit the cap exactly as before. The authenticated API is untouched.

Test: four concurrent batch calls of 10 purls each, all rejected with
400, keep at most 10 by-package GETs in flight and still reach 10 (red
without the slots: 40).

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* fix(crawl): never fall back to rayon's global pool when walk threads can't spawn

When the walk pool could not be built (the OS refusing threads: a tight
RLIMIT_NPROC or cgroup pids.max, or a huge RAYON_NUM_THREADS), run_walk
ran the walk on the calling thread and the first parallel iterator then
tried to build rayon's global pool, which needs the same refused threads
and panics (exit 101) where the sequential walk succeeded.

- Retry the pool build with half the threads on each failure, down to 1.
- Route every parallel gather through walk_pool::par_map, which maps
  sequentially (in order) on a thread outside any rayon pool, so the
  no-pool fallback never reaches the global pool.
- RAYON_NUM_THREADS can lower the walk thread count but no longer raise
  it past available_parallelism.

Tests: halving build, par_map's sequential/ordered contract, run_walk's
no-pool path, and the randomized oracle comparison with the pool off.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* test(crawl): pin the store-entry identity_seen skip against the oracle

A store entry whose own child's package.json disagrees with the entry's
name@version, for a root-installed package, must have that child skipped
by name as the sequential walk did; nothing failed when the merge
dropped the skip.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* test(crawl): pin the resolver probe filter's always-probe components

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* test(cli): pin the one-at-a-time crawler dispatch under a tight fd limit

The npm-only tree could not tell the serial dispatch from the concurrent
one: the walk budget alone gives it one walk thread at ulimit -n 16. Add
a tree where python, bundler and composer crawlers also find packages,
and scan it at 12 on macOS (the serial dispatch is complete down to 11;
running the crawlers concurrently drops 40-80 packages at 12).

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* perf(hosted): settle only first attempts concurrently, retry wheels serially

The concurrent wheel-metadata fetch let every in-flight download run its
own retry loop. Against a host that serves one download at a time and
429s the rest with a shared Retry-After, the retries woke together,
collided again and drained their budgets, so a redirect the serial loop
makes was dropped as python_metadata_unavailable; the opt-in debug lines
also interleaved across downloads.

Now only first attempts run concurrently, with their debug lines held
back and printed at the dep's fold. The first attempt the client would
retry stops the fan-out: in-flight attempts are awaited (so the host is
idle, as the serial loop finds it) and that dep plus every later one
without a settled attempt is fetched one at a time with the full retry
budget and Retry-After pacing. Outcomes, JSON and the debug stream match
the serial loop's.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* test(redirect): require the random pnpm sweep to reach every outcome

The randomized oracle sweep now asserts it produced edits, refusals, a
duplicate that re-reads the prior rewrite and each residual warning
code, so a generator change cannot silently narrow it to the plain
rewrite path. The vendored-marker scan materializes pending splices in
its own loop instead of inside the any() predicate.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* perf(scripts): let bench.sh take explicit ports and refuse busy ones

PATCH_PORT and PROXY_PORT can now be set independently of PORT (they
still default to PORT+1 / PORT+2), and bench.sh exits 2 before starting
replay.py when any of its three ports is already listening, instead of
colliding with (or tempting someone to kill) another bench's stand-in.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* fix(scan): request one at a time under a tight descriptor limit

Every in-flight request holds its own socket, and the serial loops never
held more than one. Under a soft RLIMIT_NOFILE the crawl already treats
as tight (walk_pool::fd_limit_is_tight), the concurrent batch window
opened enough sockets to fail with EMFILE where the serial loop got
the server's own answer. crawl_fd_limit_e2e caught this at ulimit -n 12
once perf/wp1 and perf/wp2 met. api_concurrency and the wheel-metadata
fan-out now drop to 1 under that limit, restoring the serial descriptor
profile.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* perf(get): fetch patch views concurrently ahead of the download loop

The shared download loop (agent and vendored engines), the release-
variant narrowing, the vendor stager's content top-up and scan's
baseline pre-verification each awaited one patch-view GET at a time.
Each now plans the exact views its loop fetches (same refusal, ledger
and held-view checks, over inputs the loop never mutates), runs them
through the ordered concurrency helper and takes the next result where
it used to await the request. Results fold in selection order, and
each request's --debug lines are held back and printed at its old turn
(new core HeldBack/hold_back_debug), so stdout, stderr and the JSON
records match the serial loop.

vex's record fetch swaps its chunk-at-a-time JoinSet for a sliding
window of the same size, consumed in input order.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* perf(vendor): reuse scan's npm crawl instead of walking node_modules twice more

A vendored scan crawled the tree, then its vendor engine walked it
again twice: find_packages_for_rollback rediscovered every node_modules
root, and a single alias-installed ("missing") npm purl triggered a
full NpmCrawler::crawl_all in npm_paths_by_identity.

scan now keeps the npm half of its crawl (the crawler's packages and
the roots it walked, via new NpmCrawler::crawl_all_with_roots) and
hands it to vendor_records_reusing: the roots replace the targeted
lookup's root discovery (each root is still searched by find_by_purls,
so copy choice and order are unchanged) and the packages replace the
identity crawl. The snapshot is only used for the exact options it was
taken with, and only when nothing can have changed the tree since: the
JSON arm, and the interactive arm when its prompt answers without
waiting on a person. The vendor command, repair and get keep crawling.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* perf(vendor): prefetch service downloads ahead of the serial wiring loop

Each vendored npm package made its two patch-service round trips (the
package-reference POST and the archive GET) inside the serial dispatch
loop, back to back with its lockfile and ledger writes.

vendor_records now attaches a download plan to the run's client: the
npm records the loop is expected to download (in loop order, past the
Bun refusal and the takeover gate, and without a committed artifact
the ledger anchors at the record's uuid, which a re-run reuses). A
background task fetches the plan at most api_concurrency ahead, and
fetch_vendor_package takes a planned uuid's outcome instead of making
the requests. Single-uuid POSTs are kept.

The plan is advisory; every decision stays at the loop's own call, in
order: the circuit breaker is checked and its count updated there
exactly as before (the prefetch never touches it, and a call it skips
discards the prefetched bytes), skipped or unplanned calls fetch live,
and each prefetched request's --debug lines print at its call. The task
starts only at the first planned call and stops speculating after the
breaker threshold of its own consecutive availability failures; the
plan detaches (aborting the task) when vendor_records returns.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* perf(vendor): fetch lockfile-only packages from their registries concurrently

The vendor engine fetched every missing (lockfile-only) package's
pristine artifact one at a time. It now decides each purl's local rungs
first (committed-artifact staging and the --offline stop, which are
local and read-only) without emitting anything, fetches the purls left
for the registry at most api_concurrency at a time through the ordered
helper, then emits every purl's outcome in the original order: the
same warnings, failed events and stderr lines, in the same sequence.
The lockfile inventory is still parsed only when some purl reaches the
registry rung.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* fix(scan): end the batch window instead of re-running it

The window loop's defensive `else { break }` left `next` where it was,
so the outer loop rebuilt the identical `chunks[next..end]` window and
re-POSTed every chunk in it — an unbounded request loop where a stop was
meant. `ordered_concurrent` yields exactly one item per chunk, so the
branch is unreachable today; the two sibling loops (`fetch_patch_details`
and the hosted view loop) just end their `for`, and only this one could
turn a short stream into a retry storm.

Label the outer loop and break out of it, with a `debug_assert!` so a
later `ordered_concurrent` variant that can short-circuit fails loudly in
tests rather than quietly re-requesting in production.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* feat(api): let SOCKET_API_CONCURRENCY pace the patch-API windows

`scan` now keeps up to 8 patch-API requests in flight (4 on the public
proxy) where it used to await one at a time. api.socket.dev answers 32
in flight without a 429, but an endpoint in front of a self-hosted
`--api-url` — a corporate reverse proxy, a WAF, a CDN — may cap
concurrent requests per client, and a rejected batch or detail fetch is
only counted, never retried: a `--json` scan would report fewer patches
with `status: "success"` and nothing on stderr. Until those loops learn
to honor `Retry-After`, an operator who hits that has no way back short
of rebuilding, because both caps are compile-time constants.

Give `api_concurrency` an env override, clamped to 1..=32, with 1
reproducing the serial loop exactly (`buffered(1)`). On the public proxy
it can only lower the cap: that server's semaphore is shared across
anonymous callers, so widening it is not one client's call to make. An
unset, empty or unparsable value keeps today's defaults — a stray export
must not fail a scan — and a set-but-unusable one says so under --debug.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* perf(scan): open the proxy batch window at the first chunk

The chunk-0-alone rule exists to cap what a stale token costs the
authenticated API before the downgrade: one rejected request, as the
serial loop always paid. The fallback arm it protects is guarded by
`!use_public_proxy`, so on a token-less run — the common unauthenticated
path — it can never fire, and serializing the first chunk buys nothing.
depscan pays 1 + ceil(55/4) round trips for its batch phase where 14
would do.

Open the full window at chunk 0 when we are already on the proxy. Output
is untouched; the new e2e pins that the second chunk now arrives while
the first is unanswered, that SOCKET_API_CONCURRENCY=1 puts the same run
back on one request at a time, and that both produce identical stdout.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* perf(hosted): keep records, not whole payloads, in the view window

The wet-run view loop buffered whole `PatchResponse` values, and a view
carries every file's `blobContent` / `beforeBlobContent` — so peak
memory became the in-flight cap times the largest patch payload, where
the loop only ever wanted the record's hashes. Measured on 24 packages
whose views return 4 MB each: maxRSS 34 MB serial vs 73 MB concurrent.

Fold each response to its record inside the window.
`record_from_patch_response` is pure, so stdout, the `record_warnings`
order and the ledger bytes are unchanged.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* refactor(telemetry): one prepared-event builder per scan event

`scan` sends its two events through the spawn trackers, so the inline
`track_patch_scanned` / `track_patch_scan_failed` — kept as this
published crate's public API — are the only wrappers with no caller and
no test. Each still named its event type and `"scan"` a second time, so
a rename could reach one path and not the other with nothing to catch
it.

Give each event one `prepare_*` builder that owns type, command and
metadata, and express the inline and background wrappers on top of it
(`fire_prepared` / `spawn_prepared`), the way the metadata builders
already were. A new test posts both paths at one mock and compares the
bodies modulo timestamp.

Also record the one accepted `--debug` divergence from the inline send:
"Sending telemetry to …" still prints where the event fires, but its
"Telemetry sent successfully" twin now prints where the send finishes,
so the pair is no longer adjacent.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* perf(crawl): record only the targets a resolver visit matched

`visit_resolver_dir` handed the sequential fold one probe slot per
pending target, and every dir of a BFS level is live at once, so
`find_by_purls` peak memory scaled with `level dirs × targets`.
Pass 2 enqueues a pnpm store whole, so a 2,000-entry store probed for
1,500 uninstalled manifest purls cost 166 MB of resolver state (and
grew linearly past it) where the one-probe-at-a-time walk stayed flat.

Whether a probe matches depends only on the dir and the target, never
on what earlier dirs resolved, so decide it in the visit and carry one
index per MATCH: the same fold, the same order, the same copies, with
memory proportional to matches. Measured on a 2,000-entry store:
6.4 MB at 50, 500 and 1,500 targets (was 11.5 / 58.5 / 166 MB).

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* fix(crawl): skip the main-sized-stack probe when no walk pool exists

`walk_runs_on_a_main_sized_stack` allocates a 4 MiB frame inside
`run_walk`, which fits only because the walk pool's threads carry the
main thread's 8 MiB. When `build_with_fallback` returns `None` — the
case it exists for: the OS refusing threads under `RLIMIT_NPROC`, a
cgroup `pids.max`, or low memory — `run_walk` falls back to the
blocking-pool thread and its 2 MiB default, and the frame overflows:

    thread 'tokio-rt-worker' has overflowed its stack
    fatal runtime error: stack overflow, aborting        (SIGABRT)

That takes the whole socket-patch-core lib binary with it, all 4,313
tests, not just this one — the shape PR #227 already fixed once for the
core fs.rs rlimit test. There is nothing to pin on a machine that could
not spawn a walk thread, so return early instead, and say in the module
docs which stack the fallback path actually runs on.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* docs(crawl): peak memory now scales with the walk thread count too

The walk pool budgets descriptors and says so; nothing budgets memory.
`gather_package` and `visit_resolver_dir` call `read_package_json_sync`
from inside `par_map`, and `read_regular_to_string_sync` sizes its
buffer from the file with no cap, so one outsized package.json in an
untrusted tree now costs up to `walk_threads()` copies where the
sequential walk paid one. Real trees are unaffected (a depscan crawl
grew 104 -> 122 MB) but the amplification is real on a pathological
input, and the two ways to bound it — a size cap, which changes what
the crawler inventories, or a read semaphore, which trades back the
throughput this walk exists for — are decisions, not oversights.
Say that in the module docs rather than leave it implicit.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* test(cli): name the descriptor budget when a tight-limit scan drops packages

`tight_descriptor_limit_scans_every_ecosystem_the_same` runs at
`ulimit -n 12`, and the measured cliff on macOS is 11 (complete) / 10
(445 of 485 packages): exactly one descriptor of slack. Anything that
holds one more open across the crawl — a config read, a cert store, a
log file — turns this into a 40-package JSON diff that says nothing
about why. Compare `scannedPackages` first, with a message that names
the budget and the limit, and record the measured cliff next to the
limits array so the slack is not a surprise. The byte-for-byte stdout
comparison still runs, unchanged, right after.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* test(cli): give the ecosystem-order fixture an install in all nine

`crawl_all_ecosystems_matches_serial_order` compares the concurrent
crawl against a serial run built in the test's own copy of the fixed
order — but the fixture only produced npm, pypi and cargo packages, so
the other six contributed nothing to the concatenation and their
POSITION was unobservable. Reordering the consumption array to
Golang, Gem, Deno, …, Maven left the test green.

That order is shipped behavior: scan chunks the crawl-ordered purls
into batches, so it decides batch composition, the `batch N/M failed`
warning text and order, and `last_batch_error` (the global purl sort
happens only afterwards). Stage one install per ecosystem in the
`--global-prefix` root — each crawler's own global-prefix layout,
chosen not to collide — and assert every ecosystem contributed. The
same reordering now fails.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* docs(crawl): say what the parallel gather costs, not only what it buys

Two trades the sequential walk did not make, both deliberate, neither
written down:

- `gather_package` reads a store entry's self-named child unconditionally,
  and `merge_scan_events` then discards it for every root-linked direct
  dep. The gather cannot know the merge-time dedup state, and deferring
  that read to the merge thread would put the store's transitive-only
  reads — never skipped, and the bulk of a virtual store — on the serial
  path to save one open per direct dep.
- `crawl_all_sync` buffers every root's events before merging, so peak
  memory follows dirs visited rather than unique packages (about +18% on
  depscan). Merging per root would barely move it: 5,080 of depscan's
  5,520 packages sit in the root's virtual store, so the largest root's
  events are the peak either way.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* fix(hosted): resume a deferred wheel attempt, never restart its budget

The concurrent wheel-metadata fan-out settles only first attempts and
hands a retryable failure back to the one-at-a-time path. That path ran
the WHOLE of `fetch_hosted_wheel_metadata`, so the discarded attempt
became a free extra try: a host that 503s a wheel three times and serves
it on the fourth request failed the serial loop but succeeded here —
different `redirect.skipped`, different `redirect.redirected`, and in a
wet run a uv.lock the serial loop never writes. The pause the discarded
attempt earned was dropped with it, so a 429 host was re-requested at
once instead of after its `Retry-After`, and the attempt's debug lines
were dropped too, under-reporting real traffic on the `--debug` stream.

`download_artifact_first_attempt` now hands back what the attempt left
unspent (`DeferredAttempt`) instead of `None`, and
`download_artifact_resuming` reports that failure, waits out its
`Retry-After` and spends only the REMAINING attempts. So a wheel costs
the host the one-at-a-time loop's requests however the fan-out splits
them, and every request it really made is on the debug stream.

What is left is the overlap itself: a host that answers a burst
differently than it answers the same requests one at a time can still
return a status the serial loop would not have seen. The comment at the
fan-out now says that, instead of claiming no outcome can be worse.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* test(redirect): compare every RewriteResult channel from one shared oracle

The npm/yarn and pnpm equivalence sweeps each carried their own copy of
the xorshift `Rng` and their own `snapshot`, and the copies had already
drifted: the npm/yarn one compared bytes, edits and warnings, the pnpm
one those plus `refused_pnpm_uuids`. Neither looked at `binary_files` or
at the other eleven uuid sets, so a rewriter that started writing one
would have been compared nowhere.

Both now use one `rewrite_oracle_support` module whose snapshot covers
every `RewriteResult` field, each uuid set named so a mismatch says
which one, and whose `assert_same` keeps the pnpm sweep's first-differing-
byte message for both.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* perf(vendor): hand back slices from the pnpm key-line parser

`parse_key_line` is the pnpm surgery's inner loop: every structural
probe and edit runs it over whole `packages:` / `snapshots:` sections,
once per vendored package. It allocated three Strings per call, so on
depscan's 2 MB lock it was the single hottest frame in a vendored run
(271 ms of the engine's ~400 ms of lock CPU, sampled at 1 ms).

It now returns slices of the line. Call sites that keep a piece past
the next edit to `lines` copy it themselves, and the four that spliced
while still holding one build their wiring record first — the record
and the rewritten line are byte-identical either way, and `YamlBlock`
and `dep_field_lines` keep their owning contracts (their values
outlive the edits that follow).

Pure refactor: a new sweep pins the parser against the owning version,
kept verbatim as the test oracle, over every shape a real lock mixes
(bare and both quote styles, `file:` keys containing `:`, peer
suffixes, empty/inline values, list items, stray colons, unbalanced
quotes, a stray CR, non-ASCII) at every indent the callers use.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* style(vendor): drop the rustfmt churn this work package never asked for

`registry_fetch.rs` and `toml_surgery.rs` came along with A6 carrying
nothing but reflowed `format!` arguments and test literals — 45 lines a
reviewer has to read and confirm value-preserving in two files whose
logic this branch never touches, and a conflict waiting for anyone else
editing those test modules. Restore both to the integration branch.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* fix(api): bound the vendor prefetch by what the loop has reached

The plan ran as a detached task with an ordered channel, which cost more
than the speedup was worth in three ways the oracle tests could not see,
because they compare outcomes and never request volume:

* A package the loop refused before the service (a backend's own
  pre-flight: an unsupported lock entry, an override conflict) was still
  passed over through the channel, and because `buffered` delivers in
  order, the NEXT package the loop wanted waited behind that download —
  a request the serial loop never made could stall the wiring loop for
  the whole retry ladder.
* The task ran to the end of the plan whatever the loop did, so a run
  that stopped consulting it left up to `window` in flight plus `window`
  queued: grants minted and archives downloaded, held in memory and
  thrown away, for packages the run never vendors.
* On an outage it opened `window` retry ladders at once and only noticed
  at consumption time, aiming four times the serial loop's requests at a
  service that was already answering 503 — the opposite of what the
  run-level breaker exists to do.

So the task now requests only plan positions in `[at, at + reach)`,
where `at` is where the loop has got to and `reach` opens at one and
widens to `window` at the first answer that proves the service is up,
and it stops issuing at the breaker's threshold. Outcomes are delivered
as they finish and `take` puts an early one aside for its own call, so a
passed-over download never gates the loop. A uuid `fetch_vendor_package`
refuses without any I/O is dropped as the plan is attached, rather than
POSTed speculatively.

Nothing observable moves: every outcome, its debug lines and the
breaker's count are still decided at the loop's own call, in loop order.
New tests pin the bounds that were invisible before — an outage costs
exactly the serial loop's requests, one consumed package never costs
more than a window of grants, and a stalled passed-over download does
not delay the next call. The guard also detaches only the plan it
attached, and the prefetch task goes through `hold_back_debug` rather
than a second spelling of it.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* fix(vendor): honor the one-request-at-a-time escape hatch in the prefetch

`SOCKET_API_CONCURRENCY` is documented as the operator's escape hatch for
an endpoint that caps in-flight requests per client, and `1` as
"restores the old strictly serial loops exactly". That held for every
`buffered` consumer but not for the vendor service: `prefetch_archives`
only declined on a disabled service, a plan of one, or a tight
descriptor limit, so an operator who asked for one request at a time
still got speculative grants running ahead of the loop — one more socket
than they allowed, for packages the run may refuse.

`wants_prefetch` now answers the whole question (service enabled, and a
cap above one, which a tight descriptor limit already forces), and the
engine asks it BEFORE walking every package against the records, the
ledger entries and the redirect ledger to name the plan — a
`--vendor-source build`, `--offline` or serial run paid that
O(packages x ledger) walk only to drop the Vec.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* perf(vendor): pace registry downloads with a cap of their own

The pristine fetch for lockfile-only purls was bounded by
`api_concurrency`, which is the authenticated PATCH API's cap: measured
against that service, and turned up or down by `SOCKET_API_CONCURRENCY`.
But `fetch_pristine_package` downloads from registry.npmjs.org, PyPI,
RubyGems, crates.io, the Go proxy and Maven Central, and
`build_registry_client` has no retry, no backoff and no `Retry-After`
handling at all. So an operator lowering the cap for their own patch
proxy silently throttled public registries that never went through it,
and one raising it to 32 fanned 32 tarball downloads at a registry where
a 429 turns a purl from vendored into `vendor_fetch_failed` and exit 1 —
a divergence the serial loop could not produce.

`registry_concurrency` is its own constant at 4 (still latency-flat on
the lockfile-only ladder), with no env override and the same
descriptor-limit rule. While here, the fetch plan and the lazy lock
inventory ask one `needs_registry` predicate instead of spelling the
same `matches!` twice, and the unreachable arm that would re-fetch every
remaining purl says so with a `debug_assert!`, like its twin in `get`.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* fix(cli): check that a prefetched view belongs to the patch taking it

Three of the concurrent view consumers took the next result positionally
and trusted the plan to have named the same patches the loop reaches.
The plans are right today — each mirrors its loop's own skip rule — but
the failure mode when one drifts is silent and wrong, not loud:

* `preverify_vendor_baselines` would compare a package's installed bytes
  against ANOTHER patch's file hashes, and store that response under
  this patch's uuid in the map the download phase then reuses instead of
  fetching — so the wrong record and files are written with no request
  ever made.
* `filter_to_installed_releases` would hash-match a variant against
  another release's files, keeping the wrong distribution (or warning
  that none matches one that does) and pairing each uuid with another
  variant's response.
* `stage_vendor_sources_in_memory` would `break`, returning `Ready` with
  blobs missing and nothing in `failed` — a silently incomplete staging
  where the serial loop produced `no_local_source`.

All three now match the identity the way `fetch_selected_patches`
already did — `Some((planned, view)) if planned == uuid`, a
`debug_assert!` and a live fetch — so a drift is a test failure in debug
and merely a repeated request in release, never a wrong answer. Two new
tests fetch under reversed latencies over a `selected` list mixing every
plan outcome, and pin both the annotations and the uuid → response
pairing; each fails on the drift it guards.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* refactor(ui): ask prompt whether a confirm waits, instead of re-deriving it

`scan` decides whether to hand its pre-prompt npm crawl to the vendor
engine by asking whether the prompt below will stop for a person, and it
answered that by hand-copying `confirm`'s early return and its
`interactive:` field. The copy was right, but the drift it invites is the
unsafe direction: a `confirm` that waits where `prompt_waits` says it
does not means the engine resolves purls against a `node_modules` the
user was free to reinstall while the prompt sat there.

`ui::prompt::confirm_waits` is now that one question, and `confirm`
itself asks it for its `interactive` field, so the two cannot come
apart. A unit test pins them together over the whole `{yes, json}` cube.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* refactor(crawl): make a new crawler option a compile error in the snapshot

`NpmCrawlSnapshot::taken_with` is what makes reusing scan's crawl sound:
it says the snapshot was taken with exactly these options. It compared
the three fields `CrawlerOptions` has today, so a fourth one that
changes what the npm crawler walks would leave it answering true, and
the vendor engine would resolve purls against a crawl of a different
tree — no compile error, no test failure, wrong paths. Destructure the
struct so the compiler asks the question instead.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* perf(scan): keep the npm crawl snapshot only where it is read

`crawl_all_ecosystems_with_npm` copies the npm half of the crawl —
every `CrawledPackage`'s four heap fields — so the vendor engine can
resolve purls against it instead of walking `node_modules` twice more.
Only the vendored arms read it, but `scan` built it on every run: on
depscan that is about 5,000 packages cloned on a `--mode hosted
--dry-run` that never constructs a vendor engine, in a work package
whose whole point is the hosted path's wall time.

Vendored mode takes the snapshot; every other mode calls
`crawl_all_ecosystems` and pays nothing.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* perf(vendor): keep fewer whole archives in memory than the API allows

A prefetched service download is a whole tarball resident in memory
where the serial loop held exactly one, and the plan ran at the patch
API's in-flight cap — eight, or up to 32 with the operator override.
Measured on eight 24 MB archives, peak RSS was 209 MB serial against
307-312 MB planned; a `-g` run or a project with large npm artifacts
scales that linearly, and nothing bounded it.

The count bound is now `window` rather than `window` in flight plus
`window` queued, and archives get their own window of four under the API
cap. The wiring loop is fsync-bound between packages, so four downloads
ahead keep it fed; each further one only buys peak memory. A byte budget
on top is deliberately NOT added here: it is a second gate over the same
futures, with a deadlock to get wrong and no cheap test, and the count
bound already takes the worst case from 16 archives to 4.

Also records what `vex_sources`' sliding window changed that the work
package listed as a no-op: the chunked JoinSet folded by COMPLETION
order, so which refusal was reported in the auth-fallback note was a
race. It now follows `pending` order, with a test.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* test(api): pin what a package the loop refuses costs in requests

The oracle suite compares outcomes and the breaker's count, so nothing
in it would have noticed the speculation widening. This pins the request
side of the bound the module docs now state: with the loop consulting
the plan at two positions, the task may only ever request a window from
each, so a planned package the loop refuses before the service costs one
grant and never a retry ladder.

The docs also name the measured figure, so the waste is a number a
reviewer can check rather than a shape: 74 download grants on a depscan
vendored run where the serial loop made 71 — the two
`vendor_lock_entry_unsupported` packages and the one
`vendor_override_conflict`, all refused by a backend pre-flight the plan
cannot see.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* fix(hosted): pace the wheel-metadata window with the documented knob

The hosted wheel-metadata window sized itself from `fd_limit_is_tight()`
alone, so `SOCKET_API_CONCURRENCY` — the escape hatch for an endpoint that
caps in-flight requests per client — reached every other patch-API window
but not this one. A `scan --mode hosted` on a uv.lock project with four
patched wheels still fired four concurrent artifact GETs at the patch
server, and the capping endpoint's rejections landed those deps in
`skipped` as `python_metadata_unavailable`: the silent degradation the
knob exists to prevent.

Size it the way `prefetch_archives` already does, from
`api_concurrency(..)` capped by the window's own whole-wheel-per-request
ceiling. `api_concurrency` returns 1 under a tight descriptor limit, so
the descriptor behavior is unchanged.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* perf(vendor): stop the download plan amplifying an outage

Two ways a mid-list vendor-service outage cost the service more requests
than the serial loop, neither of which the down-from-the-first-package
test could see:

- When the task's own breaker opened it returned, dropping the stream.
  The requests still in flight were for packages BEHIND the failures —
  ones the loop had yet to reach — so they were cancelled mid-response
  and the loop then re-issued each of them live. Their retry ladders were
  paid twice. It now stops STARTING work and drains what is in flight, so
  those outcomes reach the loop instead. The stop is sticky: a success
  draining out from behind the failures resets the consecutive-failure
  count, and must not let the speculation resume against a service the
  loop is giving up on.
- Once the window had widened over a few granted packages, a failure did
  not hold it back: every position the loop arrived at admitted another
  one, so a service that went down part-way down the list was handed a
  window of retry ladders for packages the serial loop — one failure from
  opening its own breaker — asked nothing for. Nothing past a failing
  position is started now until the loop has consumed that position.

Outcomes are unchanged in both: every decision still happens at the
loop's own call, in loop order. The mid-list outage test now pins the
request count against the serial run, and a new test pins the one case
the plan cannot make free — the loop caught up with the window, the whole
window failing at once — to at most a window of ladders.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* refactor(api): hold back every concurrent window's debug lines

get.rs, scan/discovery.rs and fetch_stage.rs wrapped each concurrent
request in `hold_back_debug` so its `--debug` lines print at the
consuming turn; scan's by-package loop, scan's batch window, hosted's
record-view loop and vex's record fetch called the client bare and leaned
on `buffered` first-polling its futures in input order. That happens to
hold today, but it is a futures-util implementation detail and it is
exactly the property `hold_back_debug` exists to stop depending on.

It also leaked: a batch window the fallback drops had already announced
the chunks whose answers are discarded, so a mid-run downgrade printed
`POST .../patches/batch` lines for requests the one-at-a-time loop never
made. Held back, a dropped chunk's lines are dropped with it.

`proxy_batch_post` logged its line before awaiting the proxy's in-flight
permit, so on the public proxy the line no longer sat next to the request
it announces. Log after the permit.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* perf(crawl): cap the walk pool at what an I/O-bound walk can use

`walk_threads` took the machine's parallelism under the descriptor
budget, with no ceiling: on a 96-core runner at the usual Linux soft
limit of 1024 the budget is 480, so the pool was 96 threads — 96 OS
threads and 768 MiB of reserved 8 MiB stacks spun up on the first walk of
every invocation, plus one uncapped package.json buffer each, to walk a
tree that is descriptor- and page-cache-bound and flat well before 16
threads.

Cap it at 16. The descriptor budget keeps its own function and its own
assertions — above the tight limit it is never the binding term, and
below it the pool is still one thread — so what the limit buys is
unchanged and only the machine-sized tail is bounded.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* docs(changelog): name what the concurrency work costs the service

Two request-count changes and one determinism fix that live only in code
comments today, where an operator would meet them in a WAF log or a quota
dashboard first: a vendored run's speculative download grants (74 where
the serial loop made 71 on depscan, and turned off entirely by
SOCKET_API_CONCURRENCY=1), a mid-run token revocation costing the
authenticated batch endpoint a window of requests instead of one, and
`vex`'s API-fallback note no longer racing over which refusal it quotes.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* fix(vex): pace the record fetch with the documented knob

`vex` and `scan --vex` fetch patch records ten at a time against the
patch API, sized by a bare constant. That made it the one patch-API
window `SOCKET_API_CONCURRENCY` could not reach, while the README says
the knob paces `scan`'s patch-API requests — and `scan --vex` goes
straight through here. An operator behind something that caps in-flight
requests per client set the knob to 1 and still had ten views in flight.

Size it from `api_concurrency` under the window's own ceiling, like the
other windows. The default drops from ten to eight authenticated and four
on the public proxy, whose one shared server-side limit is why that
number is four everywhere else.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* perf(redirect): bound the Cargo.lock block search to the next block

lock_block_end searched `\n[metadata]`, `\n[[patch.unused]]` and
`\n[patch` from the block body to EOF for every block of every dep. Those
tables are absent from v3/v4 locks, so each call scanned the rest of the
file and the hosted cargo rewriter went quadratic (24-28 s of CPU on a
1.8k-block lock with 58 redirected deps).

The trailing markers are now searched only up to the next `[[package]]`.
Each marker's only newline is its first byte, so a hit that starts before
the next block also ends before it: the bounded minimum equals the
unbounded one. The old function stays as a test oracle, checked at every
char offset of randomized v1/v3/v4 (and CRLF) locks and at every block of
a 1.2k-block lock.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* perf(build): use the aarch64 SHA-256 instructions

sha…