Experiment/4a unit of return - #50

… controls

Add experiments/4A_unit_of_return/README.md: frozen 4-arm protocol (top-k / whole-doc / oracle / closed book) plus the confound controls for agent-based apparatus derived from E7/E11/E17/E25 and the 2026-09-25 port incident. Record the design in AGENT_DIARY and the experiments registry. No run yet.

BASELINE.md (freeze + append-only Run + system snapshot), RESEARCH.md (Mem2ActBench, LLM-judge noise budget, chunking) and README review corrections + applied deltas. Key delta: the closed-book/oracle 4-arm is prior art, so the claim is repositioned to replication on a public code+prose corpus plus plumbing-level partial arrival; judged arms require >=10 trials with majority and kappa.

Line-by-line check against arXiv:2601.19935v1 and arXiv:2606.13685v1. Fixes: Mem2ActBench failure taxonomy is 5 types (not 4); Coin Flip recommendation is 10-20 trials majority with t=0 as an additional measure (not '>=10 at t=0'). Add [verified: Table/Section] markers and mark the plumbing-vs-model-side differentiation as [agent inference], not a source claim.

Root cause of the 2026-09-26 loss: the E7/E11 frozen symptom list lived in %TEMP% and was never committed, so verbatim regression became impossible. Recovery: the handout content was found in opencode.db (write rid=75331, 2026-09-22; cross-checked against read rid=75864) and committed to experiments/4A_unit_of_return/frozen/ (sha256 a6f719df100ec0e3256a91e1d78469a1dc0ba3a9595aa266046199fe9f68de0a). New rule: frozen inputs live only in the repo; tests/test_frozen_inputs_tracked.py fails if any experiments/**/frozen file is untracked.

…controls

Close two gaps from review. (1) Judge-confound architecture: judge sees only the arm output bundle, identical reference across arms, opaque arm tokens, no self-grading, plus a leak control (same artifact under two labels must yield the same verdict). (2) G6 becomes executable: scripts/frozen_overlap_check.py fails on exact duplicates or >=3 shared content tokens vs any used list (stopwords filtered); verified with a negative control. Recorded in 4A README and in the global AGENTS rules (mini/compressed/full).

…production)

Blind mapper via opencode (longcat-2.0 x5, qwen3.7-plus x3, deepseek-v4.1-flash x3), isolated dir, MCP off, --pure, tools denied. Valid 5/11; in all valid runs #16 -> NONE (arrival symptom unreachable); the only failure is the same known false positive (#11 Safari/CSS -> a-generated-document-is-unverified), concentrated in deepseek (0/3 valid). Qualitative reproduction of E7 confirmed; numeric rate NOT reproduced (5/11 vs ~10/11) because --variant was not pinned. Raw runs and manifest under results/recovered_e7/.

Replace ad-hoc PowerShell+Tee with scripts/f4_blind_run.py (subprocess argv-list, UTF-8, ANSI-strip); --variant is required so an unpinned run fails loudly. Rerun with --variant high: valid 10/11 (longcat 4/5, qwen 3/3, deepseek 3/3), controls 6/6 in all valid, #16 -> NONE everywhere. The earlier 'deepseek 0/3' was an unpinned-budget confound, not a model property. Pitfall #19 added to the registry.

Explicit per-run check: #11 (must-NONE) = NONE in 10/11 runs; the only non-NONE is exactly the run classified INVALID (longcat run5), so #11 passed in every valid run and is model-independent (~9% FP, the catalogue's neighbour-domain boundary, same class as the published arrival_fp). Fresh held-out candidates for F4b drafted by a context-free subagent from our ledgers (8 items, sources recorded); frozen_overlap_check.py -> OVERLAP: PASS.

… review

v1 rejected (5/8 reindex skew, semantic twin of arrival symptom and of used #2). v2 replaced memory-retraction twin (used #13). Final 8 items span distinct families (cache, security, concurrency, docs, CI, observability, query semantics, destructive tool). Token G6 = PASS; semantic review recorded because the token checker cannot see twins.

The G6 overlap checker counted a trailing numbered 'next steps' section as symptoms (f4b reported 11 items instead of 8). It now stops at the first non-item line after the block starts. Verified: f4b=8, recovered handout=16, both PASS.

…rtifact

Five design attacks with defenses, distinct from the numbers-level Red Team (F6): dead/wrong gate sensor (G6 counter bug), sampling concentration (v1 reindex skew), semantic twin contamination (token checker blind), unpinned budget confound, broken launch harness. All five closed.

…ies)

Fetched from Tirthahq/crystal-memory@master (Apache-2.0, Spanda Works). Material change vs E7: 12 -> 15 entries and a new arrival index as front door; four new entries. Stored as a frozen experiment input per the new rule.

Per review: do not pick a single index (would break comparability to E7 and invite target-shopping). F4b runs two conditions (symptom + arrival) over the same 14 items, models and variant. Timeline recorded: no F4b run against either index yet, so the decision is pre-run. Twin check of the 8 fresh items vs the 25 arrival phrases: 0 flags. Freeze references the catalogue sha256 5429c4ff.

Two conditions over the same 14 items (8 fresh held-out symptoms + 6 controls). Handout sha256 and catalogue sha256 recorded in manifest.json. Overlap gate scoped to previously-used lists (sibling condition excluded), both PASS.

One opencode run with --variant high on the 14-item handout takes ~5 min, so the 300s cap killed calls (TimeoutExpired). Timeout is now configurable (default 900) and a timed-out run is recorded instead of raising. scripts/f4b_run_all.py runs both conditions x 3 models sequentially, launched detached with a log.

Item #14 ('email spacing') was a semantic twin of a catalogue arrival
sentence, so it produced a guaranteed false positive and invalidated
every F4b run. Replaced with a genuine out-of-domain Docker item
(base image tag removed); manifest sha256 updated.

- scripts/f4b_validate.py: structural validator with --selftest
  negative control (fails on generic/error-marker outputs)
- scripts/f4b_run_all.py: 6 parallel streams (2 conditions x 3 models)
- README Red Team A6 documents the false-control finding
- archive the invalid run under results/f4b_invalid_control14

22 runs (2 conditions x 3 models x runs), variant pinned high, all rc=0.
Validator clean 13/22; arrival 10/11, symptom 3/11.

Key finding: the arrival-index condition generalizes (must-hit 33/33,
must-NONE 32/33). The symptom-index drop is isolated to item #3, a
morphological twin of the catalogue's own arrival sentence
('timing out / retry' vs 'times out / try') -> trivially matched under
arrival, not under the symptom key. G6 token-overlap gate cannot see it
(table rows are not numbered; morphology not normalized) - OPEN for F6.

- scripts/f4b_aggregate.py: reproducible per-item/per-condition summary
- results/f4b/RESULTS.md + manifest.json + raw outputs
- runner now stores handout as a repo-relative path (F0 privacy class)

Six attacks, none FATAL. Verdict: arrival-index generalization
DEFENDED (10/11, must-hit 33/33); symptom-index claim OPEN/lowered
to 'direction' - the drop is one contaminated item (#3, a
morphological twin of the catalogue arrival sentence) plus no
closed-book arm and control-only validation.

AGENT_DIARY: F4b numbers + root cause for item #3.
KNOWN_ISSUES: G6 novelty gate is blind to table-row and
morphological twins of index phrases (Open).

F4b exposed that frozen_overlap_check.py v1 was blind to probes
that paraphrase a catalogue arrival phrase (item #3: 'timing out /
retry' vs 'times out / try'): it only saw numbered lines and did no
morphology. v2 compares numbered probes of previous lists AND the
catalogue arrival register, with light stemming; item #3 is now
flagged with zero false positives.

Controls: --selftest (negative twin flagged + positive clean) and
tests/test_frozen_overlap_check.py pin the behaviour. Docs, diary and
KNOWN_ISSUES updated (G6 gap -> Fixed).

scripts/f5_retrieve_arms.py serves arms A (top-k chunks), B (whole
top-1 doc), C (oracle file), D (closed book) over the live index
read-only, and emits objective metrics + context bundles for the
judged reader arm. Population split is data set before results.

Pilot on 12 in-sample benchmark2 tasks (calibration, NOT an F5
result): A hit@1 8.3%, hit@3 41.7%; B top1-is-gold 8.3%; B context
27x larger than A. Phase Zero, gates and the fresh-set requirement
are recorded in f5/README.md.

Fresh 16-query set (8 code + 8 prose) authored by a context-clean
subagent, validated (files exist, evidence_span literal), passes the
G6 gate (max token overlap with benchmark2 = 2) and is frozen with
sha256 e048aa12.

Objective result (n=16): A hit@1 5/16, B top-1-doc-is-gold 5/16 (whole
doc gives no top-1 gain at 4x context), strong code>prose split
(4/8 vs 1/8 hit@1). Judged reader arm not yet run - direction, not
result.

scripts/f5_judged_run.py implements the blind judged reader arm
(reader context per arm, different judge model, opaque tokens,
majority, leak-check). Dry-run and direct calls validated.

Measured blocker: a realistic reader call via the opencode CLI is
~200s (trivial 8.4s), so the full design (n=16 x 4 arms x 10-20
trials) is hours-to-days and cannot run as designed. Options (raw API
= HALT, reduced pilot, or defer to F0c) are recorded in
f5/JUDGED_PLAN.md. AGENT_DIARY notes the incident: a broad
'opencode' process filter killed the host session.

A reader/judge call that silently falls back to another model (observed
'google/gemma-4-e2b' on a real call) would corrupt the experiment
(channel #1: model identity). _run now marks such outputs with
[MODEL-MISMATCH:...] / [ERROR] / [TIMEOUT] and the run records them as
'invalid' instead of grading them. Dry-run and ruff clean.

16 queries x 4 arms x 5 trials (80/arm). Reader longcat, blind judge
qwen. Overall: A 16.3%, B 30.0%, C 95.0%, D 0.0%. By population:
code B 47.5% >> A 7.5%; prose A 25% > B 12.5% (weak). Majority and
judge-stability (12.5% non-unanimous) recorded. Raw outputs normalized
for personal paths inherited from repo sources (F0c tail).

F5 judged pilot: A 16.3%, B 30.0%, C 95.0%, D 0.0%;
unit of return affects the reader, not gold retrieval.

Full judged run: 16 queries x 4 arms x 10 trials (160/arm).
A 16.3%, B 34.4%, C 97.5%, D 0.0%. code B 50% >> A 6.3%;
prose A 26% > B 19% (weak). Majority and judge stability (9.4%
non-unanimous) recorded. MANIFEST.json hashes every 4A artifact.
Raw normalized for personal paths (F0c tail).

Replace C:\\Users\\misha\\ (and Git-Bash /c/Users/misha/) with <user> in 48 files
under experiments/scripts/tests. 0 username leaks remain; full suite
1869 passed / 5 skipped.

…pth)

opencode reveals the --file name to the model. Arm/query no longer
appear in ctx/cand filenames, so neither reader nor judge can see them.
Verified benign (arm labels are opaque to the models) but hardened anyway.

13 attacks over F4b/F5: 11 DEFENDED, 2 OPEN non-FATAL, 0 FATAL. INVENTORY.json lists all 1226 experiment artifacts.

The reader guard was startswith("["), which flagged a valid answer as
invalid whenever the agent led with a bracketed heading (observed live:
the agent's own "[ИТОГ]" summary). Observed once in 24 units.

The judge had no guard at all: a failed call fell through to the keyword
fallback in _parse_verdict and was recorded as a verdict instead of
being rejected. Both now check FAIL_MARKERS explicitly.

Also adds arm S: the retrieved chunk plus the file's imports and the
signatures of functions it calls. Callee-to-signature resolution was
measured at 11.3% (274/2432) with a repo-wide index, 1.0% file-local,
so S is imports + a small number of local callee signatures.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

…utside scope

…D (chunked wins hit rate, graph wins tokens)

… (5/5 pass)

…sts pass

… red team clean

…0% saves 29% tokens

…NodeRAG

strict >50% majority (B prose 1/8, B ALL 5/16; F5S-13/B 5/5 tie counted out), guard line fixed (0 invalid in trials=10, one in t5 pilot F5S-03/B), changelog entries in root + en/ru/zh.

…-return

# Conflicts:
#	AGENT_DIARY.md
#	KNOWN_ISSUES.md
#	experiments/4A_unit_of_return/README.md

F821 cli.py unreachable allowlist line; BLE001 staleness.py narrowed except; F541 store.py plain string; F401 test_redact.py dropped import