Skip to content

fix(messaging): reliable batching, bindings and delivery outcomes (#2356) - #2357

Merged
jerryfane merged 4 commits into
mainfrom
fix/2356-reliable-messaging
Oct 24, 2026
Merged

jerryfane merged 4 commits into
mainfrom
fix/2356-reliable-messaging

Conversation

@jerryfane

@jerryfane jerryfane commented Oct 24, 2026 •

Copy link
Copy Markdown
Collaborator

Assignment and scope

Implements the owner-approved messaging work in #2356. Implementing seat: gitmoot (OpenAI Astra), reporting to Jarvis. No public binary release was made. Coordinator clearance 355822 released the restart/resource hold subject to the normal merge and deployment gates.

Merged and deployed: reviewed head 353cd92af00e561da8a5e149e235d5e7ccc68c90 entered the required GitHub merge queue after a fresh supported /gitmoot merge decision; merged main is 574d0705976d280fb5ad6fcacbfef6b5ddb27514, with an identical source tree. Final-head CI, independent review, all 19 local race shards (4,262 tests), normal suite and smoke checks pass. Merge-queue CI and post-merge main CI both passed 27/27 checks. CLI and dashboard services now run the metadata-stamped merged build; documentation is published and live behavior verified.

  • Batch working recipients for 120 seconds from the oldest compatible notification; idle recipients have no batching hold. Recheck every five seconds, release on idle transitions, and preserve unsafe/offline/unknown-state admission rules. Review results use the same peer policy; owner Grams retain their separate policy.
  • Persist verified local agent:NAME identities when adopting seats, refusing missing, duplicate, archived or remote identities rather than guessing from labels.
  • Initialize SQLite connection-local settings on every writable/read-only physical connection. Persist known notification outcomes with bounded bookkeeping retries, never transport retries. Reserve the reply writer before reading its parent, avoiding SQLite read-to-write upgrade failure without replaying an insertion.
  • Store explicit coalesced-batch relationships and project the survivor's real status through message CLI, inbox and dashboard. Invalid historical relationships fail closed as uncertain; status repair never resends mail.
  • Update CLI/skill references, repository and website documentation, and machine-readable documentation. Remove obsolete timing configuration and implementation/wording assertions.

Verification

Passed at earlier heads (not claimed as final-head verification):

  • go generate ./... with git diff --exit-code; go build -p=2 ./...; go vet -p=2 ./... using pinned Go 1.26.4.
  • Initial-head go test -p=2 -timeout=35m ./...: 43 packages passed, four without tests. Final-correction local full run was blocked by the disk guard, as recorded below.
  • npm run build, including generated LLM documentation and broken-link checking.
  • All 28 CI checks passed at initial head f27e49748872cf8b653edcfdd31624170125fe88: run 40088160564, and correction head 3a163374e70befd2f9ea40351d128a2431cc77f4: run 40092588179. Each includes tagged E2E and every race shard. The existing workflow compiles all five race packages with -tags e2e and asserts each complete test list is partitioned exactly once; no tests were excluded.
  • Individual test-check prove runs: 22 red-old/green-new results; 19 existing guards pass both versions because their protected behavior is unchanged. Those guards cover owner-Gram spacing/outcomes, bounded/authorized hooks, route failures, single claims, drain independence, roster access and directive TTL. Two relation-schema fixtures fail on the absent old column; the consumer status regression reaches actual message projection on both versions. Full receipts and the per-test disposition are retained in /root/fleet-tools/state/gitmoot-message-audit-20261024/verification.json.
  • Actual isolated CLI/daemon/dashboard smoke: idle delivery 4.35 seconds; working batch 121.58 seconds from its oldest item despite a younger arrival; stable-name routing after pane replacement; lost-receipt batch and its collapsed member both rendered uncertain, with no additional transport admission. Real reply CLI waited 0.55 seconds for a competing writer and stored exactly one reply. Transport used an isolated Herdr registry/OMP socket simulator, not production seats. Saved receipts and screenshot are in the same durable directory.
  • Binary rollback compatibility: the saved 996f40d5 binary reads and writes a new-head isolated database, and the new binary reads its reply. Both messages and the additive column remain. No database restore or production restart was used. Receipt: rollback-compatibility.json in the durable audit directory.
  • At 3a163374, review corrections pass generation/build/vet, focused seat/owner/timing tests and website build. Actual seat-add CLI stayed blocked beyond 12 seconds before the registry timeout fix; afterward it returned in 10.04 seconds without saving a role.

Latest correction restores immediate owner-review health classification through wakeOutboxRowHold, preserving the separate five-minute hold for ordinary owner alerts. Intermediate ca008b9f failed CI on a nonexistent test constant; 353cd92a uses the existing replyWakeAttemptedUnknownAfter and passes CI. The new regression is proven against 3a163374: new code passes; old code compiles but reports pending=0 held=2 instead of one urgent review pending and one ordinary alert held. Receipt: /root/.local/state/test-check/receipts/20261024T124532Z-712b4b9d431f15d3fbd20b4b3647d356/receipt.json. This is the 42nd proof, separate from the 41 earlier outcomes.

Complete local race coverage passed after authorized recovery. At the unchanged final head, all 19 sequential -race -tags e2e partitions passed: CLI 2,335 tests, workflow 1,209, DB 386, daemon 206, pipeline 126 — 4,262 total. Each complete list was partitioned exactly once and matched observed test names, without exclusions. Minimum available space across successful shards was 25,505,763,328 bytes, above the unchanged 5% floor. Exact-head compiled CLI binary was reused; successful normal checks were not rerun. Earlier aggregate timeouts, disk failures and the interrupted shard remain saved as failures, not relabelled passes. Results: /root/fleet-tools/state/gitmoot-message-audit-20261024/final-race-353cd92a/results.json. Janitor's heavy-check window is released (355800); no shared cache or protected tree was cleared by this seat.

Operational work and holds

Outside this source diff, the shared peer-channel rule is installed and acknowledged by Enyo/Enyo-auth. Verified local bindings were migrated with saved rollback evidence. Nuli-research's original live session identity was verified before restoring its missing agent name. Coordinator decision 354944 authorized disabling exactly 24 delivery routes for dxbre-paper, numbra-posthog-review, entmoot-app and entmoot-web; registrations, permissions, histories and message rows were preserved. Mac-studio remains unbound. Workspace-janitor was initially left unresolved; Jarvis later explicitly registered a fresh owner-authorized agent:janitor seat (355640), superseding that status without replaying old mail.

The coordinator's fresh message reply 354948 SQLITE_BUSY failure is preserved without replay. An isolated regression reproduced the same reply-path error even after connection initialization was fixed; write-first reservation passes that regression. This does not establish the historical production lock holder or the October 16 failure's exact trigger.

Deployment and review

  • Exact-head level-3 approval: local-review-review-router-18e17749e1b2dd15-1, effective model devin/swe-2, no fix target. Fresh supported merge decision: comment 6385826781. Direct merge was correctly refused by the repository queue rule; GitHub's required queue was used without --admin, manual green status or bypass.
  • Queue CI: 40105805441; post-merge main CI: 40106856441. Both passed all 27 checks at merged 574d0705.
  • Restart at 2026-10-24T15:29:47Z followed two fresh empty running/queued non-session snapshots. An earlier queued health-delivery job was allowed to finish naturally; no work was cancelled. Private DB/config/binary backups were verified, including SQLite quick_check. Both running executable hashes match the merged build.
  • Live proof: CLI and rendered dashboard now show existing coalesced message 355531 as submitted, not resolved, following delivered survivor 18943. Member/survivor attempt counts are unchanged, so no replay occurred. The actual workflow API responds successfully. Doctor passes from a named checkout; existing auth-unvalidated, registration and stale-route advisories remain. The first doctor call failed only because the deployment checkout was detached; that receipt is preserved.
  • Docs: merged-source website build passed, 223 old-site files were byte-verified in backup, and all 207 published build files match the new output. The public CLI reference exposes the new timing/binding/status contract. A direct Python raw-byte probe received HTTP403; the supported reader successfully verified the published page, so public raw-byte equality is not claimed.
  • Durable deployment receipts, rollback copies and the live dashboard image are under /root/fleet-tools/state/gitmoot-message-audit-20261024/. Rollback uses the saved old executables; never restore the database snapshot over newer message writes.
  • Route rollback is limited to restoring the 24 saved enabled flags; do not rename or redirect retired identities.

Refs #2356. Authorized deployment and live verification are complete; remaining Janitor integration proposal work is separate.

@jerryfane

Copy link
Copy Markdown
Collaborator Author

Independent review dispatched for exact head f27e497: job local-review-review-router-18e17234b19e8e56-1, reviewer gm-review-opus; router-selected effective model devin/swe-2 (not the registered default). Explicit --no-fix-target; no implementation job was dispatched. Implementing seat record: session-implement-gitmoot-18e1722e9c6d5c87. Review level is level3_required at this head. Separate holds remain: CLI race gate after an aggregate 35-minute timeout, CI completion, and coordinator prohibition on production restart. Full isolated race run is in progress; no tests excluded and no timeout increase.

@jerryfane

Copy link
Copy Markdown
Collaborator Author

Agent: gm-review-opus
Runtime: omp
Job: local-review-review-router-18e17234b19e8e56-1

Decision: approved
Summary: APPROVED at f27e497. Reliable-messaging PR is correct on all changed production paths: durable coalesced_into links replace text-claimed coalescing with survivor-following status projection (uncertain survivors, missing/invalid/cyclic links all render 'uncertain', verified by tests); bounded BUSY-only bookkeeping retries never replay transport; per-connection busy_timeout/synchronous pragmas fix the connection-replacement gap and are proven by lock-hold tests; reply writer reservation fixes read-to-write upgrade contention; state-aware oldest-item batching is consistent between daemon drain and turn hooks; recipient admission is fail-closed (unknown/unbound/non-idle-non-working defer; unregistered panes rejected at seat add). Owner Gram keeps its separate 5m hold + review-result exemption. No blocking findings; all findings are P3 robustness/docs notes. Limits: -race unreachable from read-only seat (cgo denied); go generate not run (read-only worktree); full suite not rerun per dispatch (parent owns race gate).

Findings

  • Readiness probe moved before the unroutable-skip and the due-check: every pending group now costs a herdr/provider probe per 5s tick even while held or unroutable (previously only claimed candidates probed, and unroutable groups never did). Probe results taken while still inside the hold are discarded, and a probe error on an unroutable group now aborts the whole drain, widening failure coupling. Low correctness risk; extra load and coupling are new. (P3)
    • location: internal/cli/reply_wake_outbox.go:261-283
  • Recipients with a defer reason (offline/unbound/unsafe) now classify pending immediately (rowHold=0), and the loop drains every 5s, so 'reply wake outbox drain unhealthy' prints every ~5s for as long as a bound seat is merely offline with pending mail; previously it surfaced only after the 5m hold at ~30s cadence. Deliberate-looking but confirm the amplified, immediate unhealthy logging is intended. (P3)
    • location: internal/cli/reply_wake_outbox.go:413-443 + internal/cli/reply_wake_drain_loop.go:33-38
  • StableAgentBinding runs 'herdr agent list' on bare context.Background(); every neighbouring external call in this file is bounded by orgSeatExternalTimeout (10s). A hung herdr wedges 'org seat add --pane' indefinitely. (P3)
    • location: internal/cli/org.go:309-315
  • Two bare 5*time.Minute literals replace the removed config.DefaultWakeCoalesceHold for the retained Owner-Gram policy; no named constant ties them together. (P3)
    • location: internal/cli/reply_wake_outbox.go:197,413
  • Stale text: test comment still describes the removed 2026-10-08 review-result hold exemption; website org paragraph still says seat add 'canonicalizes new bindings to ids' (now agent:NAME); the new peer_messaging brief field is undocumented. (P3)
    • location: internal/cli/reply_wake_review_result_test.go:15; website/docs/reference/cli.md:~1680; skills/gitmoot/references/CLI.md org brief section
  • Anchoring asymmetry: classify's held accounting anchors on the group's oldest row, while authorizedMessageEventBatch anchors the hold on the oldest routable row. For a group whose oldest row is unroutable but a newer compatible row is still inside the hold, health counts pending (unhealthy) while delivery is correctly held; resolves when the hold elapses. Narrow edge. (P3)
    • location: internal/cli/message_event_source.go:147-156 vs internal/cli/reply_wake_outbox.go:229,419-423

Tests Run

  • CGO_ENABLED=0 go build -buildvcs=false ./... — pass (26.8s)
  • CGO_ENABLED=0 go vet ./... — pass
  • CGO_ENABLED=0 go test -count=1 ./internal/db/ — ok, 226.6s (full package incl. new tests)
  • CGO_ENABLED=0 go test -count=1 -v -run 'TestCoalesced|TestNotificationOutcomeSurvives|TestReadOnlyMessageLookup|TestDurableWriteBudget|TestBookkeepingCounter|TestTransactionalWakeWrites|TestDeliveredFinish' ./internal/db/ — all 11 new tests PASS (incl. TestCoalescedLinkMigrationPreservesHistoricalOutcomes 8.07s, TestCoalescedMessageRejectsBrokenAndCyclicLinks)
  • CGO_ENABLED=0 go test -count=1 -v -run 'TestMessageNotificationTimingFollowsRecipientState|TestWorkingTurnHookWaitsButStopReleasesMail|TestReviewResultUsesSameWorkingHoldAsMessages|TestReplyWakeOutboxDrainLoopFlushesHeldTailWithZeroEnabledRepos|TestReplyWakeDrainLoopDeliversNewReviewResultOnNextTick|TestReplyWakeOutboxInertHealthLogsOnceAcrossDrainLoopTicks|TestOrgSeatAddRejectsUnverifiedRegisteredIdentity' ./internal/cli/ — all PASS incl. 5 unverified-identity subtests
  • CGO_ENABLED=0 go test -count=1 -run 'Message|Wake|Org|Gram|Notification|Pending|Brief|Deliver' -timeout 15m ./internal/cli/ — ok, 69.0s
  • CGO_ENABLED=0 go test -count=1 ./internal/config/ ./internal/cockpit/ — ok
  • Not run: -race (seat denies cgo: read-only sandbox; parent owns race gate), go generate (read-only worktree), full ./internal/cli suite (per dispatch: parent running full gate)

Raw runtime output was retained in local Gitmoot state and is not posted here.

@jerryfane

Copy link
Copy Markdown
Collaborator Author

Review round 1: gm-review-opus (effective devin/swe-2), job local-review-review-router-18e17234b19e8e56-1, APPROVED exact head f27e497 with six P3 notes. Executed build/vet, full db tests, focused and broad affected CLI tests, config/cockpit; race unavailable in its read-only seat. No fixes applied by reviewer. Disposition: (1) working-to-idle release requires continued probing; extra unroutable probes/error coupling acknowledged as non-blocking cost, not claimed fixed; (2) retain truthful pending classification and existing unhealthy logging at the new cadence, rather than suppress errors; (3) fixed registry lookup with existing10s timeout—real CLI remained blocked beyond12s before, returns in10.04s after without saving a role; (4) one named owner hold constant; (5) corrected stale timing/binding text and documented peer_messaging; (6) cited anchor asymmetry is not supported: groupCreated is populated only inside the deliverable-route branch, so an unroutable row cannot anchor it. Corrections are undergoing complete checks before a new exact-head review; current approval will not cover the changed head. Durable full verdict and dispositions saved in the epic audit directory. Production restart hold unchanged.

@jerryfane

Copy link
Copy Markdown
Collaborator Author

Final-head review requested at 3a16337: local-review-review-router-18e175fe1c02e161-1, gm-review-opus with router-selected devin/swe-2, explicit --no-fix-target. Prior approval f27e497 is only the delta baseline, not approval of this head. Same persistent gitmoot Astra seat implemented the corrections. Review level remains level3_required. Resource constraints and all prior finding dispositions were supplied; reviewer must not launch a full suite or bypass guards. CI is running independently; local complete checks and coordinator restart clearance remain open.

@jerryfane

Copy link
Copy Markdown
Collaborator Author

Agent: gm-review-opus
Runtime: omp
Job: local-review-review-router-18e175fe1c02e161-1

Decision: approved
Summary: APPROVED at exact head 3a16337 (delta-only vs prior approved head f27e497; one commit, 5 files, +18/-8). Three of six named findings are resolved by this delta and verified in code: f3 (StableAgentBinding now runs under a 10s orgSeatExternalTimeout context; the herdr invocation is exec.CommandContext-bound, consistent with the operator's ~10.04s smoke), f4 (ownerGramCoalescingWindow constant names both owner-hold sites), f5 (test comment corrected; website paragraph now says 'stores agent:NAME'; peer_messaging documented in skills + website trees and verified emitted live by org brief --role owner --json on a throwaway home). f1, f2, f6 remain open as P3-reported (none blocks merge per note 129657); f6's mechanism refined: groupCreated is populated only inside the pendingWakeDeliverable branch, so classify anchors on the oldest DELIVERABLE row. One NEW P3 introduced by this delta: the constant substitution at reply_wake_outbox.go:414 dropped wakeOutboxRowHold's isReviewResultWake zero-hold on the health path only while deliverOwnerGram (owner_gram.go:90) still applies it — a pending owner review-result row under 5m now counts held instead of pending for up to 5m, contradicting the in-file comment 'including immediate review results'. Narrow, self-resolving health-fidelity gap; delivery itself unchanged and correct. gh unauthenticated — PR comments unreadable; dispositions evaluated against code. Full ./... suite NOT run per dispatch disk guard (95%); focused build/vet/tests + light binary smoke ran instead.
Severity: P3

Findings

  • Still open: readiness probe still precedes the unroutable-skip and due-check; every pending group costs a probe per 5s tick and a probe error aborts the drain (P3)
  • Still open: deferred recipients classify pending immediately, so 'reply wake outbox drain unhealthy' logs every ~5s while a bound seat is merely offline (P3)
    • continues_uid: fix(messaging): reliable batching, bindings and delivery outcomes (#2356) #2357-f2
    • detail: Unchanged at 3a16337. Mechanism confirmed end-to-end: status.reason!='' sets rowHold=0 -> health.pending++ -> classify returns error -> runReplyWakeOutboxDrainOnce writes 'drain unhealthy' every tick; DeferMessageNotifications only stamps last_error (db/message_delivery.go:19), rows stay pending. Deliberate-looking; still worth confirming the amplified cadence is intended.
    • file: internal/cli/reply_wake_outbox.go
    • line: 436-438,510-513
    • state: answered
  • Resolved: StableAgentBinding now bounded by orgSeatExternalTimeout (P3)
    • continues_uid: fix(messaging): reliable batching, bindings and delivery outcomes (#2356) #2357-f3
    • detail: RESOLVED at 3a16337: bindingCtx = context.WithTimeout(ctx, orgSeatExternalTimeout=10s) wraps the herdr call; internal/cli/herdr.go:25 uses exec.CommandContext so a hung herdr is killed at the bound, matching neighbouring bounded calls. Consistent with the operator's ~10.04s seat-add smoke.
    • file: internal/cli/org.go
    • line: 309-315
    • state: answered
  • Resolved: ownerGramCoalescingWindow constant names both 5m sites (P3)
  • Resolved: stale comment fixed, website paragraph corrected, peer_messaging documented in both trees (P3)
    • continues_uid: fix(messaging): reliable batching, bindings and delivery outcomes (#2356) #2357-f5
    • detail: RESOLVED at 3a16337: test comment now describes working-hold/idle-release; website/docs/reference/cli.md:~1684 now says 'validates the registered local agent and stores agent:NAME'; peer_messaging documented in skills/gitmoot/references/CLI.md:2173 and website cli.md:1707. Verified live: built ./cmd/gitmoot and org brief --role owner --json on a throwaway home prints a populated peer_messaging field.
    • file: internal/cli/reply_wake_review_result_test.go
    • line: 15
    • state: answered
  • Still open: health/delivery hold anchors still diverge when a group's oldest row is unroutable (P3)
    • continues_uid: fix(messaging): reliable batching, bindings and delivery outcomes (#2356) #2357-f6
    • detail: Still present at 3a16337, with one refinement to the recorded mechanism: groupCreated is populated ONLY for deliverable routes (:423 inside pendingWakeDeliverable), so classify anchors on the group's oldest DELIVERABLE row, while delivery anchors startedAt on the group's oldest row (:230) and authorizedMessageEventBatch on the oldest routable row (message_event_source.go:147-156). The asymmetry class stands in both directions; it resolves when the hold elapses. P3 stays reported.
    • file: internal/cli/reply_wake_outbox.go
    • line: 420-424
    • state: answered
  • New in this delta: owner review-result rows lost their immediate-delivery exemption on the health path (P3)
    • detail: f27e497 ran rowHold := wakeOutboxRowHold(obligation, 5*time.Minute); 3a16337 replaced it with the flat ownerGramCoalescingWindow. wakeOutboxRowHold (:89-94, comment still reads 'including immediate review results') returns 0 for review-result rows and is still applied by the DELIVERY path in owner_gram.go:90 — so an owner review-result row is due immediately per delivery policy but classifyWakeOutboxObligations now counts it held for up to 5m when it sits pending (claim lost to a concurrent drainer, >ownerGramMaxItems backlog, or a store without ownerGramStore). Effect: health reports held instead of pending for up to 5 minutes, delaying the 'drain unhealthy' signal for the row class policy marks most urgent. Narrow, self-resolving at 5m; no delivery-path impact. One-token fix: wakeOutboxRowHold(obligation, ownerGramCoalescingWindow).
    • file: internal/cli/reply_wake_outbox.go
    • line: 414

Tests Run

  • git diff f27e497..3a16337 --stat: 5 files, +18/-8 (delta scope confirmed)
  • go build -buildvcs=false ./internal/cli/ (CGO_ENABLED=0, seat-owned GOCACHE under $TMPDIR): PASS
  • go vet ./internal/cli/: clean, no output
  • go test -count=1 -timeout 600s -run 'TestReviewResult|OwnerGram|ReplyWake|WakeOutbox|StaleNotification|MessageEvent|OrgSeat' ./internal/cli/: ok 16.6s — 86 tests PASS / 0 FAIL
  • go test -count=1 -timeout 300s -run 'TestOrg|OrgSeat|StableAgentBinding|SeatAdd' ./internal/cli/: ok 18.0s — 93 tests PASS / 0 FAIL
  • built ./cmd/gitmoot to $TMPDIR; gitmoot org brief --role owner --json --home <tmp> printed a populated peer_messaging field (f5 verified live); gitmoot org init --home <tmp> failed 'fork/exec herdr: permission denied' — expected read-only-seat sandbox, unrelated to the change
  • source diff at both heads: f27e497 'rowHold := wakeOutboxRowHold(obligation, 5*time.Minute)' vs 3a16337 'rowHold := ownerGramCoalescingWindow'; semantic delta for isReviewResultWake owner rows confirmed by reading wakeOutboxRowHold :89-94 and deliverOwnerGram owner_gram.go:90
  • NOT RUN: full go test ./... — dispatch's disk guard (root fs at 95%) forbids another full suite/heavy E2E; parent owns complete CI/local coverage per requester instructions

Raw runtime output was retained in local Gitmoot state and is not posted here.

@jerryfane

Copy link
Copy Markdown
Collaborator Author

Review round2 APPROVED 3a16337, job local-review-review-router-18e175fe1c02e161-1, with focused build/vet/tests and a live org-brief smoke. It also correctly caught a NEW P3 caused by my constant cleanup: health lost the owner review-result zero-hold exemption, although delivery remained immediate. I have restored wakeOutboxRowHold(obligation, ownerGramCoalescingWindow) locally and added a regression comparing urgent-review health/delivery against an ordinary held owner alert. These two local changes are NOT yet verified or published: disk-consuming verification is paused under coordinator355339. This review therefore is not clearance to merge pending corrections. Also correcting my round1 f6 disposition: the original claim about unroutable rows anchoring health was wrong, but the reviewer refined a residual delivery-vs-health anchor difference; I acknowledge that narrow P3 rather than claiming the whole asymmetry class absent. Existing readiness-probe and logging-cadence P3 costs remain explicit. No production restart or gate waiver.

@jerryfane

Copy link
Copy Markdown
Collaborator Author

Exact-head review requested for 353cd92: job local-review-review-router-18e17749e1b2dd15-1; gm-review-opus routes to devin/swe-2, independent of implementing gitmoot/Astra. --no-fix-target; this seat owns findings. Review level remains L3 at this head. Delta restores the immediate owner-review health exemption and adds a mixed owner-review/ordinary-alert regression. Intermediate ca008 failed CI on an undefined test constant; corrected to existing replyWakeAttemptedUnknownAfter. That failure remains recorded, not waived. The ca008 queued review was cancelled as superseded. Coordinator355339/355358 prohibit disk-consuming local checks until measured headroom returns; final reviewer is restricted to source inspection and remote CI evidence. Final-head CI, local regression proof against 3a16337, and full local verification are not yet passes. Earlier 3a CI and approval cover only 3a. PR/epic and durable records updated; restart hold unchanged.

@jerryfane

Copy link
Copy Markdown
Collaborator Author

Final head353cd92af00e561da8a5e149e235d5e7ccc68c90: new owner-health regression is now PROVEN red-old/green-new via test-check prove against3a163374. TestOwnerReviewResultHealthMatchesImmediateDelivery passes new code (0.51s); old code fails its behavior assertion with pending=0 held=2 instead of one pending urgent review and one held ordinary alert. Both versions compile; build_error_suspected=false. Receipt /root/.local/state/test-check/receipts/20261024T124532Z-712b4b9d431f15d3fbd20b4b3647d356/receipt.json. Owner-authorized Go cache recovery restored headroom; fresh measurement25,939,750,912 bytes free. Local checks resumed sequentially under unchanged5% floor with live stop-on-low-margin monitoring. No further Phobos cleanup requested (message355412); protected trees untouched. Generation/build/vet/full normal suite now running one command at a time. Remaining complete race coverage, final review/CI and restart gate are not waived.

@jerryfane

Copy link
Copy Markdown
Collaborator Author

Agent: gm-review-opus
Runtime: omp
Job: local-review-review-router-18e17749e1b2dd15-1

Decision: approved
Summary: Delta review 3a16337..353cd92 (two commits: ca008b9 + 353cd92). The single production change, reply_wake_outbox.go:414, restores wakeOutboxRowHold(obligation, ownerGramCoalescingWindow) on the classify/health path, reinstating the immediate-delivery exemption for owner review-result rows that 3a16337's flat ownerGramCoalescingWindow dropped — exactly the unnamed P3 recorded last round, now resolved and matching the delivery path (owner_gram.go:90, invoked with ownerGramCoalescingWindow at reply_wake_outbox.go:198). Verified by source read that the exemption applies only to owner-gram rows: non-owner branches overwrite rowHold at :425 and :437, so the change cannot leak into seat-wake holds. New regression test TestOwnerReviewResultHealthMatchesImmediateDelivery is semantically sound: insertOwnerEscalation targets 'owner' (=orgChartRootRole, daemon_workflow.go:931), insertAwaitedFactWakeForTest stores WakeOutboxSourceAwaitedFact so SubjectKind=review_verdict makes isReviewResultWake true; health must return pending=1/held=1 WITH error (classify errs whenever pending>0, :510-513 — the assertion correctly expects err), then drain delivers the due review row through deliverOwnerGram (claim+finish, owner_gram.go:121-157) leaving the 5m-held alert held=1, and exactly one accepted receipt 'gram-review-46'. Without the fix the first assertion fails (review row counts held, not pending), so the test pins the defect rather than tautology. ca008's CI failure confirmed via API: it referenced nonexistent replyWakeAttemptTimeout; 353cd92 substitutes the defined constant replyWakeAttemptedUnknownAfter (:29). Remote CI at exact head 353cd92: all 27 check-runs completed success including build/vet/test, e2e(tagged), and every race shard for internal/cli which compiles and runs the new test. Named prior findings re-verified at this head: f1 still open (delivery.ready probe at :262-267 still precedes the unroutable-skip :279-284 and batch hold chec
[truncated]

Findings

  • Still open: readiness probe still precedes the unroutable-skip and due-check; every pending group costs a probe per 5s tick and a probe error aborts the drain (P3)
    • continues_uid: fix(messaging): reliable batching, bindings and delivery outcomes (#2356) #2357-f1
    • detail: Checked at 353cd92 by source read: delivery.ready is invoked at :262-267 per group before the empty-matchingRules skip at :279-284 and before the batch due check inside authorizedMessageEventBatch (:274); a probe error still returns and aborts the drain (:264-266). Unchanged by this delta (delta touched only :414 + test file). P3, non-blocking.
    • file: internal/cli/reply_wake_outbox.go
    • line: 261-284
    • state: answered
  • Still open: deferred recipients classify pending immediately, so 'reply wake outbox drain unhealthy' logs every ~5s while a bound seat is merely offline (P3)
    • continues_uid: fix(messaging): reliable batching, bindings and delivery outcomes (#2356) #2357-f2
    • detail: Checked at 353cd92: status.reason != '' still forces rowHold=0 at :436-438 so the row classifies pending the same tick, and the defer path at :297-306 still only stamps the reason via DeferMessageNotifications, leaving the row pending for the next tick's identical verdict. Mechanism and cadence unchanged by this delta; amplified per-tick unhealthy logging stands as designed or not — still worth an owner confirmation, still P3 non-blocking.
    • file: internal/cli/reply_wake_outbox.go
    • line: 297-306,436-438
    • state: answered
  • Resolved: StableAgentBinding bounded by orgSeatExternalTimeout (P3)
  • Resolved: ownerGramCoalescingWindow names both 5m sites — and the :414 site now correctly routes through wakeOutboxRowHold (P3)
    • continues_uid: fix(messaging): reliable batching, bindings and delivery outcomes (#2356) #2357-f4
    • detail: Re-verified at 353cd92: constant defined :24, used by deliverOwnerGram :198 and classify :414. This delta improves on the prior resolution: :414 is now wakeOutboxRowHold(obligation, ownerGramCoalescingWindow), closing the health/delivery exemption divergence the prior round recorded as a new P3. Stays resolved.
    • file: internal/cli/reply_wake_outbox.go
    • line: 24,198,414
    • state: answered
  • Resolved: stale test comment, website paragraph, and peer_messaging docs (P3)
    • continues_uid: fix(messaging): reliable batching, bindings and delivery outcomes (#2356) #2357-f5
    • detail: Re-verified at 353cd92 by grep: peer_messaging documented in skills/gitmoot/references/CLI.md:2173 and website/docs/reference/cli.md:1707; 'validates the registered local agent' wording present in website cli.md. Delta does not touch these files. Stays resolved.
    • file: internal/cli/reply_wake_review_result_test.go
    • line: 15
    • state: answered
  • Still open: health/delivery hold anchors diverge when a group's oldest row is unroutable (P3)
    • continues_uid: fix(messaging): reliable batching, bindings and delivery outcomes (#2356) #2357-f6
    • detail: Checked at 353cd92: groupCreated is still populated only inside route==pendingWakeDeliverable && !isOwnerWake (:420-424), so classify anchors held accounting on the group's oldest deliverable row while the drain anchors on the group's oldest row (:230) and authorizedMessageEventBatch on the oldest routable row. Asymmetry class unchanged by this delta; self-resolves when the hold elapses. P3 non-blocking.
    • file: internal/cli/reply_wake_outbox.go
    • line: 409-424
    • state: answered

Tests Run

  • git diff 3a16337..353cd92 --stat: 2 files +26/-1 (internal/cli/owner_gram_test.go, internal/cli/reply_wake_outbox.go) — delta scope confirmed
  • git diff + source read of reply_wake_outbox.go:361-514, owner_gram.go:78-158, db/wake_outbox.go:414-481 — restored wakeOutboxRowHold call is the one-token fix recorded last round; exemption scoped to owner-gram rows only; test assertions match classify semantics (err expected when pending>0)
  • git show ca008b9: confirmed it used nonexistent constant replyWakeAttemptTimeout; 353cd92 replaces it with defined replyWakeAttemptedUnknownAfter (reply_wake_outbox.go:29, 30s)
  • curl https://api.github.com/repos/gitmoot/gitmoot/commits/353cd92af00e561da8a5e149e235d5e7ccc68c90/check-runs: 27/27 completed success — build/vet/test, e2e(tagged), race builds + all shards (cli 8/8, pipeline 4/4, db 2/2, workflow 4/4, daemon 1/1), classify diff; new test compiled and passed on the exact head in remote CI
  • curl check-runs for ca008b9: build/vet/test, e2e(tagged), race build internal/cli = failure — consistent with the reported nonexistent-constant compile break, superseded correctly
  • LOCAL go build/vet/test/race NOT executed: paused by operator directive Coordinator355339/355358 for this seat; remote CI at the exact head exercised the changed code instead

Raw runtime output was retained in local Gitmoot state and is not posted here.

@jerryfane

Copy link
Copy Markdown
Collaborator Author

Independent review APPROVED exact head353cd92af00e561da8a5e149e235d5e7ccc68c90: local-review-review-router-18e17749e1b2dd15-1, gm-review-opus on actual devin/swe-2, --no-fix-target; implementing seat gitmoot/Astra. Delta source review against approved3a checked owner-only health exemption, real drain/claim semantics and regression assertions, and verified all27 remote CI check-runs green at353 (plus successful review-level status). Reviewer did NOT run local builds/tests under its resource restriction; prior substantive full-scope review and current remote execution are recorded, not invented local execution. No new findings: inherited P3 probe cost, offline logging cadence and narrow hold-anchor mismatch remain acknowledged; timeout, constant/helper restoration and docs findings resolved. Full verdict/payload/events saved in /root/fleet-tools/state/gitmoot-message-audit-20261024/review-2357-353cd92a.json. Local guarded checks continue sequentially after authorized cache recovery. Merge and restart gates remain closed until remaining checks and explicit restart clearance.

@jerryfane
jerryfane marked this pull request as ready for review October 24, 2026 14:51
@jerryfane

Copy link
Copy Markdown
Collaborator Author

Final local race verification PASS at353cd92af00e561da8a5e149e235d5e7ccc68c90: all19 complete -race -tags e2e partitions ran sequentially, selecting and observing4262 tests (CLI2335, workflow1209, DB386, daemon206, pipeline126). No missing/unexpected tests, no exclusions, no weakened timeouts or5% disk guard. Lowest available space across successful shards25,505,763,328 bytes. Prior interrupted/failed runs remain saved as failures; only incomplete shards resumed after explicit owner authorization. Results and per-shard logs/guard receipts: /root/fleet-tools/state/gitmoot-message-audit-20261024/final-race-353cd92a/results.json. Combined with final-head CI28 green, independent approval, generation/build/vet/full normal suite, proven owner-health regression and actual isolated daemon smoke, verification is complete. PR marked ready. Janitor heavy-check window released355800; explicit production restart clearance requested from Jarvis355801. No merge, deployment or gate waiver performed.

@jerryfane

Copy link
Copy Markdown
Collaborator Author

/gitmoot merge

@jerryfane

Copy link
Copy Markdown
Collaborator Author

Gitmoot did not merge PR #2357.

Cause: repository merge rule: repository ruleset 22536038 requires GitHub's merge queue and refuses Gitmoot's direct merge, including --admin. Remedy: enqueue PR #2357 through GitHub's merge queue for head 353cd92.

Note: merge_gate.auto_merge = false, so no prior gitmoot/merge-gate marker was expected. Its absence does not mean the gate passed; this explicit /gitmoot merge request ran the gate now.

@jerryfane
jerryfane added this pull request to the merge queue Oct 24, 2026
Merged via the queue into main with commit 574d070 Oct 24, 2026
28 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant