Skip to content

feat(telemetry): report envelope v2 keeps the figures apart from the run - #1048

Merged
blafourcade merged 13 commits into
nextfrom
feat/report-envelope-v2
Oct 17, 2026
Merged

blafourcade merged 13 commits into
nextfrom
feat/report-envelope-v2

Conversation

@blafourcade

@blafourcade blafourcade commented Oct 17, 2026 •

Copy link
Copy Markdown
Contributor

🎯 What & why

aidd telemetry report --json printed its figures next to facts about the read that produced them, such as files read, calls not stored and the oldest transcript. So two reports over the same data were not byte-identical: the second reads 0 files where the first read 2. Envelope v2 keeps the figures apart from a run object. This PR also adds a guard that runs the telemetry tests under a far time zone and a non-English locale, locally and in CI.

🛠️ How it works

  • The figures are every top-level field except run: period, axis, filters, rows, totals, coverage.records and coverage.left_out. They depend only on the ledger's records, its left-out file, the declarations, the identity, the period, the filters and the axis. run, always last, holds what this read found. The definition lives in one place, envelope.md, and the usage contract and the CLI memory link to it.
  • unknown_records is no longer at the top level. It only copied totals.total.unknown_records, so v2 keeps it once, in the totals.
  • Each field was decided on its own. The reasons are in the v1 to v2 table. Two moves are less obvious:
    • skipped_ledger_lines really changes on a re-run, because the report's own settle keeps each damaged line once. A test proves it, 2 then 1.
    • consent_log_damaged says whether this read could store anything, not anything about the figures.
  • The CLI makes the same split in ReportResult (coverage and run). The text report prints exactly what it printed before.
  • Properties:
    • P1 (fast-check) checks that the figures' bytes are the same across storage order and batching, monthly partitions, the zone the calls were stored in, no-op stores, a second report, and TZ in UTC, Kiritimati (UTC+14) and Adak (UTC−10), while run carries random facts. The envelope is built inside the zone being tested.
    • P2 runs the built CLI on one sandbox and checks that the figures are identical across 3 zones × 3 locales (C, fr_FR, ar_SA), where run.files_read goes from 2 to 0.
  • Guard: pnpm test:foreign-clock first has a child Node write a number and a date, and requires both to differ from en-US, with a non-zero offset. It exits 1 otherwise, for example on a bogus locale or zone. It then runs the telemetry tests under UTC+14 and ar_SA. CI runs it as one step in the existing cli-coverage job (cli-ci.yml:303). Two hand mutations prove it is a guard (see the evidence):
    • The ledger day in local time. On origin/next, 1493 tests in scope pass it under UTC. The guard fails 36 of them.
    • Counts grouped by the default locale. The default suite passes it. The guard fails 24.
  • Readers updated: the 01-usage skill (answer and envelope reference), the usage contract, the CLI telemetry memory, and check-claude-tokens-end-to-end.cjs, which now calls itself broken on any version other than 2.

🧪 How to verify

  • cd cli && pnpm test
  • cd cli && pnpm test:foreign-clock
  • cd cli && pnpm exec vitest run tests/contexts/telemetry/application/report/report-figures.property.integration.test.ts tests/e2e/telemetry-report-figures.e2e.test.ts
  • node scripts/check-claude-tokens-end-to-end.cjs cli/dist/cli.js (and --attended)

⚠️ Heads-up

  • Envelope v2, no v1 shape.
    • coverage.files_read, unrecognised_shapes, not_stored, consent_log_damaged, oldest_transcript_at and skipped_ledger_lines moved to run.
    • version is 2.
    • The refusal is {"version": 2, "refused": "AIDD_TELEMETRY=0"}.
    • unknown_records is no longer at the top level: read totals.total.unknown_records.
    • Plugin and CLI must move together. On a version mismatch, the skill gives no figure and names the fix: update the CLI if the envelope is below 2, the plugin if it is above.
  • Release impact: ships as a minor, by the maintainer's decision. The project is in beta, so the envelope change is not marked breaking.
    • The squash-merge message carries no ! and no BREAKING CHANGE: footer. release-please reads only that message.
    • Merge by squash only. The branch history keeps d25f7ca7 feat(telemetry)!. A merge commit would carry it to main and ship this as breaking.
  • Limits:
    • Node applies a runtime TZ change only on a main thread. Under Stryker's worker threads, P1 does not vary the zone (report-figures test:218).
    • The locale is fixed when a process starts. So locale identity is checked across processes only on the fixed e2e sandbox. P1 under the guard compares against itself.
    • The guard runs on ubuntu only. Windows ICU ignores LC_ALL, and this was not run on Windows locally.
  • Mutation: telemetry 96.7 (floor 96) at 25c9f857, with every changed-line mutant killed. The rerun after review round 1 was stopped by the machine running low on memory and has not been redone. Bundle: 712,713 B, 696.0 KB of 709 KB, not raised.

🔗 Linked issue

None. Found by the R1 telemetry audit (M5, M9).

✅ I certify

  • I DO CERTIFY I READ EACH LINE OF THE PULL REQUEST BECAUSE I AM A SOFTWARE ENGINEER, NOT A AI PUPPY.

🤖 Generated with Claude Code

https://claude.ai/code/session_01VWNxk63AGKkqE8HRqHLjGp

blafourcade and others added 13 commits October 17, 2026 06:01
… run

Envelope v1 mixed the figures with facts about the read that printed
them (files read, shapes met, calls not stored, the oldest transcript,
ledger lines skipped before settling), so two reports over the same data
were not byte-identical.

Version 2 keeps every figure at top level: period, axis, filters, rows,
totals, unknown_records, and coverage.records and coverage.left_out.
They are a function of the ledger, the declarations, the identity and
the request alone. The run's facts move to `run`, which is always last.
The CLI's ReportResult makes the same split, and the text report prints
what it printed before.

Readers updated: the 01-usage skill, the usage contract, the CLI
memory, and the token end-to-end check, which now refuses a version
other than 2.

BREAKING CHANGE: `aidd telemetry report --json` prints `version: 2`.
`coverage.files_read`, `unrecognised_shapes`, `not_stored`,
`consent_log_damaged`, `oldest_transcript_at` and `skipped_ledger_lines`
moved to `run`. The refusal is `{"version": 2, "refused": ...}`. There
is no v1 shape.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01VWNxk63AGKkqE8HRqHLjGp
CI runs in UTC and English, so code that reads the machine's time zone
or language passed everywhere it was checked. `pnpm test:foreign-clock`
runs the telemetry tests with TZ=Pacific/Kiritimati (UTC+14) and
LC_ALL=LANG=ar_SA.UTF-8. It first checks that a child Node really sees
both, and fails if it does not.

CI runs it as one step in the existing ubuntu cli-coverage job: no new
job, no matrix. Hand mutations prove the guard (see the task evidence):
- the ledger day read in local time: 36 tests fail under the guard,
  where the default suite on origin/next caught nothing;
- counts grouped by the default locale: 24 tests fail under the guard,
  and the default suite stays green.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01VWNxk63AGKkqE8HRqHLjGp
Node applies a new `TZ` only on a process's main thread. Stryker runs
Vitest on worker threads, where the figures property failed its own
precondition: "UTC: expected -120 to be +0". On a worker the property
now runs in the zone the thread started in. The forked suite still
varies the zone, and `pnpm test:foreign-clock` covers the rest.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01VWNxk63AGKkqE8HRqHLjGp
Contract with a field-by-field v1 to v2 table and the reason for each
move, the plan and its phases, the red runs, the hand mutations that
prove the foreign clock guard, and the gates. Review pending.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01VWNxk63AGKkqE8HRqHLjGp
The contract and phase 3 claimed the default suite misses the zone
mutation. With P1 in place it does not: one test fails. What was
observed is now on record: on 616dbbe the same mutation passes all
1493 tests in scope under UTC, while the guard fails 36. The locale
mutation passes the default suite and fails 24 under the guard. The
guard's scope is listed in full, and a comment in attribution.ts now
points to `run.not_stored`.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01VWNxk63AGKkqE8HRqHLjGp
…ts name

A locale ICU has no data for keeps its name and formats as English, so
LC_ALL=xx_BOGUS.UTF-8 passed the old check while the tests ran in
English. The check, now in scripts/foreign-clock.mjs, has a child Node
write a number and a date and requires both to differ from en-US, and
the offset to differ from 0. A bogus locale or zone makes the guard
exit 1 before any test runs.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01VWNxk63AGKkqE8HRqHLjGp
P1 built the envelope after restoring TZ, so a zone dependence in
report-envelope.ts escaped it. The envelope and the figures' bytes are
now made inside the zone. The ledger also starts with monthly
partitions, holding calls and lines that are no record, and the second
history is stored with the clock in another zone.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01VWNxk63AGKkqE8HRqHLjGp
The top-level unknown_records copied totals.total.unknown_records. The
copy is dropped from v2 before any release, and readers use the totals.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01VWNxk63AGKkqE8HRqHLjGp
…mismatch

The skill's envelope reference now defines, once, what a figure depends
on (ledger records, left-out file, declarations, identity, period,
filters, axis) and what run holds. The usage contract and the CLI memory
link to it. On an envelope version it does not read, the 01-usage skill
names the fix: update the CLI below 2, the plugin above 2.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01VWNxk63AGKkqE8HRqHLjGp
Windows ICU takes its locale from the system and ignores LC_ALL, so the
case accepting ar_SA failed on windows-latest. It is skipped there. The
three refusal cases still run on every platform.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01VWNxk63AGKkqE8HRqHLjGp
--days resolves against today's UTC day, so the figures are pure in the
resolved from and to, not in the flags. The usage contract no longer
restates what run holds: it links to the envelope reference. Also
records the checker's round 2.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01VWNxk63AGKkqE8HRqHLjGp
@blafourcade blafourcade changed the title feat(telemetry)!: report envelope v2 keeps the figures apart from the run feat(telemetry): report envelope v2 keeps the figures apart from the run Oct 17, 2026
@blafourcade
blafourcade marked this pull request as ready for review October 17, 2026 06:39
@blafourcade
blafourcade requested a review from a team as a code owner October 17, 2026 06:39
@blafourcade
blafourcade merged commit 940e8e3 into next Oct 17, 2026
61 checks passed
@blafourcade
blafourcade deleted the feat/report-envelope-v2 branch October 17, 2026 12:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant