Skip to content

chore(audit): add docs/wire-shape scanner + 2026-05-29 sweep#451

Merged
axisrow merged 3 commits into
mainfrom
audit/full-project-docs-wire-shape
May 29, 2026
Merged

chore(audit): add docs/wire-shape scanner + 2026-05-29 sweep#451
axisrow merged 3 commits into
mainfrom
audit/full-project-docs-wire-shape

Conversation

@axisrow

@axisrow axisrow commented May 29, 2026

Copy link
Copy Markdown
Owner

Summary

Tooling

Modes: --v4 / --v5 / --reports / --all. Reuses captcha + thin-body guards from direct_cli.reports_coverage.fetch_reports_spec. Three retries × 5 s pause; persistent captcha is recorded as captcha-after-retries rather than aborting.

Findings categories:

  • docs_field_missing_in_code — JSON-schema key present in docs, absent from example_param.
  • code_field_missing_in_docsexample_param key not seen in parsed docs schema (manual review needed).
  • required_field_missing_in_payload_module — Live 4 changelog marker «Входной параметр X стал обязательным», but example_param lacks X.
  • notes_points_at_reference_urlcontract.notes cites dg-v4/reference/ while dg-v4/live/ exists.
  • live4_marker_absent — live page reachable but lacks the Live 4 changelog block.
  • docs_unreachable — captcha / 404 / network error.
  • v5_schema_seen / v5_schema_not_extracted — positive coverage / unparseable group page.

2026-05-29 results (high level)

Layer Endpoints Audited Captcha Findings
v4 Live 32 16 15 47
v5 ops ~112 93 18 105
Reports 1 1 0 1 OK

High-severity gaps surfaced:

  • GetEventsLogexample_param missing 11 fields (AccountIDS, BannerIDS, CampaignIDS, EventType, Filter, LastEventOnly, Limit, Logins, Offset, PhraseIDS, WithTextDescription).
  • CreateNewForecast — missing AuctionBids, Categories, CommonMinusWords.
  • GetBannersTags — missing CampaignIDS.
  • TransferMoney / PayCampaigns / CreateInvoiceCurrency (already fixed in fix(v4)!: restore --currency for live finance methods (regression from #441/#442/#443) #450).

Full severity-classified writeup posted as a comment to #111: #111 (comment)

Out of scope

Individual contract / payload-builder fixes for the HIGH-severity gaps are deliberately deferred to follow-up PRs — one per gap, per the project's "1 issue = 1 PR" rule. This PR ships the auditing capability + the dated report only.

Test plan

  • pytest tests/test_audit_wire_shape.py — 9 offline golden tests pass.
  • python scripts/audit_wire_shape.py --all — produces the dated report; exit 1 on findings is by design.
  • Re-run weekly to clear captcha-blocked methods and watch for new drift.

New offline-rerunnable tool that parses Yandex Direct docs HTML, extracts the
request JSON-schema, and compares with V4_METHOD_CONTRACTS.example_param and
RESOURCE_MAPPING_V5. Closes the methodological gap that allowed the v4
Currency regression (root-caused in #125, fixed by #450) — that PR happened
because verification was done against dg-v4/reference/ pages instead of
dg-v4/live/ ones.

Tooling
-------
- scripts/audit_wire_shape.py — modes: --v4, --v5, --reports, --all.
  Uses a 3-retry × 5-second-pause captcha guard reusing the same markers
  as direct_cli.reports_coverage.fetch_reports_spec. Persistent captcha is
  recorded as `captcha-after-retries` rather than aborting the run.
- tests/test_audit_wire_shape.py — 9 offline golden tests covering
  request-section extraction, schema parsing, response-side leak
  prevention, Live 4 hint detection, example_param key walk, captcha
  marker detection, and v5 per-method URL derivation.

Sweep artefacts (2026-05-29)
----------------------------
- docs/audits/PROJECT_WIRE_SHAPE_AUDIT_2026-05-29.md — 174 findings
  across 32 v4 methods, ~112 v5 operations, 1 reports surface.
- docs/audits/wire_shape.json — same data, machine-readable.

Reproducibility
---------------
    python scripts/audit_wire_shape.py --all \
        --json docs/audits/wire_shape.json \
        --markdown docs/audits/PROJECT_WIRE_SHAPE_AUDIT_2026-05-29.md

Summary writeup with severity-classified findings posted as a comment to
issue #111 (link in the comment body).

This PR adds the auditing capability and the dated report only. Individual
contract / payload-builder fixes for the HIGH-severity gaps (GetEventsLog
contract missing 11 fields, CreateNewForecast missing 3+ fields,
GetBannersTags missing CampaignIDS) are deliberately deferred to follow-up
PRs — one per gap, per the project's "1 issue = 1 PR" rule.

Audit trail: #111 (comment)

Co-Authored-By: Claude Opus 4.7 <[email protected]>
@axisrow

axisrow commented May 29, 2026

Copy link
Copy Markdown
Owner Author

@claude review. Focus on critical issues: bugs, security vulnerabilities, logical errors, data loss risks, performance problems. Do NOT nitpick style, naming conventions, minor formatting, or subjective preferences — only flag issues that could break functionality or cause real harm in production.

@claude

claude Bot commented May 29, 2026

Copy link
Copy Markdown

Claude finished @axisrow's task in 3m 33s —— View job


PR Review: docs/wire-shape scanner

Reviewed scripts/audit_wire_shape.py + tests/test_audit_wire_shape.py for correctness. This is an offline dev/audit tool (not production runtime), so I weighted findings by whether they would produce wrong audit results — which is the real harm vector here, since follow-up "fix" PRs are meant to act on these findings.


🔴 1. _v5_per_method_url drops the resource segment for single-segment docs bases

scripts/audit_wire_shape.py:532-541 assumes every docs base ends in a doubled segment (…/ru/campaigns/campaigns) and unconditionally chops the last segment:

base = docs_base.rstrip("/")
parent, _, tail = base.rpartition("/")
return f"{parent}/{method}" if parent else f"{base}/{method}"

But several registry entries are single-segment (resource_mapping.py:81,86,121,157) with no docs_pages override (only 1 docs_pages entry exists in the whole file):

  • dynamictextadtargetsdocs="…/ru/dynamictextadtargets"
  • dynamicfeedadtargetsdocs="…/ru/dynamicfeedadtargets"
  • vcardsdocs="…/ru/vcards"
  • smartadtargetsdocs="…/ru/smartadtargets"

For these, op get derives …/ru/get instead of …/ru/dynamictextadtargets/get — the resource name is dropped. Result: ~24 v5 method-pages across 4 groups are fetched at wrong URLs that 404/captcha, so those resources are silently not audited while still appearing in the run. Real gaps in those 4 groups would be masked.

The test at tests/test_audit_wire_shape.py:140-145 actually enshrines the wrong output (…/ru/dynamictextadtargets + get…/ru/get), so the bug is locked in by a passing test.

Fix: only chop the tail when it duplicates the parent segment; otherwise append the method to the full base.

Fix this →


🟠 2. Parser is only validated against an idealized mock — verify findings against real HTML before acting on them

_FIELD_RE (audit_wire_shape.py:192-194) requires the closing quote to be immediately adjacent to the identifier:

r"(?:&quot;|\")([A-Za-z][A-Za-z0-9_]*)(?:&quot;|\")\s*:"

The only test fixture (_MOCK_LIVE_PAGE) is hand-built to match this exactly. Real Yandex docs render JSON inside syntax-highlighted <code> blocks where tokens are wrapped in <span>s; after strip_html replaces tags with spaces, "Currency" can become " Currency " — quote no longer adjacent → no match.

Because docs_field_missing_in_code is an exit-1 error kind (audit_wire_shape.py:762-768), a parser that mis-reads real HTML can emit false HIGH-severity gaps. The headline findings (GetEventsLog missing 11 fields, CreateNewForecast missing 3) are entirely parser-derived and not covered by any test against real captured HTML. Before opening the deferred per-gap fix PRs, spot-check 2–3 of these against the actual rendered docs (or add a golden test using a real captured page) so you're not fixing contracts based on extraction artifacts. The committed tests/reports_cache/raw/*.html show real captured pages are already available as fixtures.


🟡 3. Hardcoded Yandex URL literals violate the "No URL literals outside the registry" rule

CLAUDE.md: "Every Yandex docs/API URL is declared once … never write the URL as a string literal anywhere else. Captcha-poisoning of the docs cache (#426) was possible only because the same URL was duplicated."

This PR re-introduces that duplication:

No automated AST guard enforces this today, so CI won't catch it, but it directly contradicts a documented invariant that exists for a security reason.


✅ Looks correct

  • Captcha/thin/HTTP retry logic in fetch_doc is sound; never aborts, records captcha-after-retries, and non-ok statuses correctly continue so blocked pages don't generate false field diffs.
  • _collect_example_keys recursion over nested dicts/lists is correct (verified by the per-type mis-attribution being harmless since v4 flattens to a union set).
  • Reports smoke-marker check and exit-code gating are reasonable.

Summary: One real correctness bug (#1, with a test that locks it in) that causes 4 v5 resource groups to be silently mis-audited; one methodological risk (#2) worth resolving before the follow-up fix PRs land; one convention/security-hygiene issue (#3). None block the tool from running, but #1 and #2 affect the trustworthiness of the findings this PR is meant to produce.
· audit/full-project-docs-wire-shape

… URL

Review fixes for #451:

- _v5_per_method_url chopped the trailing path segment unconditionally,
  so single-segment docs bases (vcards, smartadtargets, dynamictextadtargets,
  dynamicfeedadtargets) lost their resource name and resolved to
  `…/ru/get` — silently mis-auditing 4 v5 groups at 404/captcha URLs.
  Now the tail is chopped only when it duplicates its parent segment;
  single-segment bases append the method to the full base.
- The test that locked in the buggy `…/ru/get` output is corrected and a
  vcards regression-guard case is added.
- REPORTS_SPEC_URL now imports REPORTS_SPEC_URLS["spec"] from the registry
  instead of duplicating the literal, per CLAUDE.md "No URL literals
  outside the registry".

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
@axisrow

axisrow commented May 29, 2026

Copy link
Copy Markdown
Owner Author

Триаж ревью (проверено по реальному коду):

🔴 Находка 1 (FIX — исправлено в 801324c): баг подтверждён. _v5_per_method_url безусловно резал хвост, и для одиночных docs-баз (vcards, smartadtargets, dynamictextadtargets, dynamicfeedadtargets — реальные записи в resource_mapping.py без doubling) метод get давал …/ru/get, теряя имя ресурса → 4 группы молча не аудировались. Теперь хвост режется только если он дублирует родительский сегмент; одиночные базы дополняются методом к полной базе. Тест, закреплявший …/ru/get, исправлен + добавлен regression-guard для vcards.

🟡 Находка 3 (частично FIX — исправлено в 801324c): REPORTS_SPEC_URL теперь импортирует REPORTS_SPEC_URLS["spec"] из реестра вместо дублирования литерала (как требует CLAUDE.md). По V4_LIVE_BASE/V4_REFERENCE_BASE: этих URL нет ни в одном реестре проекта вовсе (v4-доки не имеют выделенного registry-модуля) — это пограничный случай, вынос в реестр оставлен как отдельный follow-up, чтобы не расширять scope этого PR. AST-гарда на URL-литералы в CI сейчас нет, так что инвариант не гейтится автоматически.

🟠 Находка 2 (SKIP / follow-up): риск реальный (regex _FIELD_RE требует кавычку вплотную; golden-фикстура только синтетическая). Но заголовочные находки парсера (GetEventsLog 9+5 полей, CreateNewForecast 6 полей) я независимо сверил с живыми dg-v4/live/* доками — они совпали, т.е. это риск будущих ложных срабатываний, а не текущий баг. Добавление golden-теста на захваченном HTML вынесено в follow-up, не блокирует merge.

Локально: ruff check . чисто, полный offline-набор pytest -m 'not integration' — 1923 passed, 46 skipped (вкл. новые кейсы).

@axisrow

axisrow commented May 29, 2026

Copy link
Copy Markdown
Owner Author

@claude review. Focus on critical issues: bugs, security vulnerabilities, logical errors, data loss risks, performance problems. Do NOT nitpick style, naming conventions, minor formatting, or subjective preferences — only flag issues that could break functionality or cause real harm in production.

@claude

claude Bot commented May 29, 2026

Copy link
Copy Markdown

Claude finished @axisrow's task in 1m 9s —— View job


Re-review: docs/wire-shape scanner (post-801324c)

Re-checked the three prior findings against the current code, plus a fresh pass for new critical issues.


✅ Finding #1 (_v5_per_method_url) — correctly fixed

scripts/audit_wire_shape.py:547-553 now only chops the tail when it duplicates the parent segment:

parent, _, tail = base.rpartition("/")
if parent:
    grandparent, _, parent_tail = parent.rpartition("/")
    if tail == parent_tail:
        return f"{parent}/{method}"
return f"{base}/{method}"

Traced both paths:

  • Doubled …/ru/campaigns/campaigns + addtail="campaigns", parent_tail="campaigns" → match → …/ru/campaigns/add
  • Single …/ru/dynamictextadtargets + gettail="dynamictextadtargets", parent_tail="ru" → no match → …/ru/dynamictextadtargets/get ✅ (resource name preserved)

The test at tests/test_audit_wire_shape.py:133-158 was corrected and a vcards regression guard added. The 4 mis-audited v5 groups (vcards/smartadtargets/dynamic*) are now fetched at the right URLs. Resolved.

✅ Finding #3 (URL literals) — primary case fixed

REPORTS_SPEC_URL now imports REPORTS_SPEC_URLS["spec"] (audit_wire_shape.py:75) instead of duplicating the literal. V4_LIVE_BASE/V4_REFERENCE_BASE remain literals, but there is genuinely no v4 docs registry to source them from — deferring the registry extraction to a follow-up is a reasonable scope call. Acceptable.

✅ Finding #2 (parser regex) — risk acknowledged, deferred

Still no golden test against captured real HTML, but you independently cross-checked the headline dg-v4/live/* findings against the rendered docs. Adding a real-HTML fixture before the per-gap fix PRs land remains the right follow-up. No new blocker.


New observation (low severity — not a blocker)

notes_points_at_reference_url can fire on captcha'd pages. At audit_wire_shape.py:419-436, url_used only switches to reference_url when the live page returns http-404/http-410. A captcha response keeps url_used == live_url, so the FINDING_NOTES_REFERENCE_URL check at line 427 fires even though the live page was never actually verified to contain a Live 4 schema.

Since this is an exit-1 error kind and 15/16 v4 methods were captcha-blocked in this run, the finding asserts "live exists, switch off reference" on the basis of a page we couldn't read. It's not a false read of code state (the notes check is static), but it overstates confidence in the live page's existence. Tightening it to require fetched.status == "ok" before flagging would make the finding trustworthy under captcha:

if "dg-v4/reference/" in notes and url_used == live_url and fetched.status == "ok":

Fix this →


Verdict

All three prior findings are resolved or reasonably deferred. The _v5_per_method_url fix is correct and test-guarded. No new critical bugs, security issues, or data-loss risks. The one observation above is a minor robustness improvement to finding trustworthiness under captcha — it does not block merge, especially since this is an offline audit tool whose findings already require manual triage before any contract fix lands.

LGTM to merge.
· audit/full-project-docs-wire-shape

The FINDING_NOTES_REFERENCE_URL check fired whenever the contract notes
cited dg-v4/reference/ and url_used stayed live_url — but a captcha'd live
page also leaves url_used == live_url, so the finding asserted "live exists,
drop reference" on a page that was never actually read (15/16 v4 methods
were captcha-blocked in the 2026-05-29 run). Require fetched.status == "ok"
so the finding only fires when the live schema was genuinely verified.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
@axisrow axisrow merged commit 6f2a11c into main May 29, 2026
6 checks passed
@axisrow axisrow deleted the audit/full-project-docs-wire-shape branch May 29, 2026 03:53
axisrow added a commit that referenced this pull request May 29, 2026
…t fields (#456)

Closes the docs-drift gaps from the 2026-05-29 wire-shape audit (#451).
Additive change — only new optional flags; existing flags unchanged.

- v4events get-events-log (#452): adds --last-event-only, --with-text-description
  (Yes/No), --logins, and a nested Filter object via --filter-campaign-ids /
  --filter-banner-ids / --filter-phrase-ids / --filter-account-ids /
  --filter-event-type. Filter keys mirror the WSDL (CampaignIDS/BannerIDS/
  PhraseIDS/AccountIDS/EventType); EventType is validated against the documented
  9-value enum before any API call.
- v4forecast create (#453): adds --categories, --auction-bids (Yes/No) and
  --common-minus-words. Categories is accepted but ignored by the API per docs.

All field names verified against official dg-v4/live docs (Docs-verified
2026-05-29); audit_wire_shape.py --v4 reports 0 findings for both methods.
Contracts (example_param/notes/login_placement) updated, dry-run tests assert
exact request bodies, README examples added.

Refs #452 #453

Co-authored-by: axisrow <[email protected]>
Co-authored-by: Claude Opus 4.8 <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants