fix(extract): stop walking JSON Schema files as config manifests (#2255) - #4048
Faisal-Fayaz wants to merge 1 commit into
Conversation
…phify-Labs#2255) `_is_config_json()` decides whether a `.json` file is worth AST-walking. It matches by filename first, then falls back to a root-key probe. `$schema` was in that probe's key set -- correctly, since configs legitimately carry it -- but it is also the *defining marker* of a JSON Schema, so every schema document satisfied the probe and got walked in full. That re-opened the keyword-node explosion Graphify-Labs#1224 closed for data JSON: one 160 KB contract contributed 362 nodes labelled with bare schema keywords (`description` x113, `type` x83, `$ref` x53, ...) plus ~12 schema-only communities, polluting clustering, god-node analysis and GRAPH_REPORT.md. Two changes, both ahead of the existing probe: - Drop `$ref` from the probe keys. A root-level `$ref` is a schema construct and never a config signal. - Return False when the root object carries `$schema` *plus* a schema-definition marker (`$defs`, `definitions`, or `$id`). This is what separates "this file IS a schema" from "this config POINTS AT a schema": the latter points `$schema` at its own tool's schema (biomejs.dev, docs.renovatebot.com) and never carries these markers. The blast radius is smaller than it looks. The filename branches run first, so biome.json / renovate.json / package.json / tsconfig.json never depended on the probe at all; only arbitrarily-named files did. I confirmed this against a control matrix before and after -- every config shape keeps byte-identical node counts. The root keys are now collected once into a set so the schema check and the config probe agree on the same read of the file, rather than walking the root twice. Documented boundary, pinned by a test: a hand-written schema carrying only `$schema` + `properties` (no `$defs`/`definitions`/`$id`) is still walked. Closing that gap needs a keyword-density heuristic, which would risk skipping a real config whose keys overlap schema vocabulary -- the worse error, since it drops structure silently. Left for a deliberate change rather than guessed at. Verified: - uv run pytest tests/test_json_schema_config.py -q (14 passed; 5 fail pre-fix) - uv run pytest tests/ -q -> 28 failed, 6262 passed, 101 skipped - baseline on a clean v0.9.75 checkout: 28 failed, 6248 passed, 101 skipped The 28 failures are byte-identical between the two (missing tree-sitter grammars for erlang/R/solidity/vbnet, plus ollama retry tests) and are diffed test-id-by-test-id, not compared by count. Passing delta is exactly +14. - uv run ruff check . -> clean - uv run pyright -> 636 errors, identical to the clean v0.9.75 baseline Co-Authored-By: Claude <[email protected]>
|
Thanks for the pull request, @Faisal-Fayaz. A maintainer will review it soon. Want to talk it through while it is in review? Come join us on our Discord server. For longer-form discussion there is also GitHub Discussions. A couple of things that speed up review: make sure the test suite passes on Python 3.10 and 3.13, and that the change keeps extraction deterministic. |
There was a problem hiding this comment.
Graphify reviewed this change.
Looks safe to merge — no coupling regressions and no blocking issues, checked against the code graph (not a self-assessment).
Formal verification. PR-changed functions: 1/2 verified (0 proven, 1 may-equivalent, 0 distinguished) · 1 not verified (1 vacuous).
Not verified on this run: \_is\_config\_json (vacuous: never exercised).
Graphify review — findings
Stops _is_config_json from treating JSON Schema documents as config manifests. Files whose root has $schema plus $defs, definitions, or $id are now skipped as data JSON, and a root-level $ref no longer counts as a config signal, so schemas stop flooding the graph with keyword nodes. Filename-matched configs like biome.json and renovate.json, and probe hits on extends, compilerOptions, or dependency keys, are still extracted; a schema with $schema but no definition markers is still walked.
No blocking issues surfaced. 2 lower-confidence candidates did not survive cross-model review.
Analysis details — impact, health, verification
Impact & health
Graphify review
Impact — 45 functions depend on the 31 functions this change touches.
Health — this change adds coupling hotspots:
- new:
extract_json()— 31 callers, 7 callees - new:
walk_object()— 1 callers, 7 callees
Verification — 45 functions in the blast radius were not formally verified this run (proofs are advisory here).
Gate & verification
graphify gate
PASS — objectively clean (no health regressions, tests not run — proofs not run this pass (advisory)). Grounded, not self-assessed.
Advisory (not blocking):
- verification_scope: 45 function(s) in the blast radius were not formally verified this run
Test selection
Test selection
2 of 327 test file(s) selected (1%) via static blast radius.
tests/test_extract.py— impacttests/test_json_schema_config.py— impact, changed-test
Selection is safe under the controlled-regression assumption; always-run tests + a periodic full run are the backstops. Advisory — it never changes the check verdict.
Formal verification
Could not verify: Could not verify \_is\_config\_json.
The verifier did not have enough to check \_is\_config\_json, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: no capturable inputs from the test suite; property tier: not verifiable: all 200 sampled inputs raised on both versions — the function never executed, so 'no divergence' would be vacuous (mostly AttributeError — names the real obstacle, not a sampling gap)
No difference found (not proven): No behavior difference found in extract\_json (not a proof).
The verifier ran both versions of extract\_json on many inputs and saw identical behavior every time. Strong evidence the change is safe, but evidence, not a proof.
Guarantee: Empirical: differential testing (both versions run on many generated inputs). A divergence on an untested input remains possible, so this is 'no counterexample found', not 'proven equivalent'.
Note: An input the sampler did not try could still differ.
· 1 grounded finding(s) anchored inline below; 1 more finding(s) on lines outside this diff (see the check run).
|
Shipped in v0.9.76 (live on PyPI as Closing as shipped. |
Problem
_is_config_json()decides whether a.jsonfile is worth AST-walking. It matches by filename first, then falls back to a root-key probe:$schemabelongs in that set — configs legitimately carry it. But it is also the defining marker of a JSON Schema, so every schema document satisfied the probe and was walked in full. That re-opens the keyword-node explosion #1224 closed for data JSON.Measured impact from the report: one 160 KB contract contributed 362 nodes labelled with bare schema keywords (
description×113,type×83,$ref×53,required×20,properties×20, …) plus ~12 schema-only communities, which pollutes clustering, god-node analysis andGRAPH_REPORT.md— 13% of a 2,725-node graph.A minimal reproduction of my own (a 20-line schema with
$id+$defs) already yields 34 nodes / 34 edges, withdescription×7 andtype×7 as graph nodes.Implementation
Two changes, both ahead of the existing probe:
Drop
$reffrom the probe keys. A root-level$refis a schema construct and never a config signal. Unambiguous — no real config has one.Return
Falsewhen the root object carries$schemaplus a schema-definition marker ($defs,definitions, or$id). This is what separates "this file IS a schema" from "this config POINTS AT a schema": the latter points$schemaat its own tool's schema (biomejs.dev,docs.renovatebot.com) and never carries these markers.The root keys are now collected once into a set, so the schema check and the config probe read the file the same way instead of walking the root twice.
Why this is low-risk
The filename branches run before the probe, so
biome.json/renovate.json/angular.json/package.json/tsconfig.jsonnever depended on it. Only arbitrarily-named files did.I built a control matrix and ran it before and after — every config shape keeps byte-identical node counts:
biome.json($schema+ config keys)renovate.json($schema+extends)package.json/tsconfig.jsonfoo.eslintrc.json/api.tsconfig.jsoncustom.config.json(probe:extends)weird.json(probe:compilerOptions)contracts/events-v1.json($schema+$defs)ref.json(root$ref)I deliberately used the structural markers rather than inspecting the
$schemavalue forjson-schema.org. The value check is tempting but unreliable (a custom meta-schema can be a local path) and it would make classification depend on URL substring matching — worse for the determinism invariant than spec-defined key names.Documented limitation
A hand-written schema carrying
$schemabut no$defs/definitions/$idis still walked. Closing that needs a keyword-density heuristic, which risks skipping a real config whose keys happen to overlap schema vocabulary — the worse error, since it drops structure silently, exactly what #1224 was about.Rather than guess, it is documented in the
_is_config_jsondocstring and pinned bytest_minimal_schema_without_defs_or_id_is_still_walked, so closing it later is a deliberate change with a test rather than an accident. Happy to attempt the heuristic if you'd prefer fuller coverage.Also updated the now-inaccurate
extract_jsondocstring, which still advertised$refas a probe signal.Verification
json_config.pyand re-running). The other 9 are deliberate: config-side guards against over-correcting, plus the documented-limit case.28 failed, 6248 passed, 101 skipped. The 28 failures are byte-identical between the two runs — diffed test-id-by-test-id, not compared by count. They are missing tree-sitter grammars for erlang/R/solidity/vbnet plus ollama retry tests. Passing delta is exactly +14, my new tests.tests/test_json_schema_config.pyis the first test module forjson_config; it follows the existing inline-SRCconvention fromtest_scala_self_type.py. Tests cover the schema side ($defs, legacydefinitions, subdirectory placement, bare$ref, and that no keyword labels leak into the graph), the config side (all seven shapes above, including two configs that carry$schemathemselves), the documented boundary, and determinism across repeated calls.Notes
{"skipped": "data json (not a config/manifest)"}marker with zero nodes — the same AST pass explodes data .json into orphan key-nodes (CODE_EXTENSIONS includes .json) — 561 isolated nodes on a real repo #1224 treatment data JSON already gets. Whether the file still gets a file node isdetect()'s concern, untouched here.🤖 Generated with Claude Code