Skip to content

fix(extract): stop walking JSON Schema files as config manifests (#2255) - #4048

Closed
Faisal-Fayaz wants to merge 1 commit into
Graphify-Labs:v8from
Faisal-Fayaz:fix/json-schema-not-config
Closed

Faisal-Fayaz wants to merge 1 commit into
Graphify-Labs:v8from
Faisal-Fayaz:fix/json-schema-not-config

Conversation

@Faisal-Fayaz

Copy link
Copy Markdown
Contributor

Problem

_is_config_json() decides whether a .json file is worth AST-walking. It matches by filename first, then falls back to a root-key probe:

_CONFIG_JSON_KEYS = frozenset({
    "dependencies", ..., "extends", "$ref", "$schema", "compilerOptions",
})

$schema belongs in that set — configs legitimately carry it. But it is also the defining marker of a JSON Schema, so every schema document satisfied the probe and was walked in full. That re-opens the keyword-node explosion #1224 closed for data JSON.

Measured impact from the report: one 160 KB contract contributed 362 nodes labelled with bare schema keywords (description ×113, type ×83, $ref ×53, required ×20, properties ×20, …) plus ~12 schema-only communities, which pollutes clustering, god-node analysis and GRAPH_REPORT.md — 13% of a 2,725-node graph.

A minimal reproduction of my own (a 20-line schema with $id + $defs) already yields 34 nodes / 34 edges, with description ×7 and type ×7 as graph nodes.

Implementation

Two changes, both ahead of the existing probe:

  1. Drop $ref from the probe keys. A root-level $ref is a schema construct and never a config signal. Unambiguous — no real config has one.

  2. Return False when the root object carries $schema plus a schema-definition marker ($defs, definitions, or $id). This is what separates "this file IS a schema" from "this config POINTS AT a schema": the latter points $schema at its own tool's schema (biomejs.dev, docs.renovatebot.com) and never carries these markers.

The root keys are now collected once into a set, so the schema check and the config probe read the file the same way instead of walking the root twice.

Why this is low-risk

The filename branches run before the probe, so biome.json / renovate.json / angular.json / package.json / tsconfig.json never depended on it. Only arbitrarily-named files did.

I built a control matrix and ran it before and after — every config shape keeps byte-identical node counts:

file before after
biome.json ($schema + config keys) True, 4 nodes True, 4 nodes
renovate.json ($schema + extends) True, 4 nodes True, 4 nodes
package.json / tsconfig.json True, 5 / 3 True, 5 / 3
foo.eslintrc.json / api.tsconfig.json True, 3 / 3 True, 3 / 3
custom.config.json (probe: extends) True, 4 nodes True, 4 nodes
weird.json (probe: compilerOptions) True, 3 nodes True, 3 nodes
contracts/events-v1.json ($schema+$defs) True, 5 nodes False, 0
ref.json (root $ref) True, 2 nodes False, 0

I deliberately used the structural markers rather than inspecting the $schema value for json-schema.org. The value check is tempting but unreliable (a custom meta-schema can be a local path) and it would make classification depend on URL substring matching — worse for the determinism invariant than spec-defined key names.

Documented limitation

A hand-written schema carrying $schema but no $defs / definitions / $id is still walked. Closing that needs a keyword-density heuristic, which risks skipping a real config whose keys happen to overlap schema vocabulary — the worse error, since it drops structure silently, exactly what #1224 was about.

Rather than guess, it is documented in the _is_config_json docstring and pinned by test_minimal_schema_without_defs_or_id_is_still_walked, so closing it later is a deliberate change with a test rather than an accident. Happy to attempt the heuristic if you'd prefer fuller coverage.

Also updated the now-inaccurate extract_json docstring, which still advertised $ref as a probe signal.

Verification

uv run pytest tests/test_json_schema_config.py -q    # 14 passed
uv run pytest tests/ -q                              # 28 failed, 6262 passed, 101 skipped
uv run ruff check .                                  # All checks passed!
uv run pyright                                       # 636 errors
  • 5 of the 14 new tests fail on the pre-fix tree (confirmed by reverting json_config.py and re-running). The other 9 are deliberate: config-side guards against over-correcting, plus the documented-limit case.
  • Baseline captured on a clean v0.9.75 checkout: 28 failed, 6248 passed, 101 skipped. The 28 failures are byte-identical between the two runs — diffed test-id-by-test-id, not compared by count. They are missing tree-sitter grammars for erlang/R/solidity/vbnet plus ollama retry tests. Passing delta is exactly +14, my new tests.
  • pyright at parity: 636 before and after. (The count moved 635→636 between v0.9.74 and v0.9.75 upstream, not from this change — I measured the baseline on this exact commit.)

tests/test_json_schema_config.py is the first test module for json_config; it follows the existing inline-SRC convention from test_scala_self_type.py. Tests cover the schema side ($defs, legacy definitions, subdirectory placement, bare $ref, and that no keyword labels leak into the graph), the config side (all seven shapes above, including two configs that carry $schema themselves), the documented boundary, and determinism across repeated calls.

Notes

🤖 Generated with Claude Code

…phify-Labs#2255)

`_is_config_json()` decides whether a `.json` file is worth AST-walking. It
matches by filename first, then falls back to a root-key probe. `$schema` was
in that probe's key set -- correctly, since configs legitimately carry it --
but it is also the *defining marker* of a JSON Schema, so every schema
document satisfied the probe and got walked in full. That re-opened the
keyword-node explosion Graphify-Labs#1224 closed for data JSON: one 160 KB contract
contributed 362 nodes labelled with bare schema keywords (`description` x113,
`type` x83, `$ref` x53, ...) plus ~12 schema-only communities, polluting
clustering, god-node analysis and GRAPH_REPORT.md.

Two changes, both ahead of the existing probe:

- Drop `$ref` from the probe keys. A root-level `$ref` is a schema construct
  and never a config signal.
- Return False when the root object carries `$schema` *plus* a
  schema-definition marker (`$defs`, `definitions`, or `$id`). This is what
  separates "this file IS a schema" from "this config POINTS AT a schema":
  the latter points `$schema` at its own tool's schema (biomejs.dev,
  docs.renovatebot.com) and never carries these markers.

The blast radius is smaller than it looks. The filename branches run first, so
biome.json / renovate.json / package.json / tsconfig.json never depended on the
probe at all; only arbitrarily-named files did. I confirmed this against a
control matrix before and after -- every config shape keeps byte-identical node
counts.

The root keys are now collected once into a set so the schema check and the
config probe agree on the same read of the file, rather than walking the root
twice.

Documented boundary, pinned by a test: a hand-written schema carrying only
`$schema` + `properties` (no `$defs`/`definitions`/`$id`) is still walked.
Closing that gap needs a keyword-density heuristic, which would risk skipping a
real config whose keys overlap schema vocabulary -- the worse error, since it
drops structure silently. Left for a deliberate change rather than guessed at.

Verified:
- uv run pytest tests/test_json_schema_config.py -q    (14 passed; 5 fail pre-fix)
- uv run pytest tests/ -q -> 28 failed, 6262 passed, 101 skipped
- baseline on a clean v0.9.75 checkout: 28 failed, 6248 passed, 101 skipped
  The 28 failures are byte-identical between the two (missing tree-sitter
  grammars for erlang/R/solidity/vbnet, plus ollama retry tests) and are diffed
  test-id-by-test-id, not compared by count. Passing delta is exactly +14.
- uv run ruff check . -> clean
- uv run pyright -> 636 errors, identical to the clean v0.9.75 baseline

Co-Authored-By: Claude <[email protected]>
@github-actions

github-actions Bot commented Oct 4, 2026

Copy link
Copy Markdown

Thanks for the pull request, @Faisal-Fayaz. A maintainer will review it soon.

Want to talk it through while it is in review? Come join us on our Discord server. For longer-form discussion there is also GitHub Discussions.

A couple of things that speed up review: make sure the test suite passes on Python 3.10 and 3.13, and that the change keeps extraction deterministic.

@graphify-labs graphify-labs Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Graphify reviewed this change.

Looks safe to merge — no coupling regressions and no blocking issues, checked against the code graph (not a self-assessment).

Formal verification. PR-changed functions: 1/2 verified (0 proven, 1 may-equivalent, 0 distinguished) · 1 not verified (1 vacuous).

Not verified on this run: \_is\_config\_json (vacuous: never exercised).


Graphify review — findings

Stops _is_config_json from treating JSON Schema documents as config manifests. Files whose root has $schema plus $defs, definitions, or $id are now skipped as data JSON, and a root-level $ref no longer counts as a config signal, so schemas stop flooding the graph with keyword nodes. Filename-matched configs like biome.json and renovate.json, and probe hits on extends, compilerOptions, or dependency keys, are still extracted; a schema with $schema but no definition markers is still walked.

No blocking issues surfaced. 2 lower-confidence candidates did not survive cross-model review.

Analysis details — impact, health, verification

Impact & health

Graphify review

Impact — 45 functions depend on the 31 functions this change touches.

Health — this change adds coupling hotspots:

  • new: extract_json() — 31 callers, 7 callees
  • new: walk_object() — 1 callers, 7 callees

Verification — 45 functions in the blast radius were not formally verified this run (proofs are advisory here).

Gate & verification

graphify gate

PASS — objectively clean (no health regressions, tests not run — proofs not run this pass (advisory)). Grounded, not self-assessed.

Advisory (not blocking):

  • verification_scope: 45 function(s) in the blast radius were not formally verified this run

Test selection

Test selection

2 of 327 test file(s) selected (1%) via static blast radius.

  • tests/test_extract.py — impact
  • tests/test_json_schema_config.py — impact, changed-test

Selection is safe under the controlled-regression assumption; always-run tests + a periodic full run are the backstops. Advisory — it never changes the check verdict.

Formal verification

Could not verify: Could not verify \_is\_config\_json.

The verifier did not have enough to check \_is\_config\_json, so it is saying so rather than guessing. No false assurance is the whole point.

Guarantee: No guarantee either way, this is an honest abstention, not a pass.

Note: Reason: no capturable inputs from the test suite; property tier: not verifiable: all 200 sampled inputs raised on both versions — the function never executed, so 'no divergence' would be vacuous (mostly AttributeError — names the real obstacle, not a sampling gap)

No difference found (not proven): No behavior difference found in extract\_json (not a proof).

The verifier ran both versions of extract\_json on many inputs and saw identical behavior every time. Strong evidence the change is safe, but evidence, not a proof.

Guarantee: Empirical: differential testing (both versions run on many generated inputs). A divergence on an untested input remains possible, so this is 'no counterexample found', not 'proven equivalent'.

Note: An input the sampler did not try could still differ.

· 1 grounded finding(s) anchored inline below; 1 more finding(s) on lines outside this diff (see the check run).

Comment thread graphify/extractors/json_config.py
@safishamsi

Copy link
Copy Markdown
Member

Shipped in v0.9.76 (live on PyPI as graphifyy==0.9.76). Landed on v8 via an authorship-preserving cherry-pick, so your original commit authorship is kept. Thanks @Faisal-Fayaz for skipping JSON Schema files as manifests 🙏

Closing as shipped.

@safishamsi safishamsi closed this Oct 4, 2026
@Faisal-Fayaz
Faisal-Fayaz deleted the fix/json-schema-not-config branch October 5, 2026 03:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants