Track embedded Python inputs for incremental Cargo rebuilds - #35

Summary

Track files read by py_compile! and py_freeze! with inert, fully qualified include_bytes! dependency markers so editing an existing embedded Python input triggers a Cargo rebuild. Includes recursive frozen-directory inputs, skipped invalid inputs, and a standalone regression driver without a consumer build script. This is the scoped fix for upstream issue 5065.

This fork-only draft is for human review in moreal/RustPython. It contains exactly three files and one commit based on 06007c81fa99e813cbf4583dab326764ba0ef017, the freshly verified target main and the exact fully validated base. No source adaptation was required.

Validation on this base

• Before patch: 28 expected regression failures and 33 passing controls. After the exact patch: 61/61 regression checks passed. The consumer decodes actual macro-generated bytecode, has no build script, and preserves Rust source, manifest, and lockfile hashes/mtimes during Python-only edits.
• Workspace tests: 1,355 passed, 18 ignored. Separate C-API Cargo tests: 121 passed, 4 ignored.
• Workspace and separate C-API Clippy passed without warnings. Configured hooks, focused Clippy, and workspace format passed. Release test_compile: 185 run, 51 skipped.
• Independent automated review found no blocking findings for the disclosed scope. Focused derive/derive-impl Cargo tests executed zero unit tests and ignored 15 doctests; the standalone regression is the behavioral evidence.
• Publication checks verified exact source hashes, clean patch application, the three-file diff, and a fresh configured-hook pass against the current target.
• Reference CPython was 3.15.0rc3, a prerelease. This base targets Python 3.15 with compatibility work in progress; no complete Python 3.15 compatibility claim is made.

The supplied source patch SHA256 is 4c183532977e8f8781975635e00da1eb7742d702e7d93665fbd79c5b0aa362ff.

Run the standalone regression from the patched checkout (new output directory; Cargo dependencies already cached):

python3 extra_tests/embedded_macro_rebuilds.py --repo . --output /tmp/embedded-rebuilds-NEW

Supplemental checks on the fork's earlier 7cd4a54e base also confirmed the original stale behavior (56/56 checks in --expect-stale mode), followed by 61/61 fixed-regression checks, configured hooks, and focused Clippy. Those results are separate from the current-base validation above.

Known tracking boundaries

• A completely new directory entry alone is discovered only after another known input or Rust source triggers compilation.
• Editing a symlink target's contents is tracked; retargeting the symlink alone can retain stale bytecode because rustc canonicalizes dependencies.
• Non-UTF-8 operating-system paths preserve existing compilation behavior but cannot receive the Rust string-literal dependency marker, so incremental tracking remains unavailable. The Unix rustc/make probe records this boundary; Cargo itself rejects these source paths on the tested toolchain.

Validation is Linux-only with Rust 1.99.0. Windows/macOS execution, human/manual verification, and separate C/Python C-API integration beyond Cargo tests remain unperformed. This draft is not a ready-to-merge claim.

AI assistance

OpenAI dot/Codex assisted with investigation, implementation, regression utilities, test execution, automated review, and this description. The exact model identifier was not exposed, so none is asserted. Human review/use and the upstream AI-policy submission gate remain pending; policy compliance is not claimed by opening this review draft in the requested fork.