Integrate remaining Python 3.15 final runtime and libraries - #8954

Summary

Remaining Python 3.15 integration based on official CPython v3.15.0 (d220409a13b77de7a6c30abe16ea90db01528edc). This remains a broad integration draft, not a claim of complete compatibility. Meaningful runtime and library changes continue to be separated into focused PRs.

Main reconciliation and compiler reduction

Reconciliation commit: 1880d095ecc534c1000d344c0b98ddfe56307b7a. Validated source tree: 9a61a9a8f4a8bf1830af4c10db95e5acdd2392ca.

• Integrate main df0d1b74c04408365f66666a03054d673dbccee7 on top of mega 0706c9bc463b9067d298b70448740704e64cab8c, preserving landed Implement PEP 798 comprehension unpacking #9342 comprehension unpacking/diagnostics, Fix Windows runtime compatibility issues #9406 Windows runtime compatibility, Preserve ctypes py_object provenance across casts and pointer writes #9391 ctypes provenance, Share ctypes backing-owner snapshots between aggregate copies #9392/Fix ctypes subclass bitfields and native type metadata #9395 ctypes ownership/type metadata, and Decide tp_new inheritance of static types at type initialization #9386 static tp_new inheritance.
• Resolve the three compiler conflicts by preserving the cloned parser options required by Implement PEP 798 comprehension unpacking #9342. Keep the new unpacking-diagnostic ownership checks intact.
• Remove the duplicate early lazy_future_import_error helper and its three call sites (64 lines). The existing main-owned unsupported_grammar_error already rejects lazy __future__ imports with the same ordinary message and four-byte lazy range. This deliberately restores main's error precedence and incomplete-interactive-input behavior; it is not claimed to be behavior-identical to the previous mega ordering.
• Do not introduce a new parser rule, new vendoring, test assertion changes, duplicate snippets, or a tiny standalone cleanup PR.

The two other pre-existing compiler residuals remain unchanged: the vendored parser's concatenated-literal diagnostic range and the obsolete-comparison adapter associated with #9074. Their ownership and validation are separate from this cleanup.

Validation of previous reconciliation 1880d095

• cargo check --locked --offline -j1 -p rustpython-compiler --lib --target x86_64-unknown-linux-gnu passed.
• cargo clippy --locked --offline -j1 -p rustpython-compiler --lib --target x86_64-unknown-linux-gnu -- -D warnings passed.
• Both ran with a fresh target and verified source, dependency, tool, native-library, and configuration inputs. The compiler and codegen source paths were confirmed; pre/post input verification and supervised process cleanup passed.
• Compiler rustfmt and staged diff checks passed. Explicit affected-file conflict, Ruff format/check, redundant-test-patch, rustfmt and prettier hooks all passed without changing the validated tree. The normal commit hook also passed its applicable checks. Cspell alone was omitted for this commit under the maintainer’s explicit one-time exception because its npm bootstrap is blocked. Cspell has not passed; the exception does not extend to later commits.

These are focused compiler compile-only results, not a new CLI build, Python execution, full workspace/C-API test run, or all-platform CI result. #9342's green CI does not automatically validate mega's different parser composition. For exact commit 1880d095ecc534c1000d344c0b98ddfe56307b7a, correctness CI completed successfully with all 24 jobs, including Linux, macOS, Windows and WASI CPython jobs. Lib Dependencies also passed. These hosted results apply to this commit, not future unpublished changes. That prior head conflicted with newer main b25ab5e76b1f287e3599bd60a58b65abf167de7b; the appended reconciliation below resolves that source composition. The prior green run does not validate the new head or establish merge readiness.

Remaining scope and limits

The earlier 1880d095 reconciliation left the inherited vendor trees unchanged. The new update below removes XML vendoring after #9197 landed. The remaining Ruff parser vendor tree is unchanged and remains subject to existing ownership/architecture constraints; retaining it does not endorse new vendoring. Parser/AST/warning-producer residuals, C-API fixtures, interpreter lifecycle/GC changes and their existing ownership constraints remain separate work. No new full-suite, parser fuzz, lifecycle/GC stress, profiler, native-stack/thread or JIT runtime test was executed for this cleanup.

Previous validation belongs to its previously identified commits, not this composition. This PR stays draft while splitting and compatibility work continue. No merge or automatic merge is requested.

AI assistance

Source reconciliation, cleanup, independent review and automated validation were assisted by dot. The runtime does not expose its exact model version. Commit trailers disclose the assistance; no human-use verification is claimed.

Reconcile landed XML and remove unused mega scaffolding

Published source commit: 4914518a068987677abfa8e9d609414b1506820a; validated tree: fad1706539e0bc52e687d0b56a6756555417b4d5.

Normal merge of pinned main b25ab5e76b1f287e3599bd60a58b65abf167de7b into mega 1880d095ecc534c1000d344c0b98ddfe56307b7a, combined with the separately reviewed six-file monitoring/lazy-predicate cleanup. This reduces the existing mega branch; no duplicate XML PR is needed.

Landed XML reconciliation

• Preserve all 14 XML library/test and Rust files exactly as merged in Align XML backend and canonical libraries with Python 3.15 #9197, including its 25 proven expected-failure retirements.
• Delete the obsolete 39-file vendor/xml tree and its Cargo workspace/hook exclusions; retain unrelated parser-vendor configuration unchanged.
• Use published xml 1.4.0 with the exact registry source and lock checksum from main.
• Drop the vendor-specific amplification APIs and restore main's 10 amplification expectations, including inherited-test wrappers. These are the existing merged-main limitations, not new skips or changed assertions. Existing registry expansion limits remain unchanged.
• Retain the landed limitations: namespace maps do not preserve redundant same-URI declarations or arbitrary declaration order; the reparse flag does not establish full Expat deferred-callback timing or buffered-WASM performance/security parity; external child feeding remains unsupported. The external-DTD canonical test proves resolver dispatch and the expected unsupported-scheme URLError, not successful external DTD parsing.

Monitoring/lazy-predicate cleanup

• Remove the otherwise unused native monitoring adapter/version state and its private _testcapi monitoring fixture/registration.
• Restore ordinary Python monitoring and sys wiring to pinned main. Remove only the later duplicate badsyntax predicate; retain the earlier unconditional exclusion and other short-circuit guards.
• The removed adapter had no actual PyMonitoring C ABI exports. Canonical consumers remain under their existing whole-class skip; no canonical monitoring assertion or skip is changed. Unrelated _testcapi fixtures remain intact.

This deliberately removes monitoring-specific private fixture names in capi-enabled builds; it is not a universal API/behavior-equivalence claim or a native C monitoring test pass.

Validation boundaries

• Independent source composition review: all 14 landed XML entries match main and all six earlier cleanup entries match reviewed tree da956ed18b6ae43fa6e8591bde1f874c3f36325f.
• Focused locked/offline metadata resolution and fresh-target cargo check -p rustpython-stdlib --lib --no-default-features --features compiler,host_env,threading --target x86_64-unknown-linux-gnu passed on this combined tree (stdlib check: 535 seconds). Shared leases, resource bounds and full pre/post source/tool/cache closure checks passed.
• Earlier derive-impl and capi-enabled VM checks passed on the separate six-file cleanup, not this combined tree. Align XML backend and canonical libraries with Python 3.15 #9197's successful CI was on its separate published head, not the combined mega candidate.
• Configured merge-conflict and rustfmt checks passed; Ruff/patch checks reported no applicable files under normal merge selection. No hook altered the source.
• No combined interpreter runtime tests, Clippy, C API runtime validation, or full-workspace success is claimed.
• The normal Git commit ran all other configured hooks; cspell alone was omitted locally under the user’s new instruction to check it in CI. Earlier exceptions were not reused. For this 4914518a head, hosted Lint/cspell and Lib Dependencies passed; Linux production and CAPI canonical jobs and WASI passed. Remaining correctness jobs were still active/queued before the follow-up below. These results do not validate the newer follow-up head.

AI assistance: dot prepared source composition, migration provenance checks and bounded validation. Exact model version is not exposed.

Remove unnecessary ASDL assets and helper exposure

Follow-up commit: cc9f601182cad6054a5e57c111ba20ce7886c282; source tree: 39469209bdab4783a33e8bb4ecf40d22ec52d4f6.

This follow-up removes the mega-only Parser/Python.asdl, Parser/asdl.py, and accompanying README. They are CPython ASDL test assets, not inputs to RustPython's Rust parser, code generator or build. The only tracked loader is the canonical test_asdl_parser suite; its source-build guard, assertions and all other Lib files remain unchanged.

In the normal RustPython checkout/build configuration, sysconfig.is_python_build() finds no CPython Modules/Setup or Modules/Setup.local marker. The parent commit's Linux canonical CI already reported test_asdl_parser skipped -- test irrelevant for an installed Python while these assets were present. This is not an unconditional skip guarantee for unusual _PYTHON_PROJECT_BASE/sys._home overrides pointing at a foreign CPython source build.

The same cleanup restores make_generator_or_coro to main's private visibility. Its two actual tracked callers are in the same module; the three other mentions are comments. No generator implementation or Python behavior is changed.

Validation: exact four-path diff, unchanged Lib tree, helper file byte-identical to pinned main, complete tracked consumer search, staged diff checks and configured conflict/rustfmt checks passed. No new local interpreter runtime, full build or Clippy run was performed for this follow-up. The normal Git commit ran all other configured hooks successfully; local cspell alone was omitted under the user's instruction to use CI. For exact follow-up head cc9f601182cad6054a5e57c111ba20ce7886c282, correctness CI completed successfully, including Linux, macOS and Windows native canonical/CAPI jobs. Hosted Lint passed, with cspell verified in both hook stages, and Lib Dependencies passed. These hosted results apply to this exact head; they do not establish complete Python compatibility or remove the draft and validation limitations above.

Reconcile cross-interpreter fixture boundaries with #9408

Follow-up commit: dfdd8a5560c511424d720ad1391645bf549b936e; source tree: 77d0bbccedc9a106b7b9ec0cb72688259868387e.

This follow-up reuses the reviewed #9408 source at 90bb550c2d52a156627ae391ca8a48fdfdcafb14: vm/crossinterp.rs and its test_support.rs are byte-identical to that revision, and the matching capsule fixture block is carried into mega without changing its unrelated interpreter lifecycle or time fixtures.

• Remove fixture-only Marshalled from production SharedValue, restore private from_function, and remove the fixture-only public pickle/marshal/script constructors. Preserve all serialization modes in a crate-private CrossinterpData bridge.
• Keep capsule creation, cloning and destruction consistent with the relocated payload. Static whole-crate consumer review found no remaining calls to the removed public helpers; their only prior external consumers were the migrated fixture block.
• Exactly three Rust files change (+86/-71); this reduces production API exposure, not total lines. No canonical Lib file, test assertion, interpreter lifecycle implementation, parser or main-owned change is modified. This is not a new duplicate PR or a wholesale main merge.

Validation: independent source integration review and staged whitespace checks passed. The normal commit ran configured hooks other than cspell successfully (conflict/rustfmt passed; other file types were inapplicable); local cspell is omitted under the user's instruction to verify it in CI. The environment reset removed the dependency/build cache, so no new local compile, Clippy or interpreter runtime result is claimed. Prior #9408 and mega CI belong to their respective heads and do not validate this new composition. For exact follow-up head dfdd8a5560c511424d720ad1391645bf549b936e, correctness CI completed successfully with all 24 jobs, including Linux, macOS and Windows production/CAPI canonical jobs. Hosted Lint passed, with cspell verified in both hook stages, and Lib Dependencies passed. These hosted results validate this exact head within the workflow coverage, not complete Python compatibility. The PR remains draft, with all earlier behavioral and validation limitations retained.

Drop nonessential codegen/VM refactor differences

Follow-up commit: cbb1ce4fd26bc0b5687174bdd1df2609fc0f9afa; source tree: d9b7b6ede1c27bb84b67c583a03a4b9dec3f43eb.

Restore four files byte-for-byte to pinned main 77046cf527fac8a1429683c1d6fe66a30e3f1be7: crates/codegen/src/compile.rs, crates/vm/src/import.rs, crates/vm/src/vm/mod.rs, and crates/vm/src/stdlib/_ast/python.rs. The first three contain only comment differences. The AST differences were a constructor return-style rewrite and reuse of an existing abstract-type list, not new compatibility behavior.

The existing abstract_types() helper and its real is_abstract consumer remain intact. AST registration explicitly binds the module's AST attribute to NodeAst before annotation initialization. Restoring main initializes the same base and twelve abstract types, with main's assignment order (AST.__annotations__ is assigned before _fields, _attributes and __match_args__, rather than afterward). This does not claim identical class-dictionary insertion order or every observable; it is normal-path source reasoning, not a new allocation-failure or runtime-equivalence test claim. Actual type-comment rejection metadata, parser/compiler behavior, cross-interpreter bridges, interpreter lifecycle, source-directory reporting, and all canonical Lib files remain untouched.

This removes four files from mega's diff against pinned main. The follow-up itself adds 20 and removes 5 lines; it reduces patch scope, not total source size. Independent source review, exact main-blob comparison and normal applicable commit hooks passed for this small restoration (conflict/rustfmt passed; other file types were inapplicable). No fresh local compile, Clippy or interpreter runtime was run; cspell is checked in CI under the user's instruction. Previous green results belong to dfdd8a556, not this new head. New-head correctness, Lib Dependencies and cspell CI are pending; draft status and previous limitations remain.

Reconcile the newly landed cross-interpreter split

Merge commit: 720ca7869188421c264f3da7f2a960ca674f6009; source tree: f84ee214d99e416e9a68b78c68a76aee27678daf.

#9408 landed as main 058bebf0f07be2508feed2470933990c9edb16c4 while the preceding cleanup was being published. GitHub's synthetic merge of cbb1ce4fd with that new main combined two equivalent import layouts and failed with Rust E0252 (CrossinterpData imported twice). This was a compile failure in the combined tree, not a test assertion failure or a reason to remove the fixture.

This normal main merge preserves the landed CSV and C API changes and all four preceding cleanup files. The only manual merge adjustment removes the duplicate grouped import, retaining the standalone import exactly as landed in #9408. Incoming canonical Lib changes are copied from main without manual edits. No test expectations, compiler/parser behavior or fixture functionality is removed.

Independent source review checks the natural merge plus the exact one-line resolution, incoming-main entries, and one remaining import. Normal applicable commit hooks passed (conflict/rustfmt; other file types were inapplicable under merge selection); local cspell is left to hosted CI as authorized. No new local compile/runtime pass is claimed. The preceding cbb1ce4fd workflow contains real compilation failures and must not be described as green; new-head correctness, Lib Dependencies and cspell CI remain pending. Draft status and earlier validation limits are retained.