Track Unpickler payload references during cyclic GC

Summary

Split the remaining Unpickler GC/lifetime closure from the Python 3.15 upgrade work. This changes crates/stdlib/src/pickle.rs (+68/-17) on main base bcfad3554 and appends two nonduplicative payload-cycle regressions to the existing extra_tests/snippets/stdlib_pickle.py (+41). It preserves the existing allocation/copy handling and private-name lookup.

• Traverse the owned read, readline, and peek slots, the buffer iterator, and every occupied memo slot, including sparse keys and repeated identities held by distinct strong slots.
• Traverse the memo proxy's reference to its Unpickler.
• Clear Unpickler outgoing references into the collector's deferred-drop vector and invalidate initialization state.
• On successful reinitialization, replace the old state/configuration and retire their references with the old memo after releasing the state lock.

No Lib changes, new snippet files, Rust Python-behavior tests, collector changes, or CPython-only decorator changes are included.

Bounded validation

Source-bound candidate built with the main Ruff pin 9290a884; compiler/codegen/VM/stdlib/pylib/root were explicitly invalidated and actually recompiled. All tracked source hashes and symlink targets matched the reviewed source after building. Prior sealed binaries were preserved.

An external review harness ran each mode in a separate fresh process, with a fixed 20-second deadline, 512 MiB sampled RSS, 6 GiB address-space limit, core dumps disabled, and resource reserve guards:

• Direct reference identities/multiplicity: candidate read/readline/peek/buffers/memo/proxy counts are 1/1/1/1/2/1. Main baseline reports 0/0/0/0/0/0.
• One tiny memo, stream, or buffers cycle through a memo proxy, tested separately with callback-free weakrefs and exactly three collections: candidate passes all three; main baseline fails their release assertions.
• Successful ordinary reinitialization replaces stream/memo/buffers, releases old holders after three collections, and loads a locally serialized three-integer list: candidate and baseline both pass. This observation does not prove lock-drop ordering; that conclusion comes from source review.

Two existing canonical CPython 3.15.0 compatibility methods, test_priming_unpickler_memo and test_reusing_unpickler_objects, pass on baseline, candidate and CPython, two tests and zero skips each. Their unchanged methods and setup were AST-selected into an external harness with _pickle bindings and only io, _pickle, and unittest globals. No broad pickle test-module import or whole-suite execution occurred.

There were no crashes, hangs, guard stops, automatic retries, or weakened assertions. Inputs were locally serialized ordinary values; no malformed streams, stress/OOM, native-memory probes, reentrant finalizers or concurrent mutation were tested.

The existing pickle snippet now retains the stream and buffers cycle regressions. Canonical persistent-load reference-cycle and buffer-content tests do not cover these payload cycles. Only the new definitions and loop body were AST-selected and validated, one case per fresh guarded process: baseline fails both release assertions, while CPython and candidate pass both. The complete pre-existing snippet was not executed because its unrelated oversized-length cases are outside this local gate. Its required full pytest gate remains unrun.

CPython 3.15.0 oracle qualification

The pinned oracle does not pass every harness assertion. Its direct edge counts are 1/1/1/1/0/1, failing the two-slot memo assertion. A separately bounded memo.copy() observation returns an empty dict immediately after assignment on CPython, whereas main RustPython preserves keys 0 and 17 with both holder identities.

Source inspection of pinned _pickle.c explains this: the dict memo setter writes values into the old memo, then cleans that memo and installs the newly allocated empty array. Consequently the oracle's memo-cycle pass is vacuous for the intended cycle. We preserve this discrepancy rather than weaken the Rust ownership invariant or claim full oracle parity. Oracle stream/buffers/reinit modes pass independently.

Safety and scope qualifications

• Collector clear is invoked after world restart, under the existing exclusive-unreachable-object/deferred-drop contract. This patch inherits that contract and these tests do not prove it. It is not justified by stop-the-world alone.
• The derived memo proxy's clear remains a no-op. Clearing Unpickler outgoing edges breaks the reviewed cycles; this is not complete CPython proxy-clear parity.
• Deferred retirement applies to the successful __init__ path only. Failure-path locals can still drop under the state lock; no reentrant-finalizer claim is made.
• No hard API dependency on Prepare interpreter-local runtime state and thread lifecycle for GC isolation #8944 was found, but coordination on GC architecture and exclusivity assumptions remains necessary.

Additional gates

Passed under the same build resource guard:

• cargo check --locked --offline -p rustpython-stdlib --features threading (128s).
• cargo check --locked --offline -p rustpython-stdlib (non-threaded, default features; 54s).
• cargo clippy --locked --offline -p rustpython-stdlib --all-targets --features threading -- -D warnings (218s).
• prek run --files crates/stdlib/src/pickle.rs (4s; no source changes).

All 3,631 source manifest entries matched after the native gates. The subsequently approved snippet-only addition is the sole manifest difference from the source-bound binary. Native source and sealed binary remain unchanged. Normal hooks were then rerun on both changed files and passed in 2s without rewrites; the two added cases were validated against that same sealed binary.

The required broader workspace/CAPI runtime suites and release pickle-module command have not been run and are not waived. This bounded validation is not a collector/threading soundness proof.

AI assistance: OpenAI assistant assisted with source review, implementation preparation and bounded validation.