Implement PEP 798 comprehension unpacking - #9342

Split the PEP 798 implementation from #8954.

Current draft status

This PR implements RustPython's AST/compiler execution support, not a new Ruff grammar implementation. Ruff upstream completed comprehension unpacking in astral-sh/ruff#25104, released in 0.15.14. The Ruff revision already on RustPython main (9290a8849542134d83f77e9360e7f0b93897fd93) parses the valid list, set, dictionary, and generator-expression forms. Waiting for upstream PEP 798 support or upgrading to Ruff 0.17.2 is not necessary for those forms.

Hosted correctness CI for 247eb615 failed in the CPython test jobs on Linux, macOS, Windows, and WASI; their build and snippet steps passed. The Linux log identifies two failing modules: test_exceptions retains the obsolete expectation that [*x for x in xs] raises SyntaxError, and test_unpack_ex has fourteen malformed-input diagnostic mismatches. These are canonical-test migration and CPython-compatible error-message/location gaps, not missing valid-syntax parsing.

The successor candidate now migrates test_exceptions.py as a whole canonical file through the existing scripts/update_lib workflow and adds a narrow RustPython-side normalization of existing parser diagnostics. Ruff source, its Cargo revision, and Cargo.lock are unchanged. The full warning/type-comment producer extension in RustPython/ruff#7 is not required. The latest follow-up fixes all eight remaining malformed-input canonical diagnostics in finite local comparisons; all 30 canonical exec/doctest-mode records now match CPython. Correctness CI for the latest commit ddc36651932bf9f3a3836e23e1085ca364e25870 passed all 24 jobs, and Lib Dependencies Check passed. Surrounding malformed inputs retain the known finite-comparison differences documented below. Whole-file migration also brings other exception/location and resource-sensitive cases; the complete test_exceptions module has not been runtime-validated locally. No selective method copies or new skips/xfails were introduced.

Changes

• Compile list/set comprehension unpacking using the existing extend/update operations, and dictionary comprehension unpacking using DictUpdate.
• Flatten unpacked generator-expression items through an explicit iterator loop, including async generator wrapping. Sent values are not forwarded to the inner iterator.
• Handle unpacking dictionary comprehensions consistently in symbol-table traversal, AST conversion, validation, and generated AST documentation.
• Migrate test_unpack_ex.py, test_unparse.py, and test_exceptions.py as whole canonical files using scripts/update_lib, pinned to CPython d220409a13b77de7a6c30abe16ea90db01528edc.

The Ruff revision already on main parses these forms: starred comprehension elements use Expr::Starred, while dictionary unpacking uses ExprDictComp with no key. This PR implements the RustPython AST/compiler side. It does not change Ruff, VM instruction definitions, unrelated compiler location handling, or interpreter lifecycle code. No custom tests or snippets are added.

Historical validation

Validated original commit: 247eb615c61e3f167245d55618fbfad611b5fd37.

• Normal commit hooks, including Rust/Ruff formatting and lint checks, passed.
• cargo check --offline --locked -p rustpython-codegen -p rustpython-vm --lib passed.
• cargo clippy --offline --locked -p rustpython-codegen -p rustpython-vm --lib -- -D warnings passed.
• cargo build --offline --locked --bin rustpython succeeded. The exact Cargo artifact was independently copied and sealed with its committed standard library.
• 37 unchanged canonical examples from two bounded test_unpack_ex doctest segments passed under ordinary checkers. Their raw output matched pinned CPython exactly.
• Six finite synchronous/asynchronous observations matched pinned CPython byte-for-byte, including generator send() behavior.
• Seven unchanged canonical AST/unparse methods passed individually, with no skips.

The full test_unpack_ex module and broader parser, lifecycle, GC and fuzz suites were not run locally. Local validation used the unchanged commit above; a source-only merge check against main 5bccf31156996d42882485ba70f17edbc77a713e was conflict-free and retained the same nine-file patch.

Historical successor local validation (partial)

Validated candidate tree: 21011e3035c2f5cce41935d2bd719e274212fb5e (successor of 247eb615; separate from the historical results above). Final local commit: 6ff465bd7c678c94ce455e7f35ca51e5b166b52d, tree be9bba0359348c12c2ac366aeca31b2d7cc5148f. Normal commit hooks passed. The final tree differs from the validated tree only by two source-reviewed, token-equivalent rustfmt line wraps; the six build gates and paired runtime checks below were run before that formatting-only change, not rerun on the final commit.

• Compiler and VM checks/strict Clippy, and the minimal RustPython binary build/strict Clippy, passed: six bounded gates.
• Compared the same 34 finite compile-only inputs against freshly built official CPython d220409a13b77de7a6c30abe16ea90db01528edc. No compiled case bytecode was executed; canonical expected messages were unchanged.
• All six corrected canonical conditional-unpacking diagnostics (examples 87–92) matched CPython's message, source text and location fields. Including the existing passing example98, this is 7/15 canonical diagnostic messages, not a passing full module.
• 17/34 inputs matched every recorded field. The 19 boundary controls contain 10 exact matches and 9 remaining mismatches. Narrowing the double-star normalization removed two unintended new hints (subsequent dictionary spread and dictionary value), but did not make those two diagnostics CPython-compatible. All other results within these 34 inputs were unchanged, and the CPython oracle was stable.
• Eight canonical cases remain unresolved: dictionary unpacking in list comprehension (93), parenthesized/call generator expressions (94/95), starred/double-starred dictionary keys (96/97), double-starred dictionary value (99), parenthesized double-star expression (100), and unpacked dictionary comprehension followed by a colon (101). These need further ownership/range/diagnostic work; no parser changes or expectation relaxations were used to hide them.

The whole-file test_exceptions update is included but its full runtime suite was not run. No full-suite success, global non-regression, or green successor CI is claimed. This PR remains a draft.

Historical mapping-rest CI follow-up (RustPython only; Ruff unchanged)

• Correct the mapping-rest diagnostic to point at the rest capture rather than the following mapping item. The adapter uses the existing parser error and adjacent token information, and abstains when there are additional parser errors.
• Retire the stale SyntaxErrorTests.test_encodings expected-failure marker through the existing whole-file update_lib --patches workflow. Linux, macOS, and Windows CI all reported this test as an unexpected success. Preserve the existing Windows-only test_windows_message marker; add no skip or expected failure.
• No Ruff source, revision, version, Cargo manifest, or lockfile changes.

Validation:

• Compiler and threading-VM checks/Clippy, plus the minimal binary build/Clippy, passed (six gates).
• Compiled the same 56 finite source inputs, without executing their code, under the preserved previous RustPython binary, the new binary, and canonical CPython 3.15 (d220409a13b77de7a6c30abe16ea90db01528edc). The two CI mapping cases now match CPython exactly; all 34 previous unpacking-diagnostic results are unchanged.
• Strict full-field comparison did not pass: nine mapping outputs changed; eight match CPython exactly. The multiline/comment case now has the correct message and complete location range, but RustPython's unchanged SyntaxError.text serialization adds a trailing newline that CPython omits. The strict comparison and this residual are retained, not relaxed or hidden by dropping the case.
• The eight previously identified unpacking-diagnostic failures remain outside this follow-up. These focused results do not establish full CI success.

Unpacking diagnostic follow-up

Validated commit: ddc36651932bf9f3a3836e23e1085ca364e25870.

• Fix the eight remaining canonical unpacking diagnostics on the RustPython side. Ruff source, version, dependency pins, and test expectations are unchanged.
• Use retained AST/operator/error ownership for the list/generator/dictionary cases. For the call-generator case only, validate one exactly owned argument slice with the same fixed Ruff parser and options; do not infer clause validity from recovery tokens or implement another grammar.
• All 30 canonical cases 87–101 (exec and doctest-style single/LF modes) now match the CPython 3.15 oracle in every captured SyntaxError field. This describes local finite validation. Separately, hosted correctness CI for this exact commit passed all 24 configured jobs, including Linux, macOS, Windows, and WASI test jobs; it does not establish parity for every malformed input.
• Source-bound builds and targeted compiler Clippy (-D warnings) passed. Validation progressed through successive 71-, 85-, and 98-input compile-only comparisons, retaining earlier inputs and prior binaries/results. No previously matching control regressed. The final candidate fully matches 63/98 inputs versus 59/98 for the validated call-generator checkpoint. In that comparison, five rows changed: four fully match CPython; the multiline/comment control now has the correct message and positions but retains the existing serializer's extra LF in SyntaxError.text. The earlier comment-key LF residual also remains. Comparisons use raw fields without normalization.
• Deliberately leave uncertain malformed/nested recovery shapes unchanged rather than broadening diagnostic precedence or weakening tests.
• AI assistance: dot authored and reviewed the changes and performed the stated validation. The runtime does not expose its exact model version.

Commit-hook limitation: cspell alone was omitted for this commit with the maintainer’s explicit one-time approval because its configured npm bootstrap downloads returned HTTP 403. Cspell has not passed. The normal pre-commit hook ran: merge-conflict and rustfmt checks passed; other configured hooks had no applicable files. Hosted correctness CI for this exact commit completed successfully on 2026-10-27 at 02:50 UTC (24/24 jobs); Lib Dependencies Check also passed.