Match tokenizer token metadata with CPython 3.15 - #9445

Summary

• Match CPython 3.15 non-extra INDENT/DEDENT token text and column metadata.
• Match trailing EOF line text, implicit NL placement for final comments/whitespace, and ordinary STRING physical-line spans.
• Retire the verified CTokenizeTest.test_async and TestTokenize.test_newline_and_space_at_the_end_of_the_source_without_newline expected-failure markers through actual scripts/update_lib migrate from CPython d220409a13b77de7a6c30abe16ea90db01528edc. The remaining invalid-syntax and resource-dependent random-files markers remain unchanged.

This is a focused native tokenizer plus its dependent Lib test update, separate from the free-threading aggregate and builtin-construction changes. It does not change lexer grammar, callback ownership or the tokenizer's Reading/Yielding synchronization. The free-threading tokenizer module remains outside this patch.

Validation

• Scoped cargo check and strict Clippy passed; one guarded final executable build passed.
• Final protected executable SHA256: 64e701982b3bad37b03c0a16b68a485682bed9527ba629db51dfa0425aaaa55a, built from main 8c6c94b07174224517b6bf62abab202565382ea7 plus this native change. All 607 recorded native inputs were verified.
• Fourteen complete token-tuple comparisons matched pinned CPython exactly, in both extra-token modes across seven bounded inputs: EOF/newlines, nested dedents, trailing comments/whitespace, and an ordinary multiline string.
• Five complete unchanged canonical methods passed without skips or expected failures: CTokenizeTest.test_async, test_function, test_tabs, test_unicode, and TokenizeTest.test_basic. The additional full canonical newline-space method passed separately. Both complete updater-generated methods then passed together (2 tests, no skips or expected failures).

The follow-up corrects the non-extra INDENT location after an explicit continuation and classifies a complete leading-zero decimal token as NUMBER only in extra-token mode when its exact Ruff diagnostic identifies that token. Both complete canonical methods now pass; their markers were retired through actual update_lib migration.

These finite checks do not establish complete tokenizer or invalid-input parity. No full-module/workspace/CAPI, concurrency, reentry, crash, GC or stress test was run. Existing callback-state ownership and other tokenizer expected failures remain separate work.

AI assistance

OpenAI Codex assisted with implementation, source review, validation and this description.

Follow-up: continuation, leading-zero tokens and UTF-8 EOF line lookup

The original c178 head failed the canonical pyrepl double-width-character test because EOF line lookup used the last byte rather than a UTF-8 character boundary. The follow-up selects the final character's byte offset and shares that lookup between both EOF-line callsites. It does not change Ruff, lexer grammar, callback ownership or the free-threading tokenizer state machine.

Final protected executable SHA-256: 468375918b727a3c74d81772620eba1ca4a25c386c28ad9c3c43b3de15399a5c. Scoped check, strict Clippy and one guarded final build passed; all 607 native inputs were verified. Evidence on this final corrected executable:

• 23 prior full-tuple comparisons matched pinned CPython exactly: the original 14 and nine additional leading-zero/numeric controls.
• 14 of 16 new Unicode/empty-input comparisons matched exactly. Both oracle processes exited normally and all cases terminated. Two pre-existing non-extra-mode differences remain: empty input emits an ENDMARKER rather than no token, and CRLF input differs in NEWLINE spelling/end column and ENDMARKER line text. The failed exact comparison is retained; these two cases are not claimed as passes or silently excluded. The UTF-8 lookup change preserves the previous offsets for empty/ASCII LF endings and does not fix these separate metadata gaps.
• Nine complete canonical tokenizer methods passed, including continuation indentation, leading-zero numbers and the prior regression selections, with no skips or expected failures.
• One complete canonical pyrepl method passed: TestCursorPosition.test_cursor_position_double_width_character, using its synthetic event and mock console. This verifies the exact previously failing test locally; the full pyrepl module was not run locally.
• Actual scripts/update_lib migrate from the pinned source retired only the additional continuation-indentation and leading-zero expected-failure decorators. Both complete generated methods passed on the same protected executable, with no skips or expected failures. The remaining two markers are unchanged.

The extra Unicode comparison failure stopped the initial driver before the canonical selections; the already-reviewed canonical methods were then run separately with their own successful receipts. No blanket Unicode/tokenizer parity or new-head CI success is claimed. No concurrency, FT, deep-nesting, random-file, GC, profiler or crash-stress suite was run.