Skip to content

perf: avoid building error strings on the **kwargs call path - #15

Draft
moreal wants to merge 1633 commits into
mainfrom
perf/call-ex-kwargs
Draft

moreal wants to merge 1633 commits into
mainfrom
perf/call-ex-kwargs

Conversation

@moreal

@moreal moreal commented Sep 2, 2026

Copy link
Copy Markdown
Owner

What changed

Frame::collect_ex_args handles CALL_FUNCTION_EX (f(*args, **kwargs)). Two costs on the success path were paid unconditionally even though they only exist to build error messages:

  1. object_function_str(callable) was always computed before iterating the **kwargs mapping. This helper does two getattrs (__qualname__, __module__), two intern-pool lookups, and a Wtf8Buf concatenation — but its result (func_str) is only used in the TypeError raised when the ** argument is not a mapping. iterate_mapping_keys now takes func_str as a lazy FnOnce() -> Wtf8Buf closure, invoked only inside the error branch.

  2. iterate_mapping_keys had no exact-dict fast path. For every **kwargs call it looked up the keys method through the MRO, built a bound method, called it to materialize a dict_keys view, iterated that view, and then re-hashed every key via kw_obj.get_item(key) to fetch the corresponding value. iterate_mapping_keys now checks mapping.downcast_ref_if_exact::<PyDict>(vm) first and, when the mapping is an exact dict (not a subclass), iterates its entries directly via &Py<PyDict>'s IntoIterator, yielding (key, value) pairs with no method lookup, no bound-method/view allocation, and no re-hash. Non-dict mappings and dict subclasses (which may override keys/__getitem__) keep going through the original keys() protocol path unchanged.

The key_handler callback signature changed from FnMut(PyObjectRef) -> PyResult<()> to FnMut(PyObjectRef, PyObjectRef) -> PyResult<()> so both the fast path and the protocol path can hand over (key, value) without a redundant get_item call in the caller.

Semantics preserved: the keywords must be strings and argument after ** must be a mapping error messages and their raise ordering are unchanged (verified byte-for-byte against the unmodified binary below). Duplicate-keyword TypeErrors are raised later in the call-binding path and are untouched by this change. Insertion order of keys is preserved (both the dict fast path and keys() iterate in insertion order).

Why it is faster

f(**kw) with a small literal dict previously paid a getattr(__qualname__) + getattr(__module__) + Wtf8Buf concat on every call — for an error message never used on success — plus a keys MRO lookup, a bound-method call, a dict_keys iterator, and a get_item re-hash per key. The fast path removes all of that for the common case of calling with a plain dict.

Benchmark scripts

/tmp/choco-pi/perf/perf-call-ex-kwargs/ (not part of this commit; scripts + run_ab.sh, an interleaved-per-iteration A/B min/median runner):

  • bench_kwargs_small.py — f(a, b, c) called as f(**{"a":1,"b":2,"c":3}) in a loop (targets both findings).
  • bench_passthrough.py — def w(*a, **k): return f(*a, **k) wrapper (the audit's "15x vs CPython" case).
  • bench_kwargs_large.py — call with a 30-key kwargs dict (amplifies the per-key re-hash cost).
  • bench_kwargs_dict_subclass.py — kwargs is a dict subclass; must NOT take the fast path and must NOT regress.
  • bench_star_tuple_control.py — f(*tup) neutral control; this code path (the tuple branch) is unaffected by the change.

Each script prints its own time.perf_counter() wall time. run_ab.sh alternates baseline/patched per iteration and reports min/median across the run.

A/B results

Interleaved baseline (/tmp/perf-baseline/rustpython, unmodified base commit) vs. patched release binaries, same machine. This machine is shared with several other concurrent agents doing cargo builds, so results were re-measured once background load (concurrent rustc/cargo process count) dropped to a handful of processes; the reported numbers below are from that quieter window (n=21 reps for the first four, n=25 for kwargs_large, n=11 for the control).

Benchmark Baseline min Baseline median Patched min Patched median Speedup (min / median)
kwargs_small (3 kwargs) 1.110s 2.238s 0.931s 1.786s 1.19x / 1.25x
passthrough (*a, **k wrapper) 0.669s 0.969s 0.470s 0.759s 1.42x / 1.28x
kwargs_large (30 kwargs) 0.750s 0.769s 0.681s 0.701s 1.10x / 1.10x
kwargs_dict_subclass (control, must not regress) 0.844s 1.368s 0.633s 0.962s 1.33x / 1.42x
star_tuple_control (f(*tup), unaffected code path) 0.0986s 0.1042s 0.0973s 0.1029s 1.01x / 1.01x

All five benchmarks show a consistent, reproducible improvement for the **kwargs paths, and the neutral *tup control is flat (~1.0x) as expected since that branch is untouched by this change. Interestingly kwargs_dict_subclass also sped up: even though it can't take the exact-dict fast path, it still benefits from finding (1) — the lazy func_str closure.

Noise caveat: on the first measurement pass, while ~39 concurrent rustc/cargo processes from sibling agents were running on this shared machine, kwargs_large's min statistic showed an apparent 0.76x regression (baseline min 1.51s vs. patched min 1.99s) even though its median showed a modest 1.08x improvement. Min-of-N is noise-sensitive to a single unlucky sample when background contention swings between 1 and 4+ seconds per run; re-measuring the same benchmark alone once system load dropped back to a handful of processes reproduced a clean, consistent ~1.10x win (both min and median) shown in the table above. This is called out explicitly rather than reporting only the favorable number.

Test results

  • cargo clippy --workspace --all-targets --exclude rustpython_wasm --exclude rustpython-venvlauncher --exclude rustpython-capi: clean, no new warnings.
  • cargo test --workspace --exclude rustpython_wasm --exclude rustpython-venvlauncher --exclude rustpython-capi: all suites pass, 0 failures (re-run twice to rule out a one-off flake — see below).
  • (cd crates/capi && cargo test): 103 passed, 0 failed.
  • cargo run --release -- -m test test_extcall, test_call, test_dict, test_funcattrs, test_types: all SUCCESS.
  • extra_tests: pytest -q — 443 passed, 1 pre-existing failure (test_rustpython_stdlib_sqlite, ModuleNotFoundError: No module named '_sqlite3'); confirmed the unmodified baseline binary has the same gap (_sqlite3 isn't built into this release binary), unrelated to this change.

Transient flake noted during verification: a first full cargo test --workspace run hit a one-off failure in 8 rustpython-stdlib::_opcode::tests snapshot tests (these tests instantiate a full VM and run dis.dis() through it, and were competing with the same ~39-process background build storm mentioned above). Reverting to the unmodified base commit and re-running the identical command passed cleanly; re-applying the patch and running the same test module 5 times in a row (and the full workspace suite in a follow-up run) passed every time. Treated as environmental flakiness under extreme shared-machine contention, not a regression from this change.

Caveats

  • Benchmarks measure wall-clock time inside the RustPython process itself (time.perf_counter()), alternated baseline/patched per iteration; absolute numbers are machine- and load-dependent, but the relative min/median speedups were reproducible across multiple runs once background load was low.

dependabot Bot and others added 30 commits July 21, 2026 01:36
Bumps [which](https://github.com/harryfei/which-rs) from 8.0.4 to 8.0.5.
- [Release notes](https://github.com/harryfei/which-rs/releases)
- [Changelog](https://github.com/harryfei/which-rs/blob/master/CHANGELOG.md)
- [Commits](harryfei/which-rs@8.0.4...8.0.5)

---
updated-dependencies:
- dependency-name: which
  dependency-version: 8.0.5
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [taiki-e/install-action](https://github.com/taiki-e/install-action) from 2.82.9 to 2.83.2.
- [Release notes](https://github.com/taiki-e/install-action/releases)
- [Changelog](https://github.com/taiki-e/install-action/blob/main/CHANGELOG.md)
- [Commits](taiki-e/install-action@4684b84...43aecc8)

---
updated-dependencies:
- dependency-name: taiki-e/install-action
  dependency-version: 2.83.2
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [bitflags](https://github.com/bitflags/bitflags) from 2.13.0 to 2.13.1.
- [Release notes](https://github.com/bitflags/bitflags/releases)
- [Changelog](https://github.com/bitflags/bitflags/blob/main/CHANGELOG.md)
- [Commits](bitflags/bitflags@2.13.0...2.13.1)

---
updated-dependencies:
- dependency-name: bitflags
  dependency-version: 2.13.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [syn](https://github.com/dtolnay/syn) from 2.0.118 to 2.0.119.
- [Release notes](https://github.com/dtolnay/syn/releases)
- [Commits](dtolnay/syn@2.0.118...2.0.119)

---
updated-dependencies:
- dependency-name: syn
  dependency-version: 2.0.119
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* Add more unicode functions

Review

Add unicode decode functions

Add more functions

* Remove

* Add helper
* bytes: support keyword arguments in hex()

* bytes: unmark test_memoryview_hex_separator
…_ATTR specializations (RustPython#8350)

* Use dict keys-version stamps and entry-index hints in attr specializations

Add a keys-version stamp to Dict: a globally unique u32 assigned lazily
and reset on any key-set change (new key, deletion, clear). Value-only
updates keep the stamp.

- LoadAttrMethodWithValues / LoadAttrNondescriptorWithValues: cache the
  instance dict's stamp in the pointer cache and skip the shadow probe
  while the stamp matches.
- LoadAttrWithHint: cache the entry index at specialization time; a hit
  is an identity check on the entry key instead of a hash probe, with
  self-refresh on miss.
- StoreAttrWithHint / StoreAttrInstanceValue: replace values through the
  cached entry index via set_item_with_hint.

Fixes youknowone#42

Assisted-by: Claude

* Share keys-version stamps between dicts with identical layouts

Derive the keys-version stamp from a shape — the ordered interned-key
sequence of a hole-free dict — instead of always allocating a unique
stamp. Dicts with identical layouts (instances of the same class built
by the same __init__) now carry equal stamps, so a LOAD_ATTR cache
entry populated by one instance skips the shadow probe for every
instance sharing the layout.

Shapes are held in a fixed-size lock-free table keyed by interned key
addresses; dicts with holes, non-interned keys, or more than 32 keys
fall back to dict-unique stamps.

Assisted-by: Claude

* Address clippy and review feedback for keys-version stamps

- Use BuildHasher::hash_one for shape hashing (clippy manual_hash_one)
- Fix set_item_with_hint doc: a refreshed hint is returned on any hint
  miss, not only when the hinted slot was vacant
- Route both holey-dict instances through a single LOAD_ATTR cache site
  in the snippet test

Assisted-by: Claude

* Restore try_read_cached_descriptor doc comment to its function

The doc block and #[inline] were left attached to store_attr_dict_hinted
when it was inserted above try_read_cached_descriptor.

Assisted-by: Claude

* Specialize LoadAttrWithHint even when the entry index exceeds u16

hint_for_key returns None both for an absent key and for a present key
whose entry index does not fit in u16, so very large instance dicts
stopped specializing entirely. Use get_item_opt_refresh_hint to decide
presence, degrading an unrepresentable hint to 0: the handler then
keeps taking its full-probe fallback path.

Assisted-by: Claude
Part of RustPython#8245 to reduce the amount of work needed to review. I
simplified the embedded nul errors by forwarding to the implementations
in `vm::exceptions`.
* Keep StringIO text operations CPython-compatible

Constraint: email.feedparser depends on StringIO(newline='') recognizing CR, LF, and CRLF boundaries.
Rejected: an email-specific parser workaround | StringIO is the shared root cause.
Confidence: high
Scope-risk: moderate
Directive: Keep internal storage byte-based; convert only at StringIO API boundaries.
Tested: prek run --all-files; test_memoryio; test_email; full workspace verification by contributor
Assisted-by: Codex:gpt-5.6-sol

* Run restored StringIO conformance tests

Constraint: upstream StringIO tests already cover the repaired behavior.
Rejected: new RustPython-only regression tests | existing CPython coverage is sufficient.
Confidence: high
Scope-risk: narrow
Directive: Remove expected-failure markers only while the inherited tests pass.
Tested: prek run --all-files; test_memoryio; test_email
Assisted-by: Codex:gpt-5.6-sol

* Keep StringIO newline handling consistent

Constraint: StringIO must apply its newline mode consistently when constructing, writing, and reading text.\nRejected: a readline-only fix | it leaves CR and CRLF modes internally inconsistent.\nConfidence: high\nScope-risk: moderate\nDirective: Keep StringIO buffer contents valid WTF-8 before using unchecked views.\nTested: prek run --all-files; cargo clippy -p rustpython-vm -- -D warnings; test_memoryio; test_shlex; test_email; workspace excluding rustpython-capi; full workspace manually verified by contributor\nNot-tested: local rustpython-capi workspace test crashes with a pre-existing macOS SIGSEGV\nAssisted-by: Codex:gpt-5.6-sol
…thon#8362)

When a Connection is explicitly closed via con.close(), subsequent
operations (cursor(), commit(), rollback(), create_function(), etc.)
should raise ProgrammingError with 'Cannot operate on a closed database.'
to match CPython behaviour.

Previously, _db_lock() always returned 'Base Connection.__init__ not
called.' when self.db was None, without distinguishing between a
connection that was never initialised (subclass before __init__) and
one that was initialised and then explicitly closed.

Fix: inspect the initialized atomic flag — if True but db is None,
the connection was closed; if False, it was never initialised.

Assisted-by: GitHub Copilot:claude-sonnet-4-6
…stPython#8364)

Row(cursor, data) raised ValueError when cursor.description was None.
CPython allows this case and returns an empty key list.

- Use empty tuple when description is None instead of raising
- Include the key name in the IndexError when a string key is not found

Assisted-by: GitHub Copilot:claude-sonnet-4-6
* Fix AST identifier interning

Ensure parsed name and function identifier fields use interned strings,
matching CPython behavior.

Assisted-by: Codex:gpt-5.6-sol

* Apply PEP 8 import ordering
Raise OverflowError when a finite f64 value overflows while being converted for the struct f format. Remove the expected-failure marker from the corresponding regression test.

Assisted-by: Codex:gpt-5
* time: use C altzone when available

* Modified to also check TARGET_CC/CC_target.
Bumps [fast-uri](https://github.com/fastify/fast-uri) from 3.1.2 to 3.1.4.
- [Release notes](https://github.com/fastify/fast-uri/releases)
- [Commits](fastify/fast-uri@v3.1.2...v3.1.4)

---
updated-dependencies:
- dependency-name: fast-uri
  dependency-version: 3.1.4
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…ustPython#8363)

CPython raises ProgrammingError with "Binding N is a named parameter"
when a query uses named placeholders (, , ) but
the caller passes a sequence instead of a mapping.

RustPython's bind_parameters_sequence() did not check whether each
binding slot is a named parameter, so no error was raised.

Fix: call sqlite3_bind_parameter_name() for each slot in
bind_parameters_sequence(). If the first byte of the returned name is
not '?' (i.e. it is a named placeholder), raise ProgrammingError before
attempting to bind.

Assisted-by: GitHub Copilot:claude-sonnet-4-6
…stPython#8367)

Bumps [postcss](https://github.com/postcss/postcss) from 8.5.10 to 8.5.23.
- [Release notes](https://github.com/postcss/postcss/releases)
- [Changelog](https://github.com/postcss/postcss/blob/main/CHANGELOG.md)
- [Commits](postcss/postcss@8.5.10...8.5.23)

---
updated-dependencies:
- dependency-name: postcss
  dependency-version: 8.5.23
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…#8368)

Bumps [http-proxy-middleware](https://github.com/chimurai/http-proxy-middleware) from 2.0.9 to 2.0.10.
- [Release notes](https://github.com/chimurai/http-proxy-middleware/releases)
- [Changelog](https://github.com/chimurai/http-proxy-middleware/blob/v2.0.10/CHANGELOG.md)
- [Commits](chimurai/http-proxy-middleware@v2.0.9...v2.0.10)

---
updated-dependencies:
- dependency-name: http-proxy-middleware
  dependency-version: 2.0.10
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…ustPython#8370)

Bumps [shell-quote](https://github.com/ljharb/shell-quote) from 1.8.4 to 1.10.0.
- [Changelog](https://github.com/ljharb/shell-quote/blob/main/CHANGELOG.md)
- [Commits](ljharb/shell-quote@v1.8.4...v1.10.0)

---
updated-dependencies:
- dependency-name: shell-quote
  dependency-version: 1.10.0
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…ustPython#8359)

* Replace malformed unicode error helper calls with the _real variants

new_unicode_decode_error / new_unicode_encode_error build the exception
from a bare message without running the initializer, so the result has
none of the five attributes a unicode error must carry and str() renders
as an empty string. Convert the non-Windows call sites in csv, socket,
getlogin and the fs-path decoders to the _real constructors, passing the
source bytes/str and the failing offset from the captured Utf8Error.

The Windows-gated sites (nt, mbcs/oem codecs) and the sites whose source
object is not reachable (uname, array) are left for follow-ups.

Assisted-by: Claude Code:claude-fable-5

* Report the full invalid UTF-8 span in the converted decode errors

The conversions used valid_up_to() + 1 for the decode-error end offset,
which under-reports multi-byte invalid sequences. Take the span from the
Utf8Error instead: valid_up_to() + error_len(), or the input length for a
truncated sequence (error_len() == None), matching CPython.

Assisted-by: Claude Code:claude-opus-4-8
* Implement termios.tcgetwinsize/tcsetwinsize

Adds ioctl(TIOCGWINSZ/TIOCSWINSZ) wrappers in host_env and the
corresponding Python-facing functions in the termios stdlib module,
removing the associated expectedFailure markers in test_termios.py.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01RbzcYLnX5tMfTM9BP7FDTT

* Use ret < 0 convention for ioctl error checks in termios winsize

POSIX ioctl only guarantees -1 on failure, not exactly 0 on success;
matches the existing check_libc_neg convention used elsewhere in
host_env (e.g. fcntl.rs, posix.rs::get_terminal_size).

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01RbzcYLnX5tMfTM9BP7FDTT

* Reject non-sequence size in tcsetwinsize

extract_elements_with also accepted dicts (treating keys as elements).
Switch to try_sequence(), matching CPython's PySequence_Check, so only
real sequences (tuple/list/etc.) are accepted.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01RbzcYLnX5tMfTM9BP7FDTT

* Use `core::mem::zeroed` instead of `std::mem::zeroed`

`std::mem` just re-exports `core::mem`, but clippy prefers importing
from core when there's no OS dependency. This was breaking the wasm CI build.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01RbzcYLnX5tMfTM9BP7FDTT

* Use try_index for tcsetwinsize row/col conversion

CPython's PyLong_AsLong calls __index__ on non-int objects before
converting, so downcast_ref::<PyInt> was stricter than CPython.
try_index matches that behavior and is more concise.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01RbzcYLnX5tMfTM9BP7FDTT

* Mark test_fork/test_spawn_doesnt_hang as expected failures

Both were hidden by a skipIf on tty.tcgetwinsize, which now exists.
Root cause: pty.fork() calls os.login_tty(), which isn't implemented,
so the forked child crashes before the test body runs.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01RbzcYLnX5tMfTM9BP7FDTT

* Use unittest.skip instead of expectedFailure for pty.fork() tests

expectedFailure still runs the test body, so pty.fork()'s real fork()
still happens and the child crashes on missing os.login_tty inside the
parallel test runner's worker process, corrupting its JSON reporting
channel ("worker bug"). skip prevents the method from running at all,
avoiding that. Verified locally with -j 2.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01RbzcYLnX5tMfTM9BP7FDTT

* Clarify skip reason for pty.fork() tests

* Simplify tcgetwinsize/tcsetwinsize fd argument with Fildes destructuring

* Explain why pty.fork() tests use skip instead of expectedFailure

---------

Co-authored-by: Claude Sonnet 5 <[email protected]>
Support the CPython 3.13+ format spec syntax that puts a grouping option
after the precision, so `format(1234.56789, '.6,f')` gives
'1234.567,890'. The integer and fractional parts group independently and
may use different separators (`,.6_f` -> '1,234.567_890'), so the spec
carries a separate `frac_grouping_option`.

`parse_precision` now consumes the separator that follows the precision
digits and rejects a `,`/`_` mix. A repeated separator is deliberately
left in the spec so the trailing-text check reports it, which is how
CPython arrives at a different message there. A dot followed by neither
digits nor a separator now raises "Format specifier missing precision"
rather than an unrelated error.

Fraction digits group away from the decimal point, so the last group may
be short, and any exponent or percent tail is left intact. The
separators count toward the field width, so zero padding of the integer
part reserves room for them.

'n' takes its separators from the locale and so cannot carry one. The
complex locale path rewrites 'n' to 'g' before delegating, so it has to
validate first; that also makes `format(1+2j, ',n')` fail as it does in
CPython, which it previously did not.

Reference (CPython 3.14), Python/formatter_unicode.c:
- parsing: https://github.com/python/cpython/blob/3.14/Python/formatter_unicode.c#L257-L299
- 'n' rejection: https://github.com/python/cpython/blob/3.14/Python/formatter_unicode.c#L361-L367
- number split: https://github.com/python/cpython/blob/3.14/Python/formatter_unicode.c#L488-L516
- zero-pad width: https://github.com/python/cpython/blob/3.14/Python/formatter_unicode.c#L604-L606

Remove `@unittest.expectedFailure` from tests that now pass
(2 in test_format, 1 in test_float).

Assisted-by: Claude:claude-opus-5

Co-authored-by: Claude Opus 5 <[email protected]>
codspeed Bot and others added 24 commits August 30, 2026 15:18
Alias criterion to codspeed-criterion-compat so the existing benchmark suites can be measured by CodSpeed with CPU simulation on every pull request, and add the workflow running them.

Co-authored-by: CodSpeed Bot <[email protected]>
)

* uuid: expose stable hardware node

Cache native MAC address detection and advertise an extractable node only when a hardware address is available. This lets uuid.getnode() use the native UUID path while preserving its existing fallbacks.

Assisted-by: Codex:gpt-5.6-sol

* Prefer universally administered MAC addresses

Match uuid.getnode()'s documented interface-selection rule by scanning available addresses before falling back to a locally administered one. Ignore zero addresses and cover the selection behavior with Rust unit tests.

Assisted-by: Codex:gpt-5.6-sol
* Implement PEP 667 frame locals semantics

Assisted-by: Codex:GPT-5

* Fix C API test CString import

Assisted-by: Codex:GPT-5

* Address PEP 667 review findings

Assisted-by: Codex:GPT-5
* Fix tuple subclass preservation in type bases

Assisted-by: Codex:gpt-5.6-sol

* Refine type bases representation

Store validated base classes in the original typed tuple, while retaining a bootstrap vector only until a Python tuple can be materialized.

Assisted-by: gpt-5.6-sol
Assisted-by: Codex:gpt-5.6-sol

* Bootstrap tuple type with type hierarchy

Initialize tuple and the canonical empty tuple alongside type and object so every type stores an actual typed bases tuple from construction. Remove the bootstrap bases representation and keep raw initialization Miri-safe.

Assisted-by: Codex:gpt-5.6-sol
…ite (RustPython#8621)

* Leave a byte that starts no UTF-8 sequence at the end of a console write

`find_last_utf8_boundary` read every byte at or above 0xc0 as the start of a
sequence, giving 0xf8..=0xff an expected length of 4 and cutting the tail back
to before it.  `_find_last_utf8_boundary` stops the four-byte range at 0xf8 and
returns the full length for anything above it, leaving the byte for
MultiByteToWideChar to answer with U+FFFD.

`write_console_utf8` therefore reported fewer bytes written than it was given
whenever one of those bytes fell in the last three, and for a buffer that began
with one it cut the length to 0, converted nothing and returned Ok(0) -- a
write that never advances.

The scan also ran a fourth iteration the C loop does not; the arms happen to
answer the same there, but the bound is now the same 3.

* Drop the expectedFailure on test_winconsoleio's test_write

The marker went stale with the boundary fix in the parent commit.  The test
writes `b'\xff'*10` and `data + b'\xff'` to CONOUT$ and asserts the returned
count, which is the case `find_last_utf8_boundary` was answering wrong, so
regrtest now reports it as an unexpected success.
…c__` (RustPython#8614)

* Strip the signature prefix from builtin __doc__

A function's signature and its documentation share one string, separated
by `\n--\n\n`, as in CPython. __text_signature__ read the front of it,
but __doc__ returned the whole thing:

    >>> len.__doc__
    'len(obj, /)\n--\n\nReturn the number of items in a container.'   # was
    'Return the number of items in a container.'                      # now

44 of the 46 builtin functions leaked the prefix.

get_doc_from_internal_doc already stripped it but was reachable only from
PyType.__doc__. Split it the way CPython does, into doc_without_signature
(_PyType_DocWithoutSignature) and a wrapper mapping an empty result to
None (_PyType_GetDocFromInternalDoc), and call it from PyNativeFunction
and PyMethodDescriptor as well.

Assisted-by: Claude Code:claude-opus-5

* Terminate a generated __text_signature__ that carries no documentation

A signature and its documentation share one string, separated by
`\n--\n\n`, and readers find the signature by searching for that marker.
A #[pyfunction] with no doc comment stored the bare signature, which no
reader could parse:

    >>> time.clock_getres.__doc__
    'clock_getres(clk_id, /)'
    >>> time.clock_getres.__text_signature__
    None

Emit the marker in that case too, as Argument Clinic does for an
undocumented function. 136 of the 915 functions reachable from the
importable modules were affected.

__doc__ now reports None for them, matching CPython, because nothing
follows the marker.

Assisted-by: Claude Code:claude-opus-5

* Report a __text_signature__ for every method

A #[pymethod] built its doc only when the method carried a doc comment,
so one without a comment dropped the generated signature too:

    >>> list.append.__text_signature__
    None

Build it from either part, as #[pyfunction] does. All 1101 method
descriptors reachable from the builtin types now report a signature, up
from 1, and 713 of the 1030 that CPython also describes match it exactly.

A method that takes its receiver as an ordinary first argument instead of
`&self` reported that argument by name, so binding the method could not
drop it:

    >>> inspect.signature([].__dir__)
    (obj, /)     # was
    ()           # now

Mark it $self, or $type for a classmethod, as CPython does. 310 methods
were affected.

test_unbound_builtin_method_noargs and test_bound_builtin_method_noargs
pass now.

Assisted-by: Claude Code:claude-opus-5

* Drop the expectedFailure on test_simple_completion

rlcompleter closes the parenthesis for a callable whose signature takes
no parameters, and leaves it open when the signature cannot be read:

    def _callable_postfix(self, val, word):
        if callable(val):
            word += "("
            try:
                if not inspect.signature(val).parameters:
                    word += ")"
            except ValueError:
                pass

os.getpid reports `()` since the generated signature carries the `--`
terminator, so completing `os.getpid` yields `os.getpid()` and the test
passes.

Assisted-by: Claude Code:claude-opus-5

* Name every #[pymethod] receiver zelf

A method that needs an owned handle cannot take `&self`, so it takes the
receiver as its first argument and names it `zelf`, `self` being a Rust
keyword. 24 of the 163 such methods named it something else: `instance`
(11), `obj` (4), `exc` (3), `_self` (3), `_instance` (2).

Rename them. The generated signature is unaffected either way, since the
macro marks the receiver by position rather than by name.

Assisted-by: Claude Code:claude-opus-5

* Drop the unreachable zelf name check in func_sig

The receiver is now marked by position, so the name it carries no longer
matters, and no path reaches this check with an argument named zelf:

- a #[pymethod] or #[pyclassmethod] without a `&self` receiver has its
  first argument replaced by the marker before the name is read
- a `&self` receiver is handled as syn::FnArg::Receiver
- a #[pymethod(raw)] is passed to static_raw_func, whose PyNativeFn bound
  fixes its signature to (&VirtualMachine, FuncArgs)
- no #[pystaticmethod] or #[pyfunction] names an argument zelf
- #[pymember], #[pygetset] and #[pyslot] never reach func_sig

Every generated signature is byte for byte identical without it.

Assisted-by: Claude Code:claude-opus-5

* Keep the argument bundle when marking the receiver

A #[pyclassmethod] that takes FuncArgs receives the class inside the
bundle along with every other argument, but the marker was spent on the
bundle and the arguments went unreported:

    >>> object.__subclasshook__.__text_signature__
    '($type, /)'                    # was, claims it takes nothing
    '($type, *args, **kwargs)'      # now

Report both, the shape CPython uses for __new__. Two methods take their
arguments this way, __subclasshook__ on object and on type.

Assisted-by: Claude Code:claude-opus-5
* common: add PyPy CJK codec engines

Assisted-by: Codex:GPT-5

* common: mark CJK mapping tables generated

Assisted-by: Codex:GPT-5

* common: add reusable byte and text engines

Assisted-by: Codex:GPT-5

* common: record opt-in engine dependencies

Assisted-by: Codex:GPT-5

* stdlib: add `_multibytecodec` and the six `_codecs_*` modules

Wire the CJK codec engines in `rustpython-common` up to the VM. `_multibytecodec`
holds `MultibyteCodec`, `MultibyteIncrementalEncoder`, `MultibyteIncrementalDecoder`,
`MultibyteStreamReader` and `MultibyteStreamWriter`, the encode/decode drivers and
the error-handler protocol; each `_codecs_*` module exposes `getcodec` for the
encodings of its region.

`rustpython-stdlib` now enables the `cjk-codecs` feature of `rustpython-common`.

Add `cjk::decode_reset`, which the decoder `reset()` methods need. It differs from
building a fresh decoder state: ISO 2022 keeps its G1..G3 designations and the
escape-throughout flag, and resets only G0 and the shifted flag.

Drop the markers from the CJK codec tests and from the `test_codecs`, `test_io`,
`test_email`, `test_zipfile` and `test_traceback` cases that needed a CJK codec.
`test_codecs.BasicUnicodeTest.{test_basics,test_decoder_state}` stay expected
failures for an unrelated reason: `codecs.charmap_encode` rejects a `None` mapping.

Assisted-by: Claude Code:claude-opus-5

* stdlib: use the byte and text engines from `rustpython-common`

`binascii`, `_json` and `_socket`'s address converters now call the engines in
`rustpython-common` instead of carrying their own copies, and `rustpython-stdlib`
enables the `binascii`, `inet` and `json` features for them. That leaves
`base64` and `crc32fast` with no caller in `rustpython-stdlib`, and
`crates/stdlib/src/json/machinery.rs`, the last of the json_in_type-derived code,
goes away with its last caller.

The address converters change behaviour. `inet_aton` reads the lenient forms
(`127.1`, `16909060`, `0x7f.1`, `0177.0.0.1`), `inet_pton` rejects an octet with a
redundant leading zero and a group of five hex digits, `inet_ntop` writes the
IPv4-compatible form as `::c000:201`, and an address family with no converter
raises `EAFNOSUPPORT` instead of a bare `OSError`.

Give the JSON engine the closing-quote scan and the allocation-free `\uXXXX`
escape it needs to carry `_json`'s workload. Against the code it replaces,
`json.dumps` runs 2.3x faster with `ensure_ascii=True` and 1.7x without it, and
`json.loads` 1.2x.

`binascii.a2b_base64` now reports the messages `binascii.c` does: no
`error decoding base64: ` prefix, and the data-character count in parentheses.

Assisted-by: Claude Code:claude-opus-5

* test: enable the Windows cp932 codec case

The shared CJK engine now satisfies the Windows code-page adapter, so keeping this test marked as an expected failure turns its success into a CI failure.

Assisted-by: Codex:GPT-5
Bumps the system-configuration group with 1 update in the / directory: [system-configuration](https://github.com/mullvad/system-configuration-rs).


Updates `system-configuration` from 0.7.0 to 0.8.0
- [Changelog](https://github.com/mullvad/system-configuration-rs/blob/main/CHANGELOG.md)
- [Commits](mullvad/system-configuration-rs@v0.7.0...v0.8.0)

---
updated-dependencies:
- dependency-name: system-configuration
  dependency-version: 0.8.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: system-configuration
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…ustPython#8626)

Bumps [github/gh-aw/actions/setup](https://github.com/github/gh-aw) from 0.86.2 to 0.87.3.
- [Release notes](https://github.com/github/gh-aw/releases)
- [Changelog](https://github.com/github/gh-aw/blob/main/CHANGELOG.md)
- [Commits](github/gh-aw@48e5fa3...466b8ad)

---
updated-dependencies:
- dependency-name: github/gh-aw/actions/setup
  dependency-version: 0.87.3
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…ython#8625)

Bumps [https://github.com/astral-sh/ruff-pre-commit](https://github.com/astral-sh/ruff-pre-commit) from v0.16.3 to 0.16.4.
- [Release notes](https://github.com/astral-sh/ruff-pre-commit/releases)
- [Commits](astral-sh/ruff-pre-commit@v0.16.3...v0.16.4)

---
updated-dependencies:
- dependency-name: https://github.com/astral-sh/ruff-pre-commit
  dependency-version: 0.16.4
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…on#8630)

* Name the function a failed argument binding was binding for

A built-in call that binds badly reported neither the function nor, for a
method, the right numbers: `self` was prepended before binding, so it was
counted as both an expected parameter and a given argument. `[].append()`
read `expected at least 2 arguments, got 1`.

`Callee` carries what a message says about the function: the name, and
whether the leading argument fills the instance parameter. `PyNativeFn`
takes one, `PyNativeFunction` and `PyMethodDescriptor` supply it, and the
four method-descriptor call specializations in `frame.rs` do the same.
`bind_for` subtracts the instance from both numbers.

The messages now read as their counterparts do:

- `_PyArg_CheckPositional`: `find expected at least 1 argument, got 0`,
  with the singular when the count is one.
- `_PyArg_UnpackKeywords`: `split() got an unexpected keyword argument
  'bogus'`, and `cast() missing required argument 'format' (pos 1)` for a
  parameter a call may pass by name. A positional-only parameter is only
  ever counted, so it keeps the first form.

A binding that happens where the name isn't known leaves it off, the way
`_PyArg_Parser.fname` is NULL.

`test_find_etc_raise_correct_error_messages` now passes in both
`test_bytes` and `string_tests`.

Assisted-by: Claude

* Name the type whose constructor or initializer bound the arguments

`Constructor::slot_new` and `Initializer::slot_init` bound their arguments
without a name, and so did the slots that override them, so a constructor
that was called wrongly said only `expected 1 argument, got 0`.

Both defaults and every override now name the type the slot was written
for, not the subclass being constructed and not the module `tp_name`
carries: `float expected at most 1 argument, got 2`, and `deque expected
at most 2 arguments, got 3` for a `deque` subclass.

`slice()` and `range()` counted their own arguments and raised messages of
their own; they now raise the one `_PyArg_CheckPositional` raises, which
makes `test_range_constructor_error_messages` pass.

Assisted-by: Claude
Preserve logical memoryview descriptors across interpreter transfers, make channel-end release idempotent, and scope runtime-owned interpreter cleanup to the creating top-level runtime. Re-read owned interpreters during shutdown so finalizers cannot leave newly-created children behind.

Add regression coverage for sliced and cast memoryviews, repeated channel release, and multiple embedded runtimes.

Assisted-by: Codex:GPT-5
Related: RustPython#8599

My original parser was monolithic and inflexible. It worked as intended
for the derived data files, but anything more complicated required hacky
code. For example, the original parser always expected to build a vector
of values, yet sometimes we required other data structures such as
BTreeMaps.

I split up the parser into helper functions that are both cleaner and
more flexible. The actual parser should still yield the same results -
this doesn't "fix" anything yet. However, this is preliminary work for
fixing more of UCD since we will need a more flexible parser for fixes
like the linked PR.
Add dependency-free CPU-bound workloads adapted from pyperformance 1.14.0 to cover interpreter, standard-library serialization, regex, numeric, and garbage-collection paths under the existing Criterion and CodSpeed harnesses. Keep inputs deterministic and scale expensive cases for simulation while checking that benchmark execution does not leak interpreter state.

Assisted-by: Codex:gpt-5.6-sol
* common: share the zlib stream engine

Move the VM-independent zlib-rs stream owner and state machine into a feature-gated rustpython-common module. Make rustpython-stdlib adapt that engine instead of carrying its own flate2 implementation, including native stream copy/deepcopy, Z_BLOCK flushes, and wbits=0 decompression.

Assisted-by: Codex:GPT-5

* common: share the lzma stream engine

Move the VM-independent liblzma stream owner, filter properties, and compression state into a target-gated rustpython-common module. Keep rustpython-stdlib as the Python object and exception adapter, and remove its direct xz dependencies and lzma-only generic compressor machinery.

Assisted-by: Codex:GPT-5

* common: address compression engine reviews
Bumps [base64](https://github.com/marshallpierce/rust-base64) from 0.22.1 to 0.23.1.
- [Changelog](https://github.com/marshallpierce/rust-base64/blob/master/RELEASE-NOTES.md)
- [Commits](marshallpierce/rust-base64@v0.22.1...v0.23.1)

---
updated-dependencies:
- dependency-name: base64
  dependency-version: 0.23.1
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [libsqlite3-sys](https://github.com/rusqlite/rusqlite) from 0.38.1 to 0.38.2.
- [Release notes](https://github.com/rusqlite/rusqlite/releases)
- [Changelog](https://github.com/rusqlite/rusqlite/blob/master/Changelog.md)
- [Commits](https://github.com/rusqlite/rusqlite/commits)

---
updated-dependencies:
- dependency-name: libsqlite3-sys
  dependency-version: 0.38.2
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…r themselves (RustPython#8634)

* Raise the shared argument-binding errors from the slots that count for themselves

Sites all over the tree checked their own argument counts and keywords and
wrote the message by hand, so they drifted: `attrgetter expected 1 argument,
got 0.` carried a full stop, `TypeVar() got unexpected keyword argument(s):
bogus` named neither the keyword the way its counterpart does nor the
function the way `typevar()` is named, and `_csv`'s dialect parser handed a
whole formatted sentence to `InvalidKeywordArgument`, which then wrapped it
in another one.

`Callee::arity_error` and `Callee::unexpected_keyword` are now public, and
these sites raise through them, so there is one place the wording lives.
`FrameLocalsProxy` counts its arguments before reading its keywords, and
`min`/`max` do too, which is the order `framelocalsproxy_new` and `min_max`
check in. `weakref.ref`, `GenericAlias`, `frozenset`, `attrgetter`,
`itemgetter`, `islice`, `start_new_thread`, `TextIOWrapper`, `AttributeError`,
`NameError`, `TypeVar`, `ParamSpec`, `TypeVarTuple` and `TypeAliasType` all
follow.

Of 25 calls measured against CPython 3.14.6, 21 now produce the identical
string. The four left over need what `Callee` cannot carry: `weakref.proxy`
is a function of its own there rather than a type, `frozenset` names the
subclass being constructed, and `TextIOWrapper` names its first parameter
and counts before converting.

Assisted-by: Claude

* Give the argument-binding errors the constructors the rest of them have

Raising one of these meant writing `Callee::of::<Self>(vm).arity_error(1..=3,
0, vm)` — the vm twice, and a type to know about before you could say what
you meant. Every other error in the tree is a `vm.new_*_error`.

So the message texts move out of `Callee` into functions the vm can call, and
the vm gets `new_arity_type_error` and `new_unexpected_keyword_type_error`
alongside `new_unsupported_bin_op_error` and the rest. A slot names itself
with the `NAME` its class already carries:

    return Err(vm.new_arity_type_error(Self::NAME, 1..=3, 0));
    return Err(vm.new_unexpected_keyword_type_error(Some("typevar"), &key));

The keyword one takes the name as an `Option` because the parser sometimes
has none of its own, which is what `_PyArg_Parser.fname` being NULL means and
what `_csv`'s dialect parser wants.

`bind_for` and `check_kwargs_empty_for` take anything a `Callee` converts
from, so the slots pass `Self::NAME` there too. `Callee` itself is left to the
binding machinery, which is the only place that needs to carry a name around
rather than use one.

No message changes: 22 of 26 calls still match CPython 3.14.6 exactly, and the
43-call set still stands at 23.

Assisted-by: Claude
`collect_ex_args` (CALL_FUNCTION_EX) always computed `object_function_str`
for the callable before iterating a `**kwargs` mapping, even though the
result is only used by the "argument after ** must be a mapping" error.
It now takes that as a lazy closure. `iterate_mapping_keys` also gained
an exact-`dict` fast path that walks entries directly instead of going
through the `keys()` bound-method protocol and re-hashing each key via
`get_item`; dict subclasses still use the general protocol path.

Assisted-by: choco-pi:anthropic/claude-sonnet-4-5
Signed-off-by: Lee Dogeon <[email protected]>
@moreal moreal closed this Sep 3, 2026
@moreal moreal reopened this Sep 3, 2026
@codspeed

codspeed Bot commented Sep 3, 2026

Copy link
Copy Markdown

Merging this PR will improve performance by ×2.5

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 1 improved benchmark
✅ 65 untouched benchmarks

Performance Changes

Benchmark BASE HEAD Efficiency
⚡ gc_collect.py[rustpython] 157.3 ms 64 ms ×2.5

Tip

Curious why performance improved? Comment @codspeedbot explain why performance improved on this PR, or directly use the CodSpeed MCP with your agent.


Comparing perf/call-ex-kwargs (db50809) with main (287dcd9)

Open in CodSpeed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.