Skip to content

Add pyo3 module bridging to an embedded CPython - #6301

Draft
youknowone wants to merge 11 commits into
RustPython:mainfrom
youknowone:rustpython-cpython
Draft

youknowone wants to merge 11 commits into
RustPython:mainfrom
youknowone:rustpython-cpython

Conversation

@youknowone

@youknowone youknowone commented Nov 28, 2025 •

Copy link
Copy Markdown
Member

Adds an optional pyo3 module (crate rustpython-module_pyo3, cargo feature pyo3) that loads CPython into the process through PyO3 and lets RustPython code use CPython modules, including C extensions RustPython does not implement.

import pyo3
np = pyo3.import_module("numpy")
a = np.array([1, 2, 3])
print(a * 2, a.sum())

@pyo3.wraps
def max_prec():
    import _decimal
    return _decimal.MAX_PREC
cargo build --release --features pyo3
./target/release/rustpython crates/module_pyo3/tests/test_module_pyo3.py

Module API

  • pyo3.import_module(name) imports a module in the embedded CPython. It does not touch this interpreter's sys.modules or import machinery.
  • pyo3.wraps compiles the decorated function's source in CPython and runs calls there.
  • pyo3.load_embedded(data) is the reduce target used when pickling embedded objects.
  • rustpython_module_pyo3::borrow_module(name, vm) (Rust API) registers a CPython module in sys.modules under its own name, e.g. to use the embedded _ctypes.

Conversion rules

  • None, bool, int, float, complex, str, bytes and tuples of them are copied in both directions.
  • CPython objects arrive as instances of a RustPython wrapper of their class. Attribute access, slots and identity are forwarded. A wrapper's metaclass is the wrapper of the CPython metaclass. An immutable metaclass whose own metaclass is type (e.g. ctypes' PyCStructType) gets type.
  • RustPython list, dict, set, frozenset and bytearray are copied into CPython as the exact built-in, because C code checks them with PyList_Check and similar. CPython's changes to such a copy do not reach the RustPython original.
  • A subclass of those containers is pickled, so e.g. OrderedDict, Counter and defaultdict arrive as the CPython class of the same name. When pickling fails, the subclass is copied as the base built-in. It is never proxied.
  • Subclasses of int, float, complex, str and bytes are copied as the base value.
  • Other buffers are shared through a memoryview.
  • Callables, data descriptors and objects with a number protocol become rustpython.object proxies. The proxy class carries exactly the slots the wrapped class defines, and one proxy class is cached per slot set. Non-slot dunders the class defines (e.g. __array__, __array_ufunc__) are published, and a None definition opts out.
  • Anything else is pickled into CPython when its class overrides __eq__, because only then can a copy not be told apart from the original. It is proxied otherwise. An object compared by identity, such as io.BytesIO, is always proxied, so CPython reads and writes the original.
  • Exceptions cross with their type mapped. The original RustPython exception is restored when it comes back.
  • copy/deepcopy of embedded objects delegate to the embedded copy module. pickle stores the embedded pickle bytes behind pyo3.load_embedded.
  • Embedded warnings.showwarning is delivered to RustPython's warnings while a RustPython thread runs. The unraisable hook is routed the same way.
  • The GIL is released while RustPython code runs and is attached only around CPython calls.

Changes in crates/vm

  • slot_defs.rs: a type with its own call slot no longer inherits the base's vectorcall. A metaclass implementing Callable used to inherit type's vectorcall and bypass its call slot.
  • PyType::new_heap picks the primary base with best_base (most derived solid layout) instead of bases[0]. solid_base/best_base no longer take a VirtualMachine.
  • New PyType::assign_version_tag, which assigns a version tag under the type lock and returns it, or 0 when none can be assigned.

Testing

  • crates/module_pyo3/tests/test_module_pyo3.py covers the bridge using only the stdlib through pyo3. A new pyo3 CI job builds with --features pyo3 against CPython 3.14 and runs it on Linux, macOS and Windows.
  • Locally (Windows, CPython 3.14.2) I also ran test_ctypes with the embedded _ctypes borrowed in place of RustPython's. 4 tests fail: 2 depend on refcounts or on id() being a C address, and the 2 bitfield message tests also fail on CPython 3.14.2 itself.
  • A numpy 2.5.3 probe of 47 cases passes 44. Plain CPython gives the same result for 2 of the misses: the memoryview format 'l' on Windows and det rounding. The third, frombuffer(bytearray) not sharing memory, follows from the copy rule above.
  • A probe of 79 cases ran across pandas, scipy, pillow, lxml, orjson, msgpack, regex, cryptography, pydantic and pyyaml, and 76 pass. The 3 that fail all pass a RustPython dataclass or YAML representer hook, which arrives as a proxy that the library's exact type check rejects.

The module name pyo3 is provisional.

🤖 Generated with Claude Code

@coderabbitai

coderabbitai Bot commented Nov 28, 2025 •

Copy link
Copy Markdown
Contributor

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true
✨ Finishing touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Post copyable unit tests in a comment
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@arihant2math

Copy link
Copy Markdown
Contributor

We could just replace the ctypes implementation with this :). Would make life much easier.

This sort of strays away from the mission of having a pure-rust compiler no?

@youknowone

Copy link
Copy Markdown
Member Author

We could just replace the ctypes implementation with this :). Would make life much easier.

On environments with available CPythons, that makes sense. But we are more likely on wasm targets.

This sort of strays away from the mission of having a pure-rust compiler no?

This will be optional feature. Not every data is pickle-serializable. It hopefully will fill gaps until we have rust implementation for them

Comment thread extra_tests/test_module_pyo3.py Outdated
print("OK!\n")


# Test 3: import_module() with numpy

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now RustPython can run numpy.
Though it will not be that much practical.

@ShaharNaveh ShaharNaveh left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall, very cool!

I do agree that this goes against the project goal in some ways. BUT if we have it behind a feature flag (that's not on by default) then I don't see an issue with letting users choose this functionality.

As an added bonus can we state somewhere in the docs that we have (experimental) support for installing packages that use extension modules (not pure python), like pandas, numpy, pedantic, etc...?

@codspeed

codspeed Bot commented Oct 12, 2026

Copy link
Copy Markdown
Contributor

Merging this PR will not alter performance

✅ 62 untouched benchmarks
⏩ 4 skipped benchmarks1


Comparing youknowone:rustpython-cpython (030efca) with main (85d5956)

Open in CodSpeed

Footnotes

  1. 4 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

@youknowone youknowone changed the title rustpython-cpython Add pyo3 module bridging to an embedded CPython Oct 14, 2026
Comment thread .github/workflows/ci.yaml
with:
python-version: "3.14"

- name: Install macOS dependencies
@bschoenmaeckers

Copy link
Copy Markdown
Contributor

Will this clash with the capi function exported by RustPython?

@youknowone

Copy link
Copy Markdown
Member Author

This is not c api providing part. It uses pyo3 as library to import cpython libs
Would it be still issue?

This is not a serious feature at all anyway. It is always optional and will be turned off any case it makes any problem

@bschoenmaeckers

Copy link
Copy Markdown
Contributor

This is not c api providing part. It uses pyo3 as library to import cpython libs Would it be still issue?

I think it will be a problem, for sure on mac and linux. On these platforms symbols are in a process wide table. So a PyLong_FromLong exported from RustPython will conflict with the PyLong_FromLong from cpython. So only 1 of them wins and be linked to the module loaded into the process. This is the reason why the no_mangle attribute on our capi functions is marked as unsafe. It's UB to override symbols from other libraries in the same binary.

@bschoenmaeckers

Copy link
Copy Markdown
Contributor

So we have to make sure a user does not enable both the c-api and pyo3 feature.

copyslot_if_none copied the base's vectorcall even when the type defines
its own call slot, and inherit_from_mro took the vectorcall of the first
MRO entry that has one rather than of the entry providing the call slot.
A metaclass implementing Callable thus inherited type's vectorcall, which
bypasses the metaclass's call slot.

Assisted-by: Claude Code:claude-opus-5-5
Add the rustpython-module_pyo3 crate, enabled by the new `pyo3` feature,
which provides the `pyo3` module:

- import_module() imports a CPython module.
- CPython objects appear as `pyo3.ref` proxies (`pyo3.callable_ref` for
  callables) forwarding attributes, calls, operators, comparisons, item
  access, iteration, hashing, buffers and the descriptor protocol.
- CPython classes appear as wrapper classes with the `pyo3.pyo3_type`
  metaclass. A class statement deriving from a wrapped class creates the
  class in CPython. CPython's built-in classes map to RustPython's.
- Immutable built-in values are copied both ways; list, dict, set,
  frozenset and bytearray are copied into CPython; buffers are shared both
  ways; RustPython callables and descriptors are proxied into CPython;
  other RustPython objects are pickled into CPython when possible and
  proxied otherwise.
- Exceptions become the nearest built-in exception class on the other
  side and come back as the original object.
- wraps runs a decorated function in CPython.

Uses pyo3 0.29.

Assisted-by: Claude Code:claude-opus-5-5
PyType::new_heap used bases[0] as the primary base. It now uses the base
best_base selects, the one with the most derived solid layout, so a plain
mixin listed before a base with a native layout no longer becomes the
primary base. solid_base and best_base no longer take a VirtualMachine.

Assisted-by: Claude
Assisted-by: Grok
A CPython instance now reaches RustPython as an instance of the wrapper
class of its CPython class, whose special methods come from the mirrored
class namespace, and a wrapper class's metaclass is the wrapper of the
CPython metaclass. pyo3.ref is the slot-less payload base of wrapper
classes; pyo3.proxy, with every protocol slot, remains for objects whose
class derives from a built-in layout.

A zero-dimensional CPython buffer spans its whole length.

Assisted-by: Claude
Assisted-by: Grok
The pyo3 module now initializes CPython itself with use_environment
disabled, so PYTHONHOME and PYTHONPATH meant for RustPython are not read
by it, and routes CPython's sys.unraisablehook to RustPython's while a
RustPython interpreter runs on the thread.

Wrapper classes support weak references when the CPython class does and
mirror the CPython __mro__. RustPython objects supporting the number
protocol get reverse proxies with the matching slots, and instances of
RustPython subclasses of built-in value types reach CPython as values of
that built-in type.

Assisted-by: Claude
Assisted-by: Grok
A str or bytes converted while calling into CPython is copied once per
call, and the copies live as long as the call's result instead of being
attached to ctypes objects through _objects. A RustPython buffer exported
to CPython is released when CPython releases its last view.

Classes created through a wrapped CPython metaclass, including through
its __call__, __new__ and __prepare__, are created in CPython.

Assisted-by: Claude
Assisted-by: Grok
A RustPython object passed to CPython gets a proxy class carrying exactly
the number slots its class defines: __index__, __float__, __complex__,
unary - + ~ abs, and the binary operators with their reflected forms. One
proxy class is cached per set of slots. pyo3.proxy defines __complex__.

The conversion docs state that list, dict, set and bytearray are copied
and that CPython's changes to the copy do not reach the original. Module
docs and comments are updated to the current behavior.

Assisted-by: Grok
Assisted-by: Claude
The method assigns a version tag under the type lock when the type has
none and returns the tag, or 0 when none can be assigned.

Assisted-by: Grok
Assisted-by: Claude
- copy.copy and copy.deepcopy of an embedded object run the embedded
  copy module. Pickling one stores the embedded pickle bytes behind
  pyo3.load_embedded.
- The wrapper of an immutable embedded metaclass whose metaclass is type
  has type as its metaclass and copies IMMUTABLETYPE; mutable ones keep
  forwarding through pyo3.pyo3_type.
- rustpython.object proxies publish the dunder methods the wrapped class
  defines, decided by the first defining class in the MRO and cached per
  class by version tag, and fill the sequence length slot.
- Embedded warnings are shown through this interpreter's warnings module
  while a RustPython interpreter runs on the thread, and warning
  categories map to the matching built-in classes.

Assisted-by: Grok
Assisted-by: Claude
…d iteration

- A subclass of list, dict, set, frozenset or bytearray is pickled into
  CPython, keeping its class, and copied as the base built-in when
  pickling fails. It is never proxied.
- The pickle fallback applies only to objects whose class overrides
  __eq__. Objects compared by identity are proxied.
- Keyword names of calls into CPython are interned.
- rustpython.object proxies get __iter__ and __next__ only when the
  wrapped class defines those slots.
- A CPython exception with missing or empty args is rebuilt with str() of
  the original as its message.

Assisted-by: Grok
Assisted-by: Claude
pytest in extra_tests collected test_module_pyo3.py with the host
CPython and failed on `import pyo3`. The test now lives in
crates/module_pyo3/tests, and a new pyo3 job builds RustPython with the
pyo3 feature against CPython 3.14 from actions/setup-python and runs it
on Linux, macOS and Windows.

Assisted-by: Grok
Assisted-by: Claude
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants