Repository navigation
Add pyo3 module bridging to an embedded CPython - #6301
youknowone wants to merge 11 commits into
Conversation
|
Important Draft PR not reviewedDraft PRs are not automatically reviewed by default.
To automatically review draft PRs, update your CodeRabbit configuration: reviews:
auto_review:
drafts: true✨ Finishing touches
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
We could just replace the This sort of strays away from the mission of having a pure-rust compiler no? |
On environments with available CPythons, that makes sense. But we are more likely on wasm targets.
This will be optional feature. Not every data is pickle-serializable. It hopefully will fill gaps until we have rust implementation for them |
60c2403 to
6699c70
Compare
| print("OK!\n") | ||
|
|
||
|
|
||
| # Test 3: import_module() with numpy |
There was a problem hiding this comment.
Now RustPython can run numpy.
Though it will not be that much practical.
ShaharNaveh
left a comment
There was a problem hiding this comment.
Overall, very cool!
I do agree that this goes against the project goal in some ways. BUT if we have it behind a feature flag (that's not on by default) then I don't see an issue with letting users choose this functionality.
As an added bonus can we state somewhere in the docs that we have (experimental) support for installing packages that use extension modules (not pure python), like pandas, numpy, pedantic, etc...?
6699c70 to
361d09b
Compare
361d09b to
030efca
Compare
Merging this PR will not alter performance
Comparing Footnotes
|
030efca to
44502a5
Compare
| with: | ||
| python-version: "3.14" | ||
|
|
||
| - name: Install macOS dependencies |
|
Will this clash with the capi function exported by RustPython? |
|
This is not c api providing part. It uses pyo3 as library to import cpython libs This is not a serious feature at all anyway. It is always optional and will be turned off any case it makes any problem |
I think it will be a problem, for sure on mac and linux. On these platforms symbols are in a process wide table. So a |
|
So we have to make sure a user does not enable both the |
44502a5 to
4b62dc1
Compare
copyslot_if_none copied the base's vectorcall even when the type defines its own call slot, and inherit_from_mro took the vectorcall of the first MRO entry that has one rather than of the entry providing the call slot. A metaclass implementing Callable thus inherited type's vectorcall, which bypasses the metaclass's call slot. Assisted-by: Claude Code:claude-opus-5-5
Add the rustpython-module_pyo3 crate, enabled by the new `pyo3` feature, which provides the `pyo3` module: - import_module() imports a CPython module. - CPython objects appear as `pyo3.ref` proxies (`pyo3.callable_ref` for callables) forwarding attributes, calls, operators, comparisons, item access, iteration, hashing, buffers and the descriptor protocol. - CPython classes appear as wrapper classes with the `pyo3.pyo3_type` metaclass. A class statement deriving from a wrapped class creates the class in CPython. CPython's built-in classes map to RustPython's. - Immutable built-in values are copied both ways; list, dict, set, frozenset and bytearray are copied into CPython; buffers are shared both ways; RustPython callables and descriptors are proxied into CPython; other RustPython objects are pickled into CPython when possible and proxied otherwise. - Exceptions become the nearest built-in exception class on the other side and come back as the original object. - wraps runs a decorated function in CPython. Uses pyo3 0.29. Assisted-by: Claude Code:claude-opus-5-5
PyType::new_heap used bases[0] as the primary base. It now uses the base best_base selects, the one with the most derived solid layout, so a plain mixin listed before a base with a native layout no longer becomes the primary base. solid_base and best_base no longer take a VirtualMachine. Assisted-by: Claude Assisted-by: Grok
A CPython instance now reaches RustPython as an instance of the wrapper class of its CPython class, whose special methods come from the mirrored class namespace, and a wrapper class's metaclass is the wrapper of the CPython metaclass. pyo3.ref is the slot-less payload base of wrapper classes; pyo3.proxy, with every protocol slot, remains for objects whose class derives from a built-in layout. A zero-dimensional CPython buffer spans its whole length. Assisted-by: Claude Assisted-by: Grok
The pyo3 module now initializes CPython itself with use_environment disabled, so PYTHONHOME and PYTHONPATH meant for RustPython are not read by it, and routes CPython's sys.unraisablehook to RustPython's while a RustPython interpreter runs on the thread. Wrapper classes support weak references when the CPython class does and mirror the CPython __mro__. RustPython objects supporting the number protocol get reverse proxies with the matching slots, and instances of RustPython subclasses of built-in value types reach CPython as values of that built-in type. Assisted-by: Claude Assisted-by: Grok
A str or bytes converted while calling into CPython is copied once per call, and the copies live as long as the call's result instead of being attached to ctypes objects through _objects. A RustPython buffer exported to CPython is released when CPython releases its last view. Classes created through a wrapped CPython metaclass, including through its __call__, __new__ and __prepare__, are created in CPython. Assisted-by: Claude Assisted-by: Grok
A RustPython object passed to CPython gets a proxy class carrying exactly the number slots its class defines: __index__, __float__, __complex__, unary - + ~ abs, and the binary operators with their reflected forms. One proxy class is cached per set of slots. pyo3.proxy defines __complex__. The conversion docs state that list, dict, set and bytearray are copied and that CPython's changes to the copy do not reach the original. Module docs and comments are updated to the current behavior. Assisted-by: Grok Assisted-by: Claude
The method assigns a version tag under the type lock when the type has none and returns the tag, or 0 when none can be assigned. Assisted-by: Grok Assisted-by: Claude
- copy.copy and copy.deepcopy of an embedded object run the embedded copy module. Pickling one stores the embedded pickle bytes behind pyo3.load_embedded. - The wrapper of an immutable embedded metaclass whose metaclass is type has type as its metaclass and copies IMMUTABLETYPE; mutable ones keep forwarding through pyo3.pyo3_type. - rustpython.object proxies publish the dunder methods the wrapped class defines, decided by the first defining class in the MRO and cached per class by version tag, and fill the sequence length slot. - Embedded warnings are shown through this interpreter's warnings module while a RustPython interpreter runs on the thread, and warning categories map to the matching built-in classes. Assisted-by: Grok Assisted-by: Claude
…d iteration - A subclass of list, dict, set, frozenset or bytearray is pickled into CPython, keeping its class, and copied as the base built-in when pickling fails. It is never proxied. - The pickle fallback applies only to objects whose class overrides __eq__. Objects compared by identity are proxied. - Keyword names of calls into CPython are interned. - rustpython.object proxies get __iter__ and __next__ only when the wrapped class defines those slots. - A CPython exception with missing or empty args is rebuilt with str() of the original as its message. Assisted-by: Grok Assisted-by: Claude
pytest in extra_tests collected test_module_pyo3.py with the host CPython and failed on `import pyo3`. The test now lives in crates/module_pyo3/tests, and a new pyo3 job builds RustPython with the pyo3 feature against CPython 3.14 from actions/setup-python and runs it on Linux, macOS and Windows. Assisted-by: Grok Assisted-by: Claude
4b62dc1 to
03971b2
Compare
Adds an optional
pyo3module (craterustpython-module_pyo3, cargo featurepyo3) that loads CPython into the process through PyO3 and lets RustPython code use CPython modules, including C extensions RustPython does not implement.Module API
pyo3.import_module(name)imports a module in the embedded CPython. It does not touch this interpreter'ssys.modulesor import machinery.pyo3.wrapscompiles the decorated function's source in CPython and runs calls there.pyo3.load_embedded(data)is the reduce target used when pickling embedded objects.rustpython_module_pyo3::borrow_module(name, vm)(Rust API) registers a CPython module insys.modulesunder its own name, e.g. to use the embedded_ctypes.Conversion rules
None,bool,int,float,complex,str,bytesand tuples of them are copied in both directions.type(e.g. ctypes'PyCStructType) getstype.list,dict,set,frozensetandbytearrayare copied into CPython as the exact built-in, because C code checks them withPyList_Checkand similar. CPython's changes to such a copy do not reach the RustPython original.OrderedDict,Counteranddefaultdictarrive as the CPython class of the same name. When pickling fails, the subclass is copied as the base built-in. It is never proxied.int,float,complex,strandbytesare copied as the base value.memoryview.rustpython.objectproxies. The proxy class carries exactly the slots the wrapped class defines, and one proxy class is cached per slot set. Non-slot dunders the class defines (e.g.__array__,__array_ufunc__) are published, and aNonedefinition opts out.__eq__, because only then can a copy not be told apart from the original. It is proxied otherwise. An object compared by identity, such asio.BytesIO, is always proxied, so CPython reads and writes the original.copy/deepcopyof embedded objects delegate to the embeddedcopymodule.picklestores the embedded pickle bytes behindpyo3.load_embedded.warnings.showwarningis delivered to RustPython'swarningswhile a RustPython thread runs. The unraisable hook is routed the same way.Changes in
crates/vmslot_defs.rs: a type with its own call slot no longer inherits the base's vectorcall. A metaclass implementingCallableused to inherittype's vectorcall and bypass its call slot.PyType::new_heappicks the primary base withbest_base(most derived solid layout) instead ofbases[0].solid_base/best_baseno longer take aVirtualMachine.PyType::assign_version_tag, which assigns a version tag under the type lock and returns it, or 0 when none can be assigned.Testing
crates/module_pyo3/tests/test_module_pyo3.pycovers the bridge using only the stdlib throughpyo3. A newpyo3CI job builds with--features pyo3against CPython 3.14 and runs it on Linux, macOS and Windows.test_ctypeswith the embedded_ctypesborrowed in place of RustPython's. 4 tests fail: 2 depend on refcounts or onid()being a C address, and the 2 bitfield message tests also fail on CPython 3.14.2 itself.'l'on Windows anddetrounding. The third,frombuffer(bytearray)not sharing memory, follows from the copy rule above.The module name
pyo3is provisional.🤖 Generated with Claude Code