Add Extension Type support by jcrist · Pull Request #31 · msgspec/msgspec

jcrist · 2021-06-11T23:57:21Z

This adds support for both encoding and decoding msgpack extension types. This comes with 2 additional apis:

A new msgspec.Ext type, representing the tuple of (code: int, data: bytes_like) that describes a msgpack extension.
A new ext_hook parameter to msgspec.Decoder/msgspec.decode.

By default msgspec will deserialize all extension objects into msgspec.Ext objects. However, if a user specifies a ext_hook to msgspec.Decoder/msgspec.decode, this callback will be used instead to deserialize the extension type. Together with the existing default hook in msgspec.Encoder/msgspec.encode, this lets users fully add extra types to be serialized/deserialized.

For example, say we wanted to define a new extension type with code 42 that contains a buffer of data serialized via pickle. If we wanted to apply this to all non-builtin types serialized by msgspec, we could do this as follows:

import msgspec
import pickle
from typing import Any

def default(obj: Any) -> Any:
    """Serialize the object using pickle, and mark it as an extension type with code 42"""
    return msgspec.Ext(42, pickle.dumps(obj))

def ext_hook(code: int, data: memoryview) -> Any:
    if code == 42:
        return pickle.loads(data)
    raise TypeError(f"Unknown extension type with code {code}")

enc = msgspec.Encoder(default=default)
dec = msgspec.Decoder(ext_hook=ext_hook)

buf = enc.encode(range)  # functions normally aren't serializable with msgspec
out = dec.decode(buf)
assert out == range

Note that ext_hook has the signature ext_hook(code: int, data: memoryview) -> Any. Here data is a memoryview into larger message buffer. This lets us avoid an extra copy when forwarding the data buffer to ext_hook, but it means that if the returned object holds on to a reference to that buffer (instead of doing a copy itself) then the larger message will persist in memory until that object is freed. For most use cases of ext_hook I wouldn't expect this issue to come up, but it's worth noting. The performance improvement from avoiding the copy is worth it to add a tiny rare footgun IMO.

Fixes #26.

Still a few todos:

Make Ext accept a memoryview for data, rather than just bytes or bytearray
Add docs

Also adds tests

This takes an optional callable of the form: ```python def ext_hook(code: int, data: memoryview) -> Any: pass ``` If provided, this will be used to decode any extension types found in a message (except those that are marked via type-annotation to remain as `Ext` objects). Note that `data` is a memoryview into the larger message buffer, minimizing data copies. To prevent the whole message buffer from persisting in memory for too long, the user should ensure they don't keep a reference to the `data` buffer in the decoded object. Also renames `ExtType` to `Ext`.

jcrist · 2021-06-12T00:02:03Z

~~Hmmm, looks like there's a segfault somewhere, haven't hit that locally. Will debug further later.~~

Edit: this was due to python 3.8 seemingly not fully supporting vectorcall constructors for all builtin types. This worked for struct types since they used a custom metaclass (so the call was defined on the class's type), but doesn't work for vectorcall defined on the class itself until python 3.9

Vectorcall for constructors doesn't work universally in python 3.8 (afaict). Dropping for now.

`MS_CONSTR_STR_REGEX` slot clear was missing. It was not properly traversed and freed. Now it is. It used to fail with memory sanitized with: ``` ==2637==ERROR: AddressSanitizer: attempting free on address which was not malloc()-ed: 0x50e000e8e350 in thread T0 #0 0x7f07a10fc4d8 in free ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:52 msgspec#1 0x7f079c42ddf4 in TypeNode_Free src/msgspec/_core.c:3474 msgspec#2 0x7f079c42dbdf in TypeNode_Free src/msgspec/_core.c:3468 msgspec#3 0x7f079c55e2eb in JSONDecoder_dealloc src/msgspec/_core.c:17235 msgspec#4 0x7f07a0999132 in _PyEval_EvalFrameDefault Python/bytecodes.c:1398 msgspec#5 0x7f07a0a15f73 in _PyObject_VectorcallTstate Include/internal/pycore_call.h:92 msgspec#6 0x7f07a0a15f73 in method_vectorcall Objects/classobject.c:69 msgspec#7 0x7f07a098d82f in _PyEval_EvalFrameDefault Python/bytecodes.c:3263 msgspec#8 0x7f07a0a14e6c in _PyFunction_Vectorcall Objects/call.c:419 msgspec#9 0x7f07a0a14e6c in _PyObject_FastCallDictTstate Objects/call.c:144 msgspec#10 0x7f07a0a14e6c in _PyObject_Call_Prepend Objects/call.c:508 msgspec#11 0x7f07a0ac3381 in slot_tp_call Objects/typeobject.c:8791 msgspec#12 0x7f07a0a14206 in _PyObject_MakeTpCall Objects/call.c:240 msgspec#13 0x7f07a098d20a in _PyEval_EvalFrameDefault Python/bytecodes.c:2715 msgspec#14 0x7f07a0a14e6c in _PyFunction_Vectorcall Objects/call.c:419 msgspec#15 0x7f07a0a14e6c in _PyObject_FastCallDictTstate Objects/call.c:144 msgspec#16 0x7f07a0a14e6c in _PyObject_Call_Prepend Objects/call.c:508 msgspec#17 0x7f07a0ac3381 in slot_tp_call Objects/typeobject.c:8791 msgspec#18 0x7f07a0a1550e in PyObject_Call (/opt/hostedtoolcache/Python/3.12.15/x64/lib/libpython3.12.so.1.0+0x21550e) (BuildId: d5ce433e66c26342bfaf6b82ab1a95541a8ebade) msgspec#19 0x7f07a098d82f in _PyEval_EvalFrameDefault Python/bytecodes.c:3263 msgspec#20 0x7f07a0a14e6c in _PyFunction_Vectorcall Objects/call.c:419 msgspec#21 0x7f07a0a14e6c in _PyObject_FastCallDictTstate Objects/call.c:144 msgspec#22 0x7f07a0a14e6c in _PyObject_Call_Prepend Objects/call.c:508 msgspec#23 0x7f07a0ac3381 in slot_tp_call Objects/typeobject.c:8791 msgspec#24 0x7f07a0a14206 in _PyObject_MakeTpCall Objects/call.c:240 msgspec#25 0x7f07a098d20a in _PyEval_EvalFrameDefault Python/bytecodes.c:2715 msgspec#26 0x7f07a0a14e6c in _PyFunction_Vectorcall Objects/call.c:419 msgspec#27 0x7f07a0a14e6c in _PyObject_FastCallDictTstate Objects/call.c:144 msgspec#28 0x7f07a0a14e6c in _PyObject_Call_Prepend Objects/call.c:508 msgspec#29 0x7f07a0ac3381 in slot_tp_call Objects/typeobject.c:8791 msgspec#30 0x7f07a0a14206 in _PyObject_MakeTpCall Objects/call.c:240 msgspec#31 0x7f07a098d20a in _PyEval_EvalFrameDefault Python/bytecodes.c:2715 msgspec#32 0x7f07a0a14e6c in _PyFunction_Vectorcall Objects/call.c:419 msgspec#33 0x7f07a0a14e6c in _PyObject_FastCallDictTstate Objects/call.c:144 msgspec#34 0x7f07a0a14e6c in _PyObject_Call_Prepend Objects/call.c:508 msgspec#35 0x7f07a0ac3381 in slot_tp_call Objects/typeobject.c:8791 msgspec#36 0x7f07a0a14206 in _PyObject_MakeTpCall Objects/call.c:240 msgspec#37 0x7f07a098d20a in _PyEval_EvalFrameDefault Python/bytecodes.c:2715 msgspec#38 0x7f07a0ae8f35 in PyEval_EvalCode Python/ceval.c:580 msgspec#39 0x7f07a0ae7a3c in builtin_exec_impl Python/bltinmodule.c:1093 msgspec#40 0x7f07a0ae7a3c in builtin_exec Python/clinic/bltinmodule.c.h:543 msgspec#41 0x7f07a0a345a2 in cfunction_vectorcall_FASTCALL_KEYWORDS Objects/methodobject.c:438 msgspec#42 0x7f07a0a14532 in _PyObject_VectorcallTstate Include/internal/pycore_call.h:92 msgspec#43 0x7f07a0a14532 in PyObject_Vectorcall Objects/call.c:325 msgspec#44 0x7f07a098d20a in _PyEval_EvalFrameDefault Python/bytecodes.c:2715 msgspec#45 0x7f07a0ae8f35 in PyEval_EvalCode Python/ceval.c:580 msgspec#46 0x7f07a0b09f76 in run_eval_code_obj Python/pythonrun.c:1757 msgspec#47 0x7f07a0b09aaa in run_mod Python/pythonrun.c:1778 msgspec#48 0x7f07a0b099f1 in pyrun_file Python/pythonrun.c:1674 msgspec#49 0x7f07a0b09852 in _PyRun_SimpleFileObject Python/pythonrun.c:459 msgspec#50 0x7f07a0b09667 in _PyRun_AnyFileObject Python/pythonrun.c:78 msgspec#51 0x7f07a0b12943 in pymain_run_file_obj Modules/main.c:361 msgspec#52 0x7f07a0b12943 in pymain_run_file Modules/main.c:380 msgspec#53 0x7f07a0b12943 in pymain_run_python Modules/main.c:634 msgspec#54 0x7f07a0b12943 in Py_RunMain Modules/main.c:714 msgspec#55 0x7f07a0b1243a in Py_BytesMain Modules/main.c:768 msgspec#56 0x7f07a042a1c9 (/lib/x86_64-linux-gnu/libc.so.6+0x2a1c9) (BuildId: a4a7992a8e66555c8141ab2a08a8465ff6e0ea65) msgspec#57 0x7f07a042a28a in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2a28a) (BuildId: a4a7992a8e66555c8141ab2a08a8465ff6e0ea65) msgspec#58 0x558976af7094 in _start (/opt/hostedtoolcache/Python/3.12.15/x64/bin/python3.12+0x1094) (BuildId: d664fd041bd9df9902c326eb80717fa04325aaaa) 0x50e000e8e350 is located 16 bytes inside of 152-byte region [0x50e000e8e340,0x50e000e8e3d8) allocated by thread T0 here: #0 0x7f07a10fd9c7 in malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:69 msgspec#1 0x7f07a0a7c2fc in gc_alloc Modules/gcmodule.c:2330 msgspec#2 0x7f07a0a7c2fc in _PyObject_GC_NewVar Modules/gcmodule.c:2364 SUMMARY: AddressSanitizer: bad-free ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:52 in free ==2637==ABORTING ``` See https://github.com/msgspec/msgspec/actions/runs/38243375356/job/114791328508 Now it is properly cleaned.

jcrist added 8 commits June 9, 2021 08:52

Add ExtType object

1f7ee96

Support ExtType for type specs

9aa4201

Also adds tests

Support serializing ExtType objects

e272bf5

Support deserializing ExtType objects

7ba04c8

Add support for typed decoding of ExtType

77f65e3

Support skipping ExtType fields

41656cd

Add mypy signatures for Ext & ext_hook

aa7e284

jcrist mentioned this pull request Jun 11, 2021

[FEA] A fallback mechanism #26

Closed

jcrist added 4 commits June 11, 2021 21:54

Debug

0e8e78e

Fixup

c843f26

Vectorcall for constructors doesn't work universally in python 3.8 (afaict). Dropping for now.

Support memoryview in Ext objects

cffa190

Add docs

ef76c5d

jcrist changed the title ~~[WIP] Add Extension Type support~~ Add Extension Type support Jun 12, 2021

jcrist merged commit 7107fef into master Jun 12, 2021

jcrist deleted the exttype branch June 12, 2021 23:51

jcrist mentioned this pull request Jun 28, 2021

Add support for extension types #2

Closed

akotlar mentioned this pull request Sep 13, 2023

Add noqa's to test_ancestry_types.py in order to explicitly check types in ProbabilityInterval bystrogenomics/bystro#266

Merged

Navigation Menu

Repository navigation

Add Extension Type support - #31

Add Extension Type support#31
jcrist merged 12 commits into
masterfrom
exttype

jcrist commented Jun 11, 2021 •

edited

Loading

Uh oh!

jcrist commented Jun 12, 2021 •

edited

Loading

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

jcrist commented Jun 11, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jcrist commented Jun 12, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

jcrist commented Jun 11, 2021 •

edited

Loading

jcrist commented Jun 12, 2021 •

edited

Loading