Skip to content

Use tuple when decoding untyped messagepack arrays when needed - #29

Merged
jcrist merged 2 commits into
masterfrom
use-tuple-for-arrays-if-needed
Jun 8, 2021
Merged

Use tuple when decoding untyped messagepack arrays when needed#29
jcrist merged 2 commits into
masterfrom
use-tuple-for-arrays-if-needed

Conversation

@jcrist

@jcrist jcrist commented Jun 8, 2021 •
edited
Loading

Copy link
Copy Markdown
Member

Previously if a msgpack array was decoded without type information, we'd default to using a list. This meant that messages that used tuple for keys would error on decode without type information (since they'd default to deserializing as lists, which aren't hashable). We now automatically use tuple by default when necessary for decoding an array type (e.g. when the value is part of a key in a dict or set). This allows us to roundtrip more messages successfully without type information, while still relying on standard msgpack types.

An example may clarify this. With this PR, the following behavior occurs:

In [1]: import msgspec

In [2]: data = msgspec.encode({(1, 2): [3, 4]})

In [3]: msgspec.decode(data)  # tuple used for key by default, since it must be hashable
Out[3]: {(1, 2): [3, 4]}

In [4]: data = msgspec.encode({(1, 2), (3, 4)})

In [5]: msgspec.decode(data, type=set)  # same here, since set keys must be hashable
Out[5]: {(1, 2), (3, 4)}

In [6]: data = msgspec.encode((1, 2, 3))

In [7]: msgspec.decode(data)  # list used here though, since the array type doesn't need to be hashable
Out[7]: [1, 2, 3]

In [8]: msgspec.decode(data, type=tuple)  # but type information can still be used
Out[8]: (1, 2, 3)

jcrist added 2 commits June 7, 2021 23:07
@jcrist
Use tuple for array types if needed
d7138b0 
Previously if a msgpack array was decoded without type information, we'd
default to using a `list`. This meant that messages that used `tuple`
for keys would error on decode without type information. We now
automatically use `tuple` by default when necessary for decoding an
`array` type (e.g. when the value is part of a key in a `dict` or
`set`). This allows us to roundtrip more messages successfully without
type information, while still relying on standard msgpack types.
@jcrist
Update docs
f0354d2
@jcrist jcrist mentioned this pull request Jun 8, 2021
Closed
@jcrist
jcrist merged commit 3b12a73 into master Jun 8, 2021
@jcrist
jcrist deleted the use-tuple-for-arrays-if-needed branch June 8, 2021 04:30
sobolevn added a commit to Andarius/msgspec that referenced this pull request Oct 12, 2026
@sobolevn
Fix Meta(pattern=..) memory leak / corruption (msgspec#1256)
4f7363d 
`MS_CONSTR_STR_REGEX` slot clear was missing. It was not properly
traversed and freed. Now it is.

It used to fail with memory sanitized with:

```
 ==2637==ERROR: AddressSanitizer: attempting free on address which was not malloc()-ed: 0x50e000e8e350 in thread T0
    #0 0x7f07a10fc4d8 in free ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:52
    msgspec#1 0x7f079c42ddf4 in TypeNode_Free src/msgspec/_core.c:3474
    msgspec#2 0x7f079c42dbdf in TypeNode_Free src/msgspec/_core.c:3468
    msgspec#3 0x7f079c55e2eb in JSONDecoder_dealloc src/msgspec/_core.c:17235
    msgspec#4 0x7f07a0999132 in _PyEval_EvalFrameDefault Python/bytecodes.c:1398
    msgspec#5 0x7f07a0a15f73 in _PyObject_VectorcallTstate Include/internal/pycore_call.h:92
    msgspec#6 0x7f07a0a15f73 in method_vectorcall Objects/classobject.c:69
    msgspec#7 0x7f07a098d82f in _PyEval_EvalFrameDefault Python/bytecodes.c:3263
    msgspec#8 0x7f07a0a14e6c in _PyFunction_Vectorcall Objects/call.c:419
    msgspec#9 0x7f07a0a14e6c in _PyObject_FastCallDictTstate Objects/call.c:144
    msgspec#10 0x7f07a0a14e6c in _PyObject_Call_Prepend Objects/call.c:508
    msgspec#11 0x7f07a0ac3381 in slot_tp_call Objects/typeobject.c:8791
    msgspec#12 0x7f07a0a14206 in _PyObject_MakeTpCall Objects/call.c:240
    msgspec#13 0x7f07a098d20a in _PyEval_EvalFrameDefault Python/bytecodes.c:2715
    msgspec#14 0x7f07a0a14e6c in _PyFunction_Vectorcall Objects/call.c:419
    msgspec#15 0x7f07a0a14e6c in _PyObject_FastCallDictTstate Objects/call.c:144
    msgspec#16 0x7f07a0a14e6c in _PyObject_Call_Prepend Objects/call.c:508
    msgspec#17 0x7f07a0ac3381 in slot_tp_call Objects/typeobject.c:8791
    msgspec#18 0x7f07a0a1550e in PyObject_Call (/opt/hostedtoolcache/Python/3.12.15/x64/lib/libpython3.12.so.1.0+0x21550e) (BuildId: d5ce433e66c26342bfaf6b82ab1a95541a8ebade)
    msgspec#19 0x7f07a098d82f in _PyEval_EvalFrameDefault Python/bytecodes.c:3263
    msgspec#20 0x7f07a0a14e6c in _PyFunction_Vectorcall Objects/call.c:419
    msgspec#21 0x7f07a0a14e6c in _PyObject_FastCallDictTstate Objects/call.c:144
    msgspec#22 0x7f07a0a14e6c in _PyObject_Call_Prepend Objects/call.c:508
    msgspec#23 0x7f07a0ac3381 in slot_tp_call Objects/typeobject.c:8791
    msgspec#24 0x7f07a0a14206 in _PyObject_MakeTpCall Objects/call.c:240
    msgspec#25 0x7f07a098d20a in _PyEval_EvalFrameDefault Python/bytecodes.c:2715
    msgspec#26 0x7f07a0a14e6c in _PyFunction_Vectorcall Objects/call.c:419
    msgspec#27 0x7f07a0a14e6c in _PyObject_FastCallDictTstate Objects/call.c:144
    msgspec#28 0x7f07a0a14e6c in _PyObject_Call_Prepend Objects/call.c:508
    msgspec#29 0x7f07a0ac3381 in slot_tp_call Objects/typeobject.c:8791
    msgspec#30 0x7f07a0a14206 in _PyObject_MakeTpCall Objects/call.c:240
    msgspec#31 0x7f07a098d20a in _PyEval_EvalFrameDefault Python/bytecodes.c:2715
    msgspec#32 0x7f07a0a14e6c in _PyFunction_Vectorcall Objects/call.c:419
    msgspec#33 0x7f07a0a14e6c in _PyObject_FastCallDictTstate Objects/call.c:144
    msgspec#34 0x7f07a0a14e6c in _PyObject_Call_Prepend Objects/call.c:508
    msgspec#35 0x7f07a0ac3381 in slot_tp_call Objects/typeobject.c:8791
    msgspec#36 0x7f07a0a14206 in _PyObject_MakeTpCall Objects/call.c:240
    msgspec#37 0x7f07a098d20a in _PyEval_EvalFrameDefault Python/bytecodes.c:2715
    msgspec#38 0x7f07a0ae8f35 in PyEval_EvalCode Python/ceval.c:580
    msgspec#39 0x7f07a0ae7a3c in builtin_exec_impl Python/bltinmodule.c:1093
    msgspec#40 0x7f07a0ae7a3c in builtin_exec Python/clinic/bltinmodule.c.h:543
    msgspec#41 0x7f07a0a345a2 in cfunction_vectorcall_FASTCALL_KEYWORDS Objects/methodobject.c:438
    msgspec#42 0x7f07a0a14532 in _PyObject_VectorcallTstate Include/internal/pycore_call.h:92
    msgspec#43 0x7f07a0a14532 in PyObject_Vectorcall Objects/call.c:325
    msgspec#44 0x7f07a098d20a in _PyEval_EvalFrameDefault Python/bytecodes.c:2715
    msgspec#45 0x7f07a0ae8f35 in PyEval_EvalCode Python/ceval.c:580
    msgspec#46 0x7f07a0b09f76 in run_eval_code_obj Python/pythonrun.c:1757
    msgspec#47 0x7f07a0b09aaa in run_mod Python/pythonrun.c:1778
    msgspec#48 0x7f07a0b099f1 in pyrun_file Python/pythonrun.c:1674
    msgspec#49 0x7f07a0b09852 in _PyRun_SimpleFileObject Python/pythonrun.c:459
    msgspec#50 0x7f07a0b09667 in _PyRun_AnyFileObject Python/pythonrun.c:78
    msgspec#51 0x7f07a0b12943 in pymain_run_file_obj Modules/main.c:361
    msgspec#52 0x7f07a0b12943 in pymain_run_file Modules/main.c:380
    msgspec#53 0x7f07a0b12943 in pymain_run_python Modules/main.c:634
    msgspec#54 0x7f07a0b12943 in Py_RunMain Modules/main.c:714
    msgspec#55 0x7f07a0b1243a in Py_BytesMain Modules/main.c:768
    msgspec#56 0x7f07a042a1c9  (/lib/x86_64-linux-gnu/libc.so.6+0x2a1c9) (BuildId: a4a7992a8e66555c8141ab2a08a8465ff6e0ea65)
    msgspec#57 0x7f07a042a28a in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2a28a) (BuildId: a4a7992a8e66555c8141ab2a08a8465ff6e0ea65)
    msgspec#58 0x558976af7094 in _start (/opt/hostedtoolcache/Python/3.12.15/x64/bin/python3.12+0x1094) (BuildId: d664fd041bd9df9902c326eb80717fa04325aaaa)

0x50e000e8e350 is located 16 bytes inside of 152-byte region [0x50e000e8e340,0x50e000e8e3d8)
allocated by thread T0 here:
    #0 0x7f07a10fd9c7 in malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:69
    msgspec#1 0x7f07a0a7c2fc in gc_alloc Modules/gcmodule.c:2330
    msgspec#2 0x7f07a0a7c2fc in _PyObject_GC_NewVar Modules/gcmodule.c:2364

SUMMARY: AddressSanitizer: bad-free ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:52 in free
==2637==ABORTING
```

See
https://github.com/msgspec/msgspec/actions/runs/38243375356/job/114791328508

Now it is properly cleaned.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

See More

Successfully merging this pull request may close these issues.

1 participant

@jcrist