Repository navigation
Conversation
encode_into arbitrary buffers.
Widening the runtime to accept `None` was the alternative. I rejected it: the docstring specifies `offset : int, optional` and reserves `-1` as the sentinel for writing at the end of the buffer, which leaves `None` with no meaning to carry. Narrowing the stub makes the declared type match both the C code and the documented contract. The `int | None` traces back to the original `Optional[int]` annotation and survived the 3.10+ syntax migration in msgspec#1041. Nothing in the tree passes `None`. Residual limitation: this edits the same two stub lines as open PR msgspec#1160, which reshapes `buffer` and the return type while keeping `offset: int | None`. Whichever lands second needs a one-line rebase. The two changes are otherwise independent, since msgspec#1160 keeps reading `offset` via `PyLong_AsSsize_t`. Measurement, CPython 3.14: - `just test-typing` clean under mypy 2.3.0, pyright 1.1.411, pyrefly 1.2.0. - Reverting the stub hunk alone makes all three checkers fail on both new call sites in `basic_typing_examples.py`, so the type-level test is not vacuous. - `pytest tests/unit`: 6119 passed, 404 skipped, identical to the pre-change baseline. The two runtime assertions are pins on current behavior rather than new coverage. - `ruff format` collapses the signature onto one line once `| None` is gone; that reformat is included. No changelog entry, following merged stub PRs msgspec#1055, msgspec#1057, msgspec#1062, msgspec#1065, msgspec#1074 and msgspec#1093, which leave it to the aggregated "type stub improvements" line at release. Co-authored-by: Dylan Pulver <[email protected]>
|
There are two problems with scratch-space writes when encoding JSON into a fixed-size buffer. A one-byte buffer is sufficient for enc = msgspec.json.Encoder()
enc.encode_into(1, memoryview(bytearray(1)))With a larger buffer, a byte after the returned message length is overwritten: buf = bytearray(b"?" * 32)
n = enc.encode_into(1, memoryview(buf))
print(n, buf[:2]) # 1, bytearray(b'10'), expected bytearray(b'1?')Pls adjust the JSON fast paths so that an exactly sized buffer is sufficient and bytes outside the encoded message remain unchanged, with regression tests for both cases Please also update the |
Lets
Encoder.encode_into()(bothmsgspec.msgpackandmsgspec.json) accept any writable, contiguous buffer-protocol object (e.g. amemoryviewontomultiprocessing.shared_memory.SharedMemory), not justbytearray.Currently the encode_into only works on bytearrays, this makes it impossible to use Python's SharedMemory with a zero-copy serialization.
fails with
This changes allows for arbitrary buffers that do not support resizing.
encode_into also return the number of newly written bytes. So offset + newly_written_bytes can by used as the offset for the next element when streaming data.