Skip to content

Allow encode_into arbitrary buffers. - #1160

Open
zdxerr wants to merge 6 commits into
msgspec:mainfrom
zdxerr:encode_into_memoryview
Open

zdxerr wants to merge 6 commits into
msgspec:mainfrom
zdxerr:encode_into_memoryview

Conversation

@zdxerr

@zdxerr zdxerr commented Aug 17, 2026 •

Copy link
Copy Markdown

Lets Encoder.encode_into() (both msgspec.msgpack and msgspec.json) accept any writable, contiguous buffer-protocol object (e.g. a memoryview onto multiprocessing.shared_memory.SharedMemory), not just bytearray.

Currently the encode_into only works on bytearrays, this makes it impossible to use Python's SharedMemory with a zero-copy serialization.

from multiprocessing import shared_memory
import msgspec

enc = msgspec.msgpack.Encoder()
shm = shared_memory.SharedMemory(
    name="shared_data",
    create=True,
    size=1024,  
)

enc.encode_into(Message(datetime.now().astimezone(), Severity.ERROR, "TEST"), shm.buf, 0)

fails with

TypeError: buffer must be a `bytearray`

This changes allows for arbitrary buffers that do not support resizing.

encode_into also return the number of newly written bytes. So offset + newly_written_bytes can by used as the offset for the next element when streaming data.

@zdxerr zdxerr changed the title Allow encode_into arbitrary buffers. Allow encode_into arbitrary buffers. Sep 3, 2026
pull Bot pushed a commit to Future-Outlier/msgspec that referenced this pull request Sep 14, 2026
Widening the runtime to accept `None` was the alternative. I rejected
it: the docstring specifies `offset : int, optional` and reserves `-1`
as the sentinel for writing at the end of the buffer, which leaves
`None` with no meaning to carry. Narrowing the stub makes the declared
type match both the C code and the documented contract.

The `int | None` traces back to the original `Optional[int]` annotation
and survived the 3.10+ syntax migration in msgspec#1041. Nothing in the tree
passes `None`.

Residual limitation: this edits the same two stub lines as open PR
msgspec#1160, which reshapes `buffer` and the return type while keeping
`offset: int | None`. Whichever lands second needs a one-line rebase.
The two changes are otherwise independent, since msgspec#1160 keeps reading
`offset` via `PyLong_AsSsize_t`.

Measurement, CPython 3.14:

- `just test-typing` clean under mypy 2.3.0, pyright 1.1.411, pyrefly
1.2.0.
- Reverting the stub hunk alone makes all three checkers fail on both
new call sites in `basic_typing_examples.py`, so the type-level test is
not vacuous.
- `pytest tests/unit`: 6119 passed, 404 skipped, identical to the
pre-change baseline. The two runtime assertions are pins on current
behavior rather than new coverage.
- `ruff format` collapses the signature onto one line once `| None` is
gone; that reformat is included.

No changelog entry, following merged stub PRs msgspec#1055, msgspec#1057, msgspec#1062,
msgspec#1065, msgspec#1074 and msgspec#1093, which leave it to the aggregated "type stub
improvements" line at release.

Co-authored-by: Dylan Pulver <[email protected]>
@Siyet

Siyet commented Sep 14, 2026

Copy link
Copy Markdown
Contributor

There are two problems with scratch-space writes when encoding JSON into a fixed-size buffer.

A one-byte buffer is sufficient for b"1", but this raises ValueError:

enc = msgspec.json.Encoder()
enc.encode_into(1, memoryview(bytearray(1)))

With a larger buffer, a byte after the returned message length is overwritten:

buf = bytearray(b"?" * 32)
n = enc.encode_into(1, memoryview(buf))
print(n, buf[:2])  # 1, bytearray(b'10'), expected bytearray(b'1?')

Pls adjust the JSON fast paths so that an exactly sized buffer is sufficient and bytes outside the encoded message remain unchanged, with regression tests for both cases

Please also update the encode_into section in docs/perf-tips.rst with a fixed-size buffer example and the new return value. Clarify that offsets and the returned length are measured in bytes. For buffers with multi-byte elements, the docstring example should use memoryview(buffer).cast("B")[offset:offset + n], since buffer[offset:offset + n] slices elements rather than bytes.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants