Skip to content

Honor allow_extra_data=False for unknown-size frames in the C backend - #376

Open
SakayanagiYuna wants to merge 2 commits into
indygreg:mainfrom
SakayanagiYuna:codex/fix-unknown-size-trailing-data
Open

SakayanagiYuna wants to merge 2 commits into
indygreg:mainfrom
SakayanagiYuna:codex/fix-unknown-size-trailing-data

Conversation

@SakayanagiYuna

@SakayanagiYuna SakayanagiYuna commented Nov 6, 2026 •

Copy link
Copy Markdown

The C backend ignores allow_extra_data=False when a frame omits its
content size and max_output_size exceeds the decoded size.

For example:

import zstandard as zstd

frame = zstd.ZstdCompressor(write_content_size=False).compress(b"foobar")
zstd.ZstdDecompressor().decompress(
    frame + b"junk",
    max_output_size=64,
    allow_extra_data=False,
)

This returns b"foobar" instead of raising ZstdError for the four
unused bytes. The output-buffer resize branch prevents the subsequent
else if trailing-data check from running.

This change makes that check independent of buffer resizing. The default
behavior and explicit allow_extra_data=True remain unchanged.

Errors raised while evaluating the truth value of allow_extra_data are
also propagated correctly, preserving the original exception instead of
returning a result with an exception set and producing SystemError.

Regression coverage includes unknown-size empty and nonempty frames,
exact and spare output capacity, trailing junk and a second frame, and
decoder reuse after rejection. Additional cases cover exceptions from
__bool__ and __len__, with and without trailing data, and verify that
the original exception object is preserved and the decoder remains usable.

Validation on Windows with CPython 3.12.14:

  • Of 24 trailing-data subcases, 10 fail against the unmodified C backend;
    all pass with this change.
  • All 24 additional truth-value error subcases fail with SystemError
    before the error-handling fix and pass afterward.
  • All 48 subcases pass with both the C and CFFI backends.
  • The regular test suites pass for both backends, with their existing
    skips.

Related to #181.

When a frame omits its content size and max_output_size exceeds the
decoded size, the output-buffer resize branch prevents the subsequent
else-if trailing-data check from running. Trailing input is then
accepted even with allow_extra_data=False.

Run the trailing-data check independently after successful decompression
and any output-buffer resize.

Add regression coverage for unknown-size empty and nonempty frames,
exact and spare output capacity, trailing junk and concatenated frames,
and decoder reuse after rejection. Verify that the default behavior and
explicit allow_extra_data=True continue to accept trailing input.

Related to indygreg#181.
Check the error return from PyObject_IsTrue before testing for trailing
input. Clear the output object and use the existing cleanup path so the
original exception propagates instead of producing SystemError.

Add regression coverage for __bool__ and __len__ failures with known and
unknown content sizes, exact and spare output capacity, and input with
and without trailing data. Verify that the original exception object is
preserved and that the decompressor remains reusable after the error.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant