fix(sdk): harden ZipReader central directory parsing against hostile fields (DSPX-5103) - #418

…, and UTF-8 entry names (#398)

Jira: https://virtru.atlassian.net/browse/DSPX-4589

This PR carries everything left of DSPX-4589 now that #397 has merged.
#396 is closed as a duplicate.

It started as three zip container conformance findings from an audit
against PKWARE APPNOTE.TXT. Review turned up more reader hardening,
which is also here. None of it has known field impact; the one finding
that did is #397.

## Release note

> **Behavior change:** `ZipReader` now looks for the end of central
directory record only in the last 22 + 65,535 bytes of the archive,
which is the most a zip comment can push it back. An archive with more
than 64 KiB of data appended *after* the record (not a comment) used to
read and is now rejected with `InvalidZipException`. Zip64 archives with
trailing data never read correctly before and now do. Truncated archives
and entries now fail with `InvalidZipException` rather than being
accepted or returning short data. An archive whose central directory
does not end exactly where the record after it starts (the zip64 end of
central directory record if there is one, otherwise the end of central
directory record) is rejected; every writer we know of, this SDK's
included, writes them back to back.

## Public API note

`InvalidZipException` is unchecked (it extends `RuntimeException`). Two
places that used to report truncation through `IOException` now throw it
instead:
- `new ZipReader(channel)` and `readCentralDirectoryFileHeader()` throw
`InvalidZipException` for a filename cut short, where they used to throw
`EOFException`.
- An entry's `InputStream` throws `InvalidZipException` when the archive
ends before the entry does, where it used to return `-1` early.

Callers that catch only `IOException` around these will now see an
unchecked exception. Inside the SDK, `SDK.isTDF` already catches both.
#418 wraps the manifest and payload cases in `TDFReader` as
`SDK.MalformedTDFException`.

## Writer

### End of central directory sentinel thresholds (fixed)

`ZipWriter.finish()` decided whether the archive needed a zip64 end of
central directory record with masks that matched none of the field
widths:

```java
(numEntries & ~0xFF) != 0 || (startOfCentralDirectory & ~0xFFFF) != 0 || (sizeOfCentralDirectory & ~0xFFFF) != 0
```

Any archive with 256 or more entries was needlessly promoted to zip64,
and the offset and size checks fired three orders of magnitude too
early. Now:

```java
var isZip64 = hasZip64Entry
        || numEntries > MAX_NON_ZIP64_ENTRY_COUNT          // Short.MAX_VALUE
        || needsZip64(startOfCentralDirectory, sizeOfCentralDirectory);
```

All three thresholds stop short of what the format allows (`0xFFFE`
entries, `0xFFFFFFFE` bytes), so that a reader widening these unsigned
fields with a signed read never sees a negative value. Released versions
of this SDK do exactly that: an entry count above 32,767 would come back
negative and read as an empty archive. So an archive goes zip64 at
32,768 entries, and at the same 2 GiB (`Integer.MAX_VALUE`) offset/size
ceiling the per-entry fields already use. Routing offset and size
through `needsZip64` also keeps the `ZipWriter(out, maxNonZip64Value)`
test seam from #393 usable for them. Whenever the writer goes zip64 it
sets the offset sentinel too, which is the only one older readers check.

### UTF-8 entry names

The ticket says the central directory filename length was computed from
`String.length()`. The assignment existed (`cdFileHeader.filenameLength
= (short) fileInfo.filename.length();`), but `CDFileHeader.write()`
never read it: it wrote the length of the encoded byte array. The local
header's copy was set from the encoded bytes too. **The bytes on the
wire were already correct**; no archive was ever mis-written.

What changed:
- The dead `filenameLength` field in `CDFileHeader` is gone, and
`LocalFileHeader` now takes its length from the encoded bytes it writes,
like `CDFileHeader` does.
- A new `encodeFilename` helper rejects a name over `Short.MAX_VALUE`
UTF-8 bytes with an `SDKException` rather than silently truncating it
into the 2-byte field. The cap is `Short.MAX_VALUE` rather than `0xFFFF`
for the same signed-read reason as above: older SDK readers read the
name length as signed. Validation runs before anything is written, so a
rejected name leaves the writer usable.
- `data()` entries now set the UTF-8 flag (general purpose bit 11) in
the local header as well as the central directory. It was previously set
only in the central directory, so readers that go by the local header
could garble non-ASCII names. `stream()` entries already set it in both.

## Reader

- **End of central directory scan.** A short read used to be treated as
a signature match. The scan now:
- only matches a real signature, and throws `InvalidZipException` if it
finds none;
- is bounded to the last 22 + 0xFFFF bytes (see the release note).
Previously it was unbounded, doing a positioned 4-byte read per byte
back to the start of the file;
- skips any candidate whose declared comment would run past the end of
the archive. That rejects a record whose comment was truncated. Trailing
data after an honest comment is still allowed;
- skips any candidate whose central directory does not end exactly where
the next record starts: the zip64 end of central directory record if
there is one (including one written without sentinels, as
commons-compress does), otherwise the candidate itself. This is what
passes over a stray `PK\5\6` inside a comment or in trailing data whose
declared comment happens to fit, such as a zero-filled one, which used
to read as an empty archive. If no candidate passes, the failure of the
one nearest the end is reported.
- **Zip64 locator.** The locator is found relative to the end of central
directory record it precedes (APPNOTE 4.3.15), not relative to the end
of the file. This is what makes zip64 archives with trailing data
readable. An archive too small to hold the locator it claims is
rejected.
- **Bounds checks.** Offsets taken from the archive (the zip64 end of
central directory pointer and the central directory offset) are checked
against the archive size, and so is the entry count: a negative or
impossibly large zip64 count is rejected instead of reading as an empty
archive. So is a zip64 entry size or central directory size that reads
as a negative `long`; a negative entry size used to read as an empty
entry.
- **Short and empty reads.** Every fixed-width read loops until it is
full. `ReadableByteChannel.read` may return fewer bytes than asked, or
none at all, without being at end of stream; only `-1` is end of file.
Empty reads are retried at most 16 times in a row, without waiting, and
then fail with an `IOException` that names the channel class. So a
channel with nothing to give can't make the reader spin, but `ZipReader`
needs a blocking channel; non-blocking channels are not supported.
`loadTDF` takes caller-supplied channels, so this is reachable from the
public API.
- **Truncation is always an error.** Entry data that runs past the end
of the archive now throws `InvalidZipException` instead of ending the
stream early with a short entry. A filename cut short throws
`InvalidZipException` instead of `EOFException`.
`InputStream.read(byte[], int, int)` no longer returns 0 for a non-empty
request.
- Error messages name the record, offset and bytes found. The locator
signature error used to name the wrong record. A bad local file header
signature now names the entry and offset.

## Tests

30 new tests:

**`ZipWriterTest`** (6)
- One test per end of central directory sentinel:
- entry count: `Short.MAX_VALUE` stays non-zip64 and `Short.MAX_VALUE +
1` goes zip64, both round-tripped through `ZipReader`;
  - central directory offset;
  - central directory size.

Each is isolated so that only the end of central directory record is
zip64 and no entry is.
- `filenameLengthIsMeasuredInUtf8Bytes` writes `"🔒両.txt"` (7 UTF-16 code
units, 11 UTF-8 bytes). It checks the length at both header offsets and
the UTF-8 flag in both headers.
- `rejectsAnEntryNameTooLongToDescribe` covers the boundary: a
32,767-byte name is accepted and reads back, and 32,768 bytes is
rejected on both the `data()` and `stream()` paths. It also checks that
the writer still works after a rejection and that the result has exactly
one entry.
- `streamedEntryNameIsMeasuredInUtf8Bytes` round-trips a non-ASCII name
through `stream()`, the path a TDF payload takes, with both `ZipReader`
and commons-compress. The 32,768-entry archive is also opened with
commons-compress.

**`ZipReaderTest`** (24)

Truncated or malformed trailing records:
- truncated archives (four shapes) and an archive too short for its
zip64 locator;
- an end of central directory record pushed past the comment limit is
rejected;
- a zip64 archive with the longest possible comment (0xFFFF bytes) still
reads;
- a decoy signature inside the comment is skipped, both one whose
declared comment runs past the end and one whose comment fits:
zero-filled, a copy of the real record, and a copy claiming one entry,
each in a roomy comment, a comment it fills exactly, and trailing data,
on plain and zip64 archives;
- a truncated comment is rejected.

Bounds checks:
- a zip64 locator pointer outside the archive;
- a zip64 central directory offset outside the archive;
- a plain (non-zip64) central directory offset outside the archive;
- a negative or impossibly large zip64 entry count;
- a zip64 entry size, and a zip64 central directory size, that read as
negative;
- an end of central directory record that claims zip64 with no locator
in front of it;
- a zip64 locator that points inside the archive, but at the wrong
record.

Channel behavior:
- a channel returning short reads (1 to 8 bytes at a time);
- a channel returning empty reads;
- a channel that stops returning data, both before the archive is opened
and partway through an entry (through both `read()` and `read(byte[])`);
- a channel whose `size()` claims more bytes than it holds.

Everything else:
- a zip64 archive with trailing data reads;
- an empty archive reads;
- a central directory filename running past the end is rejected;
- entry data running past the end is rejected, through both `read()` and
`read(byte[])`;
- a zero-length read returns 0, including at the end of an entry.

Negative tests match error messages on stable values (entry name,
offset, size) rather than wording.

The writer sentinel tests were confirmed to be real regression tests by
reverting the mask fix and watching them fail. Run against the previous
`ZipReader`, the new tests for empty reads, a stalled channel, the decoy
and truncated comments, the entry count bounds, filename truncation and
entry truncation all fail. Without the fixes from earlier in this PR,
the tests for short reads, trailing data and locator bounds fail too.
The decoy tests with a comment that fits and the negative entry size
test fail against the reader as it was before the latest commit. The
plain-truncation, longest-comment, empty-archive and plain-offset tests
pass on the old code; they are there to lock that behavior in.

```
mvn -o -pl sdk test -Denforcer.skip=true -Dmaven.antrun.skip=true   [JDK 21, generated sources from an earlier build]
  -> 338 tests, 0 failures, 0 errors, 8 skipped
```

## End-to-end validation

None of this has an xtest cell of its own; it is unit-tested only. What
e2e gives this PR is a no-regression signal: xtest on this PR is green
across java/go/js against `main` and v0.26.0 (see the X-Test Results
comments). [opentdf/tests run
34357326964](https://github.com/opentdf/tests/actions/runs/34357326964)
also passed earlier with `force-supports=chunky`, including all five
java `test_chunky_roundtrip` pairs. That run predates the latest round
of reader changes.


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Bug Fixes**
* ZIP archives are read reliably when data arrives in partial chunks or
channels temporarily return no data; persistently stalled reads now fail
instead of hanging.
* Missing, truncated, or out-of-bounds archive metadata and entry data
are reported as errors.
* ZIP64 archives are handled correctly, including those with large entry
counts or central directories.

* **Improvements**
* Non-ASCII filenames use accurate UTF-8 byte lengths and flags in
archive headers.
* Entry names that exceed ZIP format limits are rejected with a clear
error.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Signed-off-by: Dave Mihalcik <[email protected]>