Skip to content

MINOR: Preserve fixed-size binary widths in Arrow schemas - #3803

Open
1fanwang wants to merge 1 commit into
apache:masterfrom
1fanwang:1fannnw/preserve-fixed-binary-width
Open

1fanwang wants to merge 1 commit into
apache:masterfrom
1fanwang:1fannnw/preserve-fixed-binary-width

Conversation

@1fanwang

Copy link
Copy Markdown
Contributor

Rationale for this change

A 16-byte fixed-size Arrow binary field comes back as variable-width Binary after writing a file with the converted Parquet schema. The bytes and nulls survive, but the declared width is lost.

What changes are included in this PR?

Positive fixed-size binary widths now map to fixed-length Parquet storage and back. Zero-width fields keep their BINARY representation because Parquet requires a positive fixed length. Decimal annotations and INT96 mappings are unchanged.

Are these changes tested?

The regression writes a real file containing a 16-byte value and a null row, reads both back, and checks the recovered Arrow schema.

Testing Done

Property Before After
Parquet physical type BINARY FIXED_LEN_BYTE_ARRAY
Declared byte width Absent 16
Recovered Arrow type Binary FixedSizeBinary

I ran the same regression on the baseline and this change with Java 17 and Maven 3.9.16. From the PR checkout:

git worktree add -b 1fannnw/repro-fixed-binary \
  ../parquet-fixed-binary-before 2df8d02678dab4bb8b926a0d3221cc652984c7ab
git diff 2df8d02678dab4bb8b926a0d3221cc652984c7ab HEAD \
  -- parquet-arrow/pom.xml \
  parquet-arrow/src/test/java/org/apache/parquet/arrow/schema/TestSchemaConverter.java |
  git -C ../parquet-fixed-binary-before apply -

mvn -B -T2 -f ../parquet-fixed-binary-before/pom.xml \
  -pl parquet-arrow -am test \
  '-Dtest=TestSchemaConverter#testFixedBinaryFileRoundTrip' \
  -Dsurefire.failIfNoSpecifiedTests=false

mvn -B -T2 -pl parquet-arrow -am test \
  '-Dtest=TestSchemaConverter#testFixedBinaryFileRoundTrip' \
  -Dsurefire.failIfNoSpecifiedTests=false
Raw logs

Before:

stored_type=BINARY
stored_width=0
restored_type=Binary
rows=2, payload_preserved=true, null_preserved=true

expected: Schema<fingerprint: FixedSizeBinary(16)>
 but was: Schema<fingerprint: Binary>

After:

stored_type=FIXED_LEN_BYTE_ARRAY
stored_width=16
restored_type=FixedSizeBinary
rows=2, payload_preserved=true, null_preserved=true

The added dependencies are test-scoped for the file roundtrip.

  • Local code review completed

Are there any user-facing changes?

Yes. Positive fixed-size binary fields retain their declared widths when converting schemas in either direction.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant