Skip to content

GH-2037: Keep writer size available after close - #3805

Open
1fanwang wants to merge 1 commit into
apache:masterfrom
1fanwang:1fannnw/report-writer-size-after-close
Open

1fanwang wants to merge 1 commit into
apache:masterfrom
1fanwang:1fannnw/report-writer-size-after-close

Conversation

@1fanwang

@1fanwang 1fanwang commented Sep 20, 2026

Copy link
Copy Markdown
Contributor

Rationale for this change

Reading the writer's size after successfully closing a Parquet file throws a null-pointer exception because its column buffers have already been released. Callers cannot report the completed file size through the writer.

Closes #2037.

What changes are included in this PR?

Save the final byte position after writing the footer, before closing the output stream. The size accessor then uses that position without accessing released buffers or a closed channel.

Are these changes tested?

Testing Done

On Java 17, this standalone program writes one row, closes the file, and compares the reported size with the real file length.

Before, on 2df8d02:

java.lang.NullPointerException: Cannot invoke "org.apache.parquet.column.ColumnWriteStore.getBufferedSize()" because "this.columnStore" is null

After:

reported=312, file=312

Run the following from the base checkout and the PR checkout, saving the source below after the build:

mvn -q -pl parquet-hadoop -am -DskipTests package dependency:build-classpath -Dmdep.outputFile=target/runtime-classpath
java -cp "parquet-common/target/classes:parquet-hadoop/target/classes:$(cat parquet-hadoop/target/runtime-classpath)" WriterSizeProbe.java size.parquet
Reproducer source

Save as WriterSizeProbe.java:

import java.nio.file.Files;
import java.nio.file.Path;
import org.apache.parquet.example.data.Group;
import org.apache.parquet.example.data.simple.SimpleGroup;
import org.apache.parquet.hadoop.ParquetWriter;
import org.apache.parquet.hadoop.example.ExampleParquetWriter;
import org.apache.parquet.io.LocalOutputFile;
import org.apache.parquet.schema.MessageType;
import org.apache.parquet.schema.MessageTypeParser;

class WriterSizeProbe {
  public static void main(String[] args) throws Exception {
    Path path = Path.of(args[0]);
    MessageType schema = MessageTypeParser.parseMessageType("message test { required int32 value; }");
    ParquetWriter<Group> writer = ExampleParquetWriter.builder(new LocalOutputFile(path))
        .withType(schema).build();
    writer.write(new SimpleGroup(schema).append("value", 7));
    writer.close();
    try {
      long reported = writer.getDataSize();
      long actual = Files.size(path);
      System.out.printf("reported=%d, file=%d%n", reported, actual);
      if (reported != actual) {
        throw new AssertionError("Reported size differs from the completed file");
      }
    } catch (NullPointerException failure) {
      System.out.println(failure);
    }
  }
}

The regressions also read back empty, single-row and multiple-row-group files through both local and FileChannel-backed outputs, including a second close.

Are there any user-facing changes?

After a successful close, the size accessor returns the completed file size, including the footer. Its behavior while the writer is open is unchanged.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ParquetWriter.getDataSize NullPointerException after closed

1 participant