Skip to content

plugins/destination/gcs/client/write.go: *storage.Writer not closed on WriteHeader/WriteContent/WriteFooter errors + nil writer on empty channel #23173

Description

@praneshnikhar

GCS: Resource leak in write.go — *storage.Writer not closed on errors

File: plugins/destination/gcs/client/write.go

Bug 1: Writer not closed on error paths

Problem: When WriteHeader, WriteContent, or WriteFooter fails, the *storage.Writer (w) is not closed. This leaks the underlying HTTP connection and GCS stream resource. The Close on the writer is what actually flushes the buffer and finalizes the object — skipping it on error paths means:

  1. The in-progress upload is aborted but the connection is leaked.
  2. The writer object is garbage collected without proper cleanup.

Current code:

h, err = c.Client.WriteHeader(w, table)
if err != nil {
    return err  // w.Close() not called
}

if err := h.WriteContent([]arrow.RecordBatch{msg.Record}); err != nil {
    return err  // w.Close() not called
}

if err := h.WriteFooter(); err != nil {
    return err  // w.Close() not called
}
return w.Close()

Fix: Call w.Close() before returning on each error path.

Bug 2: w.Close() on nil writer

Problem: When the message channel is empty (no messages received), w is never initialized and remains nil. The function then tries return w.Close() (or return nil), but the code also has an issue where the nil writer isn't handled gracefully.

Current flow: If the for msg := range msgs loop never executes (empty channel), w is nil. The function should return nil for empty writes.

Fix: Add nil writer check:

if w == nil {
    return nil
}

Reproduction

  1. Configure a GCS destination plugin.
  2. Simulate a transient failure during WriteContent (e.g., network interruption, auth failure).
  3. The error is returned but the writer is leaked — repeated syncs accumulate leaked resources.
  4. For the nil writer case: sync a source that produces 0 records for a table — the empty-write path closes a nil writer.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions