GCS: Resource leak in write.go — *storage.Writer not closed on errors
File: plugins/destination/gcs/client/write.go
Bug 1: Writer not closed on error paths
Problem: When WriteHeader, WriteContent, or WriteFooter fails, the *storage.Writer (w) is not closed. This leaks the underlying HTTP connection and GCS stream resource. The Close on the writer is what actually flushes the buffer and finalizes the object — skipping it on error paths means:
- The in-progress upload is aborted but the connection is leaked.
- The writer object is garbage collected without proper cleanup.
Current code:
h, err = c.Client.WriteHeader(w, table)
if err != nil {
return err // w.Close() not called
}
if err := h.WriteContent([]arrow.RecordBatch{msg.Record}); err != nil {
return err // w.Close() not called
}
if err := h.WriteFooter(); err != nil {
return err // w.Close() not called
}
return w.Close()
Fix: Call w.Close() before returning on each error path.
Bug 2: w.Close() on nil writer
Problem: When the message channel is empty (no messages received), w is never initialized and remains nil. The function then tries return w.Close() (or return nil), but the code also has an issue where the nil writer isn't handled gracefully.
Current flow: If the for msg := range msgs loop never executes (empty channel), w is nil. The function should return nil for empty writes.
Fix: Add nil writer check:
if w == nil {
return nil
}
Reproduction
- Configure a GCS destination plugin.
- Simulate a transient failure during
WriteContent (e.g., network interruption, auth failure).
- The error is returned but the writer is leaked — repeated syncs accumulate leaked resources.
- For the nil writer case: sync a source that produces 0 records for a table — the empty-write path closes a nil writer.
GCS: Resource leak in write.go — *storage.Writer not closed on errors
File:
plugins/destination/gcs/client/write.goBug 1: Writer not closed on error paths
Problem: When
WriteHeader,WriteContent, orWriteFooterfails, the*storage.Writer(w) is not closed. This leaks the underlying HTTP connection and GCS stream resource. TheCloseon the writer is what actually flushes the buffer and finalizes the object — skipping it on error paths means:Current code:
Fix: Call
w.Close()before returning on each error path.Bug 2: w.Close() on nil writer
Problem: When the message channel is empty (no messages received),
wis never initialized and remains nil. The function then triesreturn w.Close()(orreturn nil), but the code also has an issue where the nil writer isn't handled gracefully.Current flow: If the
for msg := range msgsloop never executes (empty channel),wis nil. The function should return nil for empty writes.Fix: Add nil writer check:
Reproduction
WriteContent(e.g., network interruption, auth failure).