This repository was archived by the owner on Mar 8, 2020. It is now read-only.
Repository navigation
This repository was archived by the owner on Mar 8, 2020. It is now read-only.
Stream output instead of buffering to memory #22
Description
It seems like the driver currently buffers the serialized JSON into memory instead of streaming it:
This causes OOM kills when processing large files.
@juanjux Is there any reason to not use Sytem.out here directly?
Activity
juanjux commentedon Jan 25, 2019
As usually happens with Java, things are somewhat convoluted. We use Jackson for the json generation. Jackson's jsonFactory.createGenerator method require and
OutputStreamclass.But its probably true that
ByteArrayOutputStreamis not the best option, I'll try with the other subclasses.juanjux commentedon Jan 25, 2019
Ok, I managed to find a way to use
System.out👍Now the problem dissapears, at least on the Java part. The file in #19 still uses a lot of memory, more than my system has (16GB) if I let it run forever (causing the kernel OOM killer to nuke it at the end), but the memory usage has moved from the Java process to the
driverone (so it seems at some points drivers or the SDK is keeping everything in memory too).But since this issue is about the stream output I'll close it once the PR I'll publish soon is merged.