Repository navigation
MsgPackLite: support InputStream as first-class object #219
Description
In scope of #208, @Totktonada stated
I think we can write a potentially large request chunk-by-chunk to a socket. To achieve that we need to learn our msgpack encoder to work in a lazy manner: receive a stream as an input parameter and return a stream as a result.
My main concern is that we can avoid materializing streams in memory.
JDBC streams support was provided for PreparedStaement in #208. Due to the lack of streams support in IPPROTO as well as MsgPackLite implementation, the driver materializes a stream in memory before it will be encoded and sent to the socket.
Rationale: it's possible to send/receive quite long binary data. (for instance, MP_BIN32 stores a byte array whose length is up to (2^32)-1 bytes == 4Gb). As a result, it will be convenient to push/fetch bytes partially chunk-by-chunk for such enormous data. The InputStream, especially BufferredStream, looks a suitable abstraction over the bytes "on-demand".
There are a few open questions:
Do we support both byte arrays and streams to represent binary data?
Maybe it's worth providing such support only for mp_bin16/32 and for mp_bin8 use byte[].
The current implementation requires a packet must be fully encoded to be sent. And vice versa response, before it will be returned, should be completely read and decoded. How to support laziness here is a big question.
It seems the last point is the most important to be clarified.
Activity
Totktonada commentedon Aug 21, 2019
Is this the questions about msgpack encoder / decoder API, a client API or JDBC client API? Can you expand it a bit, please?
If this is the question how to support it with tarantool binary protocol, then let me cite my comment to PR #208:
If the question is about the connector implementation, then I guess it requires to implement a size calculation w/o actual encoding and support of encoding by chunks in our msgpack encoder. The same for decoder.
When I did comment PR #208 I didn't consider that it is possible that a data size may be unknown according to JDBC API. We however need a data size to create a packet header. JDBC API allows to send streams w/o known size, they will need to be materized anyway. So now I think that support of lazy msgpack encoding has much less value. More on that: tarantool is not intended to be used as a storage for large data records (at least now).
So I think it not worth to implement this issue.
@nicktorwald Are there question need to be clarified (maybe the first one) or we can close this?
nicktorwald commentedon Aug 22, 2019
That question isn't so important and it was about MsgPack API, should it support both byte[] for mp_bin8 and InputStream for bigger data size (mp_bin16/32).
I think we can close the issue now.