Describe the enhancement requested
In BaseVariableWidthVector the reallocation logic of the data buffer is to always use the "next power of 2" size instead of the actual desired size (see here). The reallocation fails if the new size exceeds the MAX_BUFFER_SIZE which is Integer.MAX_VALUE by default.
The issue is Integer.MAX_VALUE is just 1 byte less than a power of 2 (231 - 1) which means that exceeding the desired capacity of 1073741824 (1GB), the reallocation would fail. It means, we are unable to get a data buffer larger than 1GB while the default maximum size is almost 2GB.
Describe the enhancement requested
In
BaseVariableWidthVectorthe reallocation logic of the data buffer is to always use the "next power of 2" size instead of the actual desired size (see here). The reallocation fails if the new size exceeds theMAX_BUFFER_SIZEwhich isInteger.MAX_VALUEby default.The issue is
Integer.MAX_VALUEis just 1 byte less than a power of 2 (231 - 1) which means that exceeding the desired capacity of1073741824(1GB), the reallocation would fail. It means, we are unable to get a data buffer larger than 1GB while the default maximum size is almost 2GB.