Repository navigation
tell() is incorrect after read_bytes() #426
Description
Using the default ._cmsgpack implementation of an Unpacker, you will get the wrong answer if you call tell() after read_bytes(). Here's a simple repro:
import msgpack
from io import BytesIO
buf = BytesIO(b'my data')
u = msgpack.Unpacker(buf)
print(u.tell())
u.read_bytes(5)
print(u.tell())
After calling u.read_bytes(5), u.tell() will still return 0. It looks like the stream_offset in _unpacker.pyx should be advanced in read_bytes(), but is currently only advanced by _unpack().
Note that this works correctly in the .fallback implementation, so a quick workaround is to use import msgpack.fallback as msgpack, but I assume this is much slower than the C implementation.
Activity
jfolz commentedon Jun 4, 2020
Kinda embarrassing, since I implemented
tell, but never explicitly documented what it was actually supposed to do. Intuitively it's meant to work liketellon a file-like to give the position of the object that theUnpackerreturns next.This prints
2 1, so we skipped the array header (1 byte) and unpack the first item instead (1 byte). We should now be at position 2.Then it prints
4 1and5 2, sotellis behaving as expected as long asread_bytesis not used.AFAIK adding
stream_offset += len(ret)here should fix this.read_bytesreads directly from the buffer (and the file-like if needed), but doesn't advance thestream_offset.fix msgpack#426, add doc to Unpacker.tell(), add tests for tell
fix msgpack#426, add doc to Unpacker.tell(), add tests for tell