Unpack for binary data

Hi,

My data in tarantool stored as binary strings, they are not UTF in any manner. It's just a sequence of bytes.
When trying to select data with this module , unpacker fails with error:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.6/dist-packages/tarantool/space.py", line 75, in select
    return self.connection.select(self.space_no, *args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/tarantool/connection.py", line 775, in select
    response = self._send_request(request)
  File "/usr/local/lib/python3.6/dist-packages/tarantool/connection.py", line 357, in _send_request
    return self._send_request_wo_reconnect(request)
  File "/usr/local/lib/python3.6/dist-packages/tarantool/connection.py", line 264, in _send_request_wo_reconnect
    response = Response(self, self._read_response())
  File "/usr/local/lib/python3.6/dist-packages/tarantool/response.py", line 62, in __init__
    self._body = unpacker.unpack()
  File "msgpack/_unpacker.pyx", line 519, in msgpack._unpacker.Unpacker.unpack
  File "msgpack/_unpacker.pyx", line 499, in msgpack._unpacker.Unpacker._unpack
msgpack.exceptions.UnpackValueError: 'utf-8' codec can't decode byte 0xff in position 0: invalid start byte

Can this be resolved somehow?

Any success with fixing that? Still have that problem with Tarantool 2.2 and connector 0.6.6 even when I don't select binary data from a space containing it

It seems, we cannot just set encoding to None when creating a connection, because at least schema reload does not support this.

However we can temporary set encoding to None:

diff --git a/unit/suites/test_dml.py b/unit/suites/test_dml.py
index 31e821e..1264180 100644
--- a/unit/suites/test_dml.py
+++ b/unit/suites/test_dml.py
@@ -70,6 +70,12 @@ class TestSuite_Request(unittest.TestCase):
                     self.con.insert('space_1', [i, i%5, 'tuple_'+str(i)])[0],
                     [i, i%5, 'tuple_'+str(i)]
             )
+        saved_encoding = self.con.encoding
+        self.con.encoding = None
+        t = [1001, 1, b'\xff']
+        self.assertEqual(self.con.insert('space_1', t)[0], t)
+        self.con.encoding = saved_encoding
+
     def test_00_03_answer_repr(self):
         repr_str = """- [1, 1, 'tuple_1']"""
         self.assertEqual(repr(self.con.select('space_1', 1)), repr_str)
@@ -122,6 +128,12 @@ class TestSuite_Request(unittest.TestCase):
             [[200, 0, 'tuple_200'], [205, 0, 'tuple_205']]
         )
 
+        saved_encoding = self.con.encoding
+        self.con.encoding = None
+        res = self.con.select('space_1', 1001)
+        self.assertSequenceEqual(res, [[1001, 1, b'\xff']])
+        self.con.encoding = saved_encoding
+
     def test_03_delete(self):
         # Check that delete works fine
         self.assertSequenceEqual(self.con.delete('space_1', 20), [[20, 0, 'tuple_20']])

NB: File another issue for Connection(host, port, encoding=None) support.

Now I'm getting this:

>> print(result)
- None
- b'Type mismatch: can not convert 'REDACTED BINARY STRING' to varbinary'

And honestly I don't understand what encodings have to do when it comes to raw binary data. Like shouldn't it be just directly converted to bytes Python type?

WIP: Encode bytes as mp_bin on Python 3

Okay, two different problems are discussed here:

A user store binary (non-utf-8 data) in a 'string' field and don't want data being encoded or decoded when working with this field. May be worked around as shown above.
A user want to send / receive mp_bin to work with a 'varbinary' field. Everything looks okay on decoding side (mp_bin is decoded to bytes), but a bytes value is encoded as mp_str. Maybe we can change it on Python 3 (RFC patch).

Ah, sorry for confusion, and thank you. Will wait for a complete fix

Any success?

We're at planning of Q3 tasks and I hope this one will be included here. I'll specifically mark it as important.

schema: support encoding=None connections

NB: Propose to msgpack-python to provide helpers like msgpack.as_bin() and msgpack.as_str() to encode a particular structure (dict, list, tuple) member as mp_bin or mp_str disregarding use_bin_type option. It may be requirement of an msgpack-based protocol to have a string here and a binary value there (exactly our case).

Usage example:

import msgpack
msgpack.dumps(dict(foo='string', bar=msgpack.as_bin('binary')))

schema: support encoding=None connections

types: support working with binary for Python 3

Update python connector comparison table

Update python connector comparison table (#2863)

Navigation Menu

Repository navigation

Unpack for binary data #105

25 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Unpack for binary data #105

Description

Activity

31hkim commented on May 19, 2020

Totktonada commented on May 19, 2020

31hkim commented on May 20, 2020

Totktonada commented on May 20, 2020

31hkim commented on May 21, 2020

31hkim commented on May 28, 2020

Totktonada commented on Jul 8, 2020

Totktonada commented on Aug 28, 2020

25 remaining items

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions