This repository was archived by the owner on Mar 8, 2020. It is now read-only.
Repository navigation
This repository was archived by the owner on Mar 8, 2020. It is now read-only.
UnicodeDecodeError in client.parse() #61
Description
I try to run bblfsh from python for this file: https://github.com/INRIA/spoon/blob/master/src/test/resources/noclasspath/IsoEncoding.java
As you can see it is the file with non-standard characters.
If I run
import bblfsh
client = bblfsh.BblfshClient("0.0.0.0:9234")
res = client.parse("IsoEncoding.java")I get
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/google/protobuf/internal/type_checkers.py", line 182, in CheckValue
proposed_value = proposed_value.decode('utf-8')
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe9 in position 69: invalid continuation byte
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Applications/PyCharm.app/Contents/helpers/pydev/pydevd.py", line 1599, in <module>
globals = debugger.run(setup['file'], None, None, is_module)
File "/Applications/PyCharm.app/Contents/helpers/pydev/pydevd.py", line 1026, in run
pydev_imports.execfile(file, globals, locals) # execute the script
File "/Applications/PyCharm.app/Contents/helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
exec(compile(contents+"\n", file, 'exec'), glob, loc)
File "/Users/k/sourced/ast2vec/how_to_use_ast2vec.py", line 4, in <module>
res = client.parse("../bad_file/IsoEncoding.java")
File "/usr/local/lib/python3.6/site-packages/bblfsh/client.py", line 56, in parse
language=self._scramble_language(language))
File "/usr/local/lib/python3.6/site-packages/google/protobuf/internal/python_message.py", line 526, in init
setattr(self, field_name, field_value)
File "/usr/local/lib/python3.6/site-packages/google/protobuf/internal/python_message.py", line 662, in field_setter
new_value = type_checker.CheckValue(new_value)
File "/usr/local/lib/python3.6/site-packages/google/protobuf/internal/type_checkers.py", line 187, in CheckValue
(proposed_value))
ValueError: b'package my.pack;\n\npublic class IsoEncoding {\n\n private String entr\xe9e;\n public IsoEncoding(String maCha\xeene) {\n this.entr\xe9e = maCha\xeene;\n }\n\n private String r\xe9cup\xe9rerUneEntr\xe9e() {\n return this.entr\xe9e;\n }\n}' has type bytes, but isn't valid UTF-8 encoding. Non-UTF-8 strings must be converted to unicode objects before being added.
Is it expected behavior for non-unicode files?
If so I think it is better to provide better error message from bblfsh client.
Metadata
Metadata
Assignees
Labels
No labels
Activity
juanjux commentedon Nov 24, 2017
Yeah, it's a protobuf error:
...has type bytes, but isn't valid UTF-8 encoding. Non-UTF-8 strings must be converted to unicode objects before being added.Do you propose to ping-pong the exception with a more specific error message?
zurk commentedon Nov 24, 2017
yes, It will be great!
Something that file should be in UTF-8 encoding for example.
juanjux commentedon Dec 12, 2017
After #65, non UTF8 (with includes ASCII and Base64) content should be rejected by the client. Tell us if this works for you.
zurk commentedon Dec 12, 2017
yes, It works for me. Thank you!