Repository navigation
Py_CompileString fails on non decode-able paths. #53922
Description
| BPO | 9713 |
|---|---|
| Nosy | @vstinner, @ideasman42 |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = <Date 2010-10-19.02:04:37.767>
created_at = <Date 2010-08-30.07:46:51.986>
labels = ['type-bug', 'expert-unicode']
title = 'Py_CompileString fails on non decode-able paths.'
updated_at = <Date 2010-10-19.02:04:37.766>
user = 'https://github.com/ideasman42'bugs.python.org fields:
activity = <Date 2010-10-19.02:04:37.766>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2010-10-19.02:04:37.767>
closer = 'vstinner'
components = ['Unicode']
creation = <Date 2010-08-30.07:46:51.986>
creator = 'ideasman42'
dependencies = []
files = []
hgrepos = []
issue_num = 9713
keywords = []
message_count = 5.0
messages = ['115202', '115282', '115943', '118838', '119103']
nosy_count = 2.0
nosy_names = ['vstinner', 'ideasman42']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = None
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue9713'
versions = ['Python 3.1', 'Python 3.2']
Activity
ideasman42 commentedon Aug 30, 2010
On linux I have a path which python reads as...
/data/test/num\udce9ro_bad/untitled.blend
os.listdir("/data/test/") returns this ['num\udce9ro_bad']
But the same path cant be given to the C api's Py_CompileString
...gives this error.
UnicodeDecodeError: 'utf8' codec can't decode bytes in position 14-16: invalid data
From this pep, non decode-able paths should use surrogateescape's
http://www.python.org/dev/peps/pep-0383/
vstinner commentedon Aug 31, 2010
The problem is not specific to Py_CompileString(): all functions based (indirectly) on PyParser_ASTFromString() and PyParser_ASTFromFile() expect filenames encoded in utf-8 with the strict error handler.
If we choose to use something else than utf-8 in strict mode, here is an incomplete list of functions that have to be patched:
And the list of impacted functions (parsing functions accepting filenames):
All these functions are public and I don't think that it would be a good idea to change the encoding (eg. to iso-8859-1). We can use a different error handler (especially surrogateespace, as suggested in the initial message) and/or create new functions accepting unicode filenames.
--
I'm working on undecodable filenames in issues bpo-8611 and bpo-9425, especially on the import machinery part. When the import machinery will be fully unicode compliant, the last part will be the "parser machinery" (Parser/*.c). It is a little bit more complex to patch the parser because there is the bootstrap problem: the parser is compiled twice, once with a small subset of the C Python API (using some mockups), once with the full API.
vstinner commentedon Sep 9, 2010
bpo-6543 changed the encoding of the filename argument of PyRun_SimpleFileExFlags() (and all functions based on PyRun_SimpleFileExFlags) and c_filename attribute of the compiler (private) structure in Python 3.1.3: use utf-8 in strict mode instead of filesystem encoding with surrogateescape.
vstinner commentedon Oct 15, 2010
See also issue bpo-10114.
vstinner commentedon Oct 19, 2010
See issue bpo-10114: fixed in Python 3.1 (r85716) and in Python 3.2 (r85569+r85570).