Repository navigation
compile() doesn't support the PEP 383 (surrogates) #54323
Description
| BPO | 10114 |
|---|---|
| Nosy | @terryjreedy, @vstinner, @benjaminp |
| Files |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = <Date 2010-10-19.01:22:19.575>
created_at = <Date 2010-10-15.12:34:50.200>
labels = ['interpreter-core', 'expert-unicode']
title = "compile() doesn't support the PEP 383 (surrogates)"
updated_at = <Date 2010-10-19.01:22:19.574>
user = 'https://github.com/vstinner'bugs.python.org fields:
activity = <Date 2010-10-19.01:22:19.574>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2010-10-19.01:22:19.575>
closer = 'vstinner'
components = ['Interpreter Core', 'Unicode']
creation = <Date 2010-10-15.12:34:50.200>
creator = 'vstinner'
dependencies = []
files = ['19246']
hgrepos = []
issue_num = 10114
keywords = ['patch']
message_count = 13.0
messages = ['118762', '118833', '118836', '118837', '118842', '118843', '118845', '118846', '118865', '118866', '118868', '118871', '119099']
nosy_count = 3.0
nosy_names = ['terry.reedy', 'vstinner', 'benjamin.peterson']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = None
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue10114'
versions = ['Python 3.2']
Activity
vstinner commentedon Oct 15, 2010
Example:
Attached patch encodes manually the filename to utf-8 with surrogateescape.
I found this problem while testing Python with an ASCII locale encoding (LANG=C ./python Lib/test/regrtest.py). Example:
terryjreedy commentedon Oct 15, 2010
I think the title is slightly misleading. As I read the patch, the issue is that PyArg_ParseTupleAndKeywords requires that string args to C functions be valid Unicode strings (and that it does this by trying to encode to utf-8). Your patch subverts this by redefining filename to be a generic object, with a looser custom-coded test. It is not clear to me that filename, out of all string args to builtins, should be excepted this way. It seems to me that any real filename should be real unicode and there is no need for fake names that are not.
vstinner commentedon Oct 15, 2010
bpo-6543 changed code->co_filename encoding from filesystem encoding+surrogateescape to utf-8+strict.
With my patch, compile('', '\udcc3\udca9', 'exec').co_filename gives 'é', it doesn't depend on the filesystem encoding. But 'é' cannot be used with all filesystem encodings, eg. with ascii locale encoding (C locale), use it raises an error.
I now think that it was a bad idea to use utf-8 instead of the fileystem encoding. All filenames should use the filesystem encoding in Python.
vstinner commentedon Oct 15, 2010
See also bpo-9713.
vstinner commentedon Oct 15, 2010
Here is a new patch [code_encoding.patch] implementing this idea:
The patch restores the situation before bpo-6543.
terryjreedy commentedon Oct 15, 2010
Pardon my ignorance, but given that code.co_filename is a string attribute given as a string, which is to say, unicode in 3.x, I do not see what filesystem encodings, or any other encoding to bytes should really have to do with the attribute. I actually would have expected compile to take your example argument 'abc\uDC80' and paste it onto the code object unchanged. The only issue to me is whether any string should be allowed or only legal-unicode strings. Anything else would seem like a 2.x holdover.
If not that, what does it do (with no decoding specified)?
vstinner commentedon Oct 15, 2010
co_filename attribute is used to display the traceback: Python opens the related file, read the source code line and display it. On Windows, co_filename is directly used because Windows accepts unicode for filenames. But on other OSes, you have to encode the filename to the filesystem encoding.
If your filesystem encoding is 'ascii' (eg. C locale) and co_filename is a non-ascii filename (eg. 'test_é.py'), encode co_filename will raise a UnicodeEncodeError. You can test it simply by using os.fsencode():
Said differently, co_filename should be encodable to the filesystem encoding (os.fsencode(co_filename) should not raise an error).
vstinner commentedon Oct 15, 2010
Remove [compile_surrogates.patch] because it creates filenames unencode to the filesystem encoding. Eg. compile('', '\udcc3\udca9', 'exec').co_filename gives 'é' even if the filesystem encoding is 'ascii'.
vstinner commentedon Oct 16, 2010
About Python 3.1 compatibility: Python 3.1 doesn't support non-ascii paths with a locale different than utf-8 (see issue bpo-8611), so it doesn't change anything for Python 3.1 (it doesn't work anyway, with utf-8 or filesystem encoding).
vstinner commentedon Oct 16, 2010
Oh, I just realized that Python 3.1.2 (last Python 3.1 release) was released the 21st March, whereas r82063 (commit for bpo-6543) was made the 17st June. So the encoding change was not released yet.
vstinner commentedon Oct 16, 2010
Commited to 3.2 (r85569+r85570). I wait for the buildbot before porting the patch to 3.1 and close the issue. There is already a regression on Gentoo buildbot with ascii locale encoding, test_doctest test_zipimport_support:
http://www.python.org/dev/buildbot/all/builders/AMD64%20Gentoo%20Wide%203.x/builds/106
vstinner commentedon Oct 16, 2010
I created bpo-10123 for the test_doctest regression.
vstinner commentedon Oct 19, 2010
Buildbots are green again (bpo-10123 is closed). I ported the fix to Python 3.1 (r85716). Close this issue.