Repository navigation
subprocess.Popen(universal_newlines=True) does not work for certain locales #59800
Description
| BPO | 15595 |
|---|---|
| Nosy | @loewis, @vstinner, @asvetlov, @cjerdonek |
| Files |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = <Date 2012-08-19.19:22:45.666>
created_at = <Date 2012-08-08.21:36:14.627>
labels = ['easy', 'type-bug', 'library']
title = 'subprocess.Popen(universal_newlines=True) does not work for certain locales'
updated_at = <Date 2012-08-19.19:22:45.665>
user = 'https://github.com/cjerdonek'bugs.python.org fields:
activity = <Date 2012-08-19.19:22:45.665>
actor = 'asvetlov'
assignee = 'none'
closed = True
closed_date = <Date 2012-08-19.19:22:45.666>
closer = 'asvetlov'
components = ['Library (Lib)']
creation = <Date 2012-08-08.21:36:14.627>
creator = 'chris.jerdonek'
dependencies = []
files = ['26728', '26729', '26733', '26734', '26899']
hgrepos = []
issue_num = 15595
keywords = ['patch', 'easy']
message_count = 15.0
messages = ['167719', '167720', '167721', '167722', '167727', '167741', '167744', '167745', '167748', '167750', '167753', '168585', '168587', '168588', '168589']
nosy_count = 5.0
nosy_names = ['loewis', 'vstinner', 'asvetlov', 'chris.jerdonek', 'python-dev']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue15595'
versions = ['Python 3.2', 'Python 3.3']
Activity
cjerdonek commentedon Aug 8, 2012
subprocess.Popen() with universal_newlines=True does not convert line breaks correctly when the preferred encoding is UTF-16. For example, the following code--
yields--
('a\n\nb', None)
instead of--
('a\nb', None)
The reason is that the code attempts to convert newlines before decoding to unicode instead of after:
http://hg.python.org/cpython/file/85266c6f9ae4/Lib/subprocess.py#l830
I am attaching a failing test case. I will upload a patch shortly.
Also see the related documentation bpo-15561.
vstinner commentedon Aug 8, 2012
What is your operating system? How do you set the locale encoding to UTF-16? Do you mean UTF-16, UTF-16-LE or UTF-16-BE?
cjerdonek commentedon Aug 8, 2012
Attaching patch for review.
cjerdonek commentedon Aug 8, 2012
Victor, I have not personally experienced this issue. I just noticed that the order of operations is wrong or not portable in the _translate_newlines() method when I was looking at the code for another reason.
cjerdonek commentedon Aug 8, 2012
I confirmed that the issue occurs for all of these. For testing purposes, you can do--
vstinner commentedon Aug 9, 2012
There is a bug, but I'm not conviced that multibyte encodings are used as locale encoding.
About your patch: you should test the 3 types of newlines, so use a string like: '1\r\n2\r3\n4'.
+ # Popen() defaults to locale.getpreferredencoding(False).
+ locale.getpreferredencoding = lambda do_setlocale: 'utf-16'
FYI it's not directly Popen() which uses the locale encoding, but TextIOWrapper.
cjerdonek commentedon Aug 9, 2012
Thanks for your comments.
Yes, I will note that to be more clear. Would you like me to add tests for UTF-16-LE, etc. (via a helper test method), or will UTF-16 alone suffice?
vstinner commentedon Aug 9, 2012
You can use a loop insteadd of an helper function.
I fail to see other encoding having the same issue, except the UTF-32 family (utf-32, utf-32-le, utf-32-be).
I don't think that writing a test for the 6 codecs is required, you can use utf-16 and utf-32-be. So you test without BOM and with BOM, and utf-16 and utf-32.
cjerdonek commentedon Aug 9, 2012
Updating patch with Victor's comments.
cjerdonek commentedon Aug 9, 2012
It looks like this also affects 3.2, but I will need to modify the test slightly because in 3.2, TextIOWrapper calls locale.getpreferredencoding() without any arguments.
cjerdonek commentedon Aug 9, 2012
Here is a patch also suitable for applying to Python 3.2.
asvetlov commentedon Aug 19, 2012
Looks good for me, but please use assertEqual instead assertEquals.
cjerdonek commentedon Aug 19, 2012
Thank you, Andrew. Here is a patch updated with that change (and also merging with tip).
python-dev commentedon Aug 19, 2012
New changeset a0f7c2f79bce by Andrew Svetlov in branch '3.2':
Issue bpo-15595: Fix subprocess.Popen(universal_newlines=True)
http://hg.python.org/cpython/rev/a0f7c2f79bce
New changeset aceb820154c3 by Andrew Svetlov in branch 'default':
Issue bpo-15595: Fix subprocess.Popen(universal_newlines=True)
http://hg.python.org/cpython/rev/aceb820154c3
asvetlov commentedon Aug 19, 2012
Fixed. Thank you.