Repository navigation
shlex have problems with parsing unicode #45511
Description
| BPO | 1170 |
|---|---|
| Nosy | @loewis, @rhettinger, @mdickinson, @abalkin, @orsenthil, @vstinner, @benjaminp, @mcepl, @ezio-melotti, @merwok, @dhellmann, @bitdancer, @jewettaij |
| Dependencies | |
| Files |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = <Date 2021-09-20.07:18:12.357>
created_at = <Date 2007-09-17.15:53:56.721>
labels = ['type-bug', 'docs']
title = 'shlex have problems with parsing unicode'
updated_at = <Date 2021-09-29.07:33:59.402>
user = 'https://bugs.python.org/dexen'bugs.python.org fields:
activity = <Date 2021-09-29.07:33:59.402>
actor = 'jewett-aij'
assignee = 'none'
closed = True
closed_date = <Date 2021-09-20.07:18:12.357>
closer = 'vstinner'
components = ['Documentation']
creation = <Date 2007-09-17.15:53:56.721>
creator = 'dexen'
dependencies = ['10587']
files = ['9025', '18224', '23161']
hgrepos = []
issue_num = 1170
keywords = ['patch']
message_count = 51.0
messages = ['55967', '55968', '55969', '58964', '77972', '77973', '91853', '106424', '109292', '110854', '110856', '111594', '111709', '111710', '111716', '111717', '111718', '111719', '111751', '111755', '111756', '112911', '122919', '122929', '140227', '140246', '140262', '140516', '140518', '140529', '140586', '140590', '140591', '140592', '144074', '144076', '144077', '144080', '144097', '144193', '144201', '144202', '146219', '146220', '348612', '402187', '402200', '402821', '402822', '402824', '402825']
nosy_count = 19.0
nosy_names = ['loewis', 'rhettinger', 'mark.dickinson', 'belopolsky', 'orsenthil', 'vstinner', 'dexen', 'benjamin.peterson', 'cgwalters', 'mcepl', 'ezio.melotti', 'eric.araujo', 'doughellmann', 'r.david.murray', 'nwerneck', 'fperez', 'Santiago.Romero', 'wombat', 'jewett-aij']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue1170'
versions = ['Python 3.5']
Activity
dexen commentedon Sep 17, 2007
Feeding unicode to shlex object created in POSIX compat mode causes
UnicodeDecodeError to be raised. It appears that shlex object defines
sting .wordchars, containing latin-1 (iso8859-1) encoded characters
with charcodes >=128, which is used to check whether a character from
input constitues a word character or not.
dexen commentedon Sep 17, 2007
A quick paste to illustrate: the exception is raised only when unicode
object is passed to shlex. Warning: the cStringIO module is unsuitable
for use there, only the StringIO. cStringIO does not output unicode.
dexen commentedon Sep 17, 2007
One remark to previous message:
the first time i created shlex object in non-POSIX mode (the default),
in later it's in POSIX mode (due to the third parameter to shlex being
True). The bug in question manifests only in POSIX mode.
BTW, that so-called POSIX mode would be more POSIX-ish, if instead of
comparing characters with a fixed, short list, would use the ctype()
function as found in standard C library. The functions takes current
locale (setable in process) into account when deciding what is leter,
whitespace, punctuation etc.
cgwalters commentedon Dec 22, 2007
Patch to add Unicode support.
Note: this patch recodes shlex.py from iso-8859-1 to utf-8, so it has
mixed encodings.
nwerneck commentedon Dec 17, 2008
Hello. I tried to patch my own shlex, and this doens't seem to be
working properly. When I try the patched module isntead of th eoriginal,
in my otherwise working program, I get the result ahead.
Is there any conversion steps missing?...
nwerneck commentedon Dec 17, 2008
OK, it worked after I found out I didn't know how to open unicode
files... Sorry for the noise, and thanks for this patch! :]
benjaminp commentedon Aug 22, 2009
The patch needs tests before it can be applied. Additionally, I'm not
sure if having a "utf" option is helpful. Is there a reason not to have
unicode support by default?
merwok commentedon May 25, 2010
shlex in 3.x works with Unicode strings. Is it still time to try to fix this bug for 2.7?
bitdancer commentedon Jul 5, 2010
shlex may use unicode in py3k, but since the file still starts with a latin-1 coding cookie and the posix logic hasn't been changed, I suspect that it does not work correctly (ie: does not correctly identify word characters, per msg55969).
It's too late for 2.7 I think, but it seems there is work still to do in py3k.
abalkin commentedon Jul 20, 2010
As discussed in msg110828 under bpo-9308, it is not clear whether logic identifying word characters in shlex is correct in presence of unicode.
abalkin commentedon Jul 20, 2010
I believe the e-mail thread that culminated in r32284, "Implemented posix-mode parsing support in shlex.py", was "shellwords" from April 2003:
http://mail.python.org/pipermail/python-dev/2003-April/034670.html
I scanned through the messages, but could not find a reference to the standard that was implemented.
33 remaining items
bitdancer commentedon Sep 17, 2011
Ezio, I don't see any indication in this ticket that this bug was actually *fixed* in 3.x. Unicode doesn't cause immediate errors in 3.x, but it isn't recognized as wordchars, etc. Am I missing something?
ezio-melotti commentedon Sep 17, 2011
I haven't looked at the shlex code (yet), my comment was just about the idea of adding constants with chars that belong to different Unicode categories.
merwok commentedon Oct 23, 2011
This bug was fixed indirectly by a StringIO fix in 27ae7d4e1983, for bpo-1548891. BTW, this report was a duplicate of bpo-6988, closed a year ago.
Python 2.7.3 will finally support unicode in shlex, so the doc change requested in this report is outdated. However, I still want to do something for this. I’ve noticed that shlex.split’s argument can be a file-like object, and I wonder if passing a StringIO.StringIO(my_unicode_string) wouldn’t work. If such a short recipe works, I’m all for including it in the 2.7 docs for users of older versions. If a longer recipe is needed, then ActiveState’s Python Cookbook would be more appropriate, and I’ll add a link to the docs. If it’s very long and requires a PyPI project, then I’m willing to link to that.
merwok commentedon Oct 23, 2011
The second message in this page reports that StringIO.StringIO works, but when I pass a unicode string with non-ASCII chars there’s a method call that fails because of implicit unicode-to-str conversion:
I’ll try to create a Shlex instance, replace self.wordchars with a decoded version and try again.
vstinner commentedon Jul 29, 2019
This issue is 12 years old has 3 patches: it's far from being "newcomer friendly", I remove the "Easy" label.
mcepl commentedon Sep 19, 2021
I cannot reproduce it with the current 3.* version. Did anybody reproduce with 3.5?
Otherwise, I suggest close this, as a 2.* bug.
vstinner commentedon Sep 20, 2021
This issue has been fixed in Python 3 by using Unicode rather than bytes in shlex. Python 2 users: it's time to upgrade to Python 3 ;-)
jewettaij commentedon Sep 29, 2021
The error messages may have gone away, but the underlying unicode limitations I mentioned remain:
Suppose you wanted to use shlex to build a parser for Chinese text. Would you have to set "wordchars" to a string containing every possible Chinese character?
I myself wrote a parser for a crude language where words can contain any character except for whitespace and parenthesis. I needed a way to specify the characters which cannot belong to a word. (That's how I solved the problem. I modified shlex.py and added a "wordterminators" member. If "wordterminators" was left blank, then "wordchars" were used instead. This was a trivial change to "shlex.py" and it added a lot of functionality.)
I would like to suggest making this change (or something similar) to the official version of "shlex.py". Would sending an email to "[email protected]" be a good place to make this proposal?
vstinner commentedon Sep 29, 2021
Yes, python-ideas is a good place to start discussion such idea. This issue is closed, if you discuss it here, you will get a limited audience.
jewettaij commentedon Sep 29, 2021
After posting that, I noticed that the second example I listed in my previous post (a language where words contain any non-whitespace, non-parenthesis character) can now be implemented in the current version of shlex.py by setting "whitespace_true" and "punctuation". (Sorry, it's been a while since I looked at shlex.py, and it's had some usefl new features.)
jewettaij commentedon Sep 29, 2021
Alright. I'll think about it a little more and post my suggestion there, perhaps. Thanks Victor.