Repository navigation
unicode support for os.listdir() #37949
Description
| BPO | 683592 |
|---|---|
| Nosy | @malemburg, @gvanrossum, @loewis, @jackjansen |
| Files |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = 'https://github.com/loewis'
closed_at = <Date 2003-03-04.19:43:47.000>
created_at = <Date 2003-02-09.21:43:09.000>
labels = ['library']
title = 'unicode support for os.listdir()'
updated_at = <Date 2003-03-04.19:43:47.000>
user = 'https://bugs.python.org/jvr'bugs.python.org fields:
activity = <Date 2003-03-04.19:43:47.000>
actor = 'jvr'
assignee = 'loewis'
closed = True
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2003-02-09.21:43:09.000>
creator = 'jvr'
dependencies = []
files = ['5013', '5014']
hgrepos = []
issue_num = 683592
keywords = ['patch']
message_count = 43.0
messages = ['42747', '42748', '42749', '42750', '42751', '42752', '42753', '42754', '42755', '42756', '42757', '42758', '42759', '42760', '42761', '42762', '42763', '42764', '42765', '42766', '42767', '42768', '42769', '42770', '42771', '42772', '42773', '42774', '42775', '42776', '42777', '42778', '42779', '42780', '42781', '42782', '42783', '42784', '42785', '42786', '42787', '42788', '42789']
nosy_count = 6.0
nosy_names = ['lemburg', 'gvanrossum', 'loewis', 'nnorwitz', 'jackjansen', 'jvr']
pr_nums = []
priority = 'normal'
resolution = 'accepted'
stage = None
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue683592'
versions = []
Activity
jvr commentedon Feb 9, 2003
The attached patch makes os.listdir() return unicode strings, on plaforms that have Py_FileSystemDefaultEncoding defined as non-NULL.
I'm by no means sure this is the right thing to do; it does seem right on OSX where Py_FileSystemDefaultEncoding is (or rather: will be real soon, I'm waiting for Jack's approval) utf-8. I'd be happy to add the code in an OSX-specific switch.
A more subtle variant could perhaps only return unicode strings if the file name is not ASCII.
gvanrossum commentedon Feb 10, 2003
Logged In: YES
user_id=6380
At the very least, I'd like it to return Unicode only when
the original string isn't just ASCII.
nnorwitz commentedon Feb 10, 2003
Logged In: YES
user_id=33168
The code which uses unicode APIs should probably be wrapped
with:
jvr commentedon Feb 10, 2003
Logged In: YES
user_id=92689
Applied both suggestions.
However, I'm not sure if my ASCII test does the right thing, or at least I don't think it does if Py_FileSystemDefaultEncoding is not a superset of ASCII.
malemburg commentedon Feb 10, 2003
Logged In: YES
user_id=38388
Your test will probably catch most cases, but it could fail
for e.g. UTF-16.
The only true test would be to first convert to Unicode and then
try to convert back to ASCII. If you get an error you can be
sure that
the text is not ASCII compatible. Given that .listdir()
involves lots of
IO I think the added performance hit wouldn't be noticable.
jvr commentedon Feb 10, 2003
Logged In: YES
user_id=92689
I don't see hot UTF-16 could be a valid value for Py_FileSystemDefaultEncoding, as for most platforms the file name can't contain null bytes. My looking at the NAMELEN() spaghetti, it seems platforms without HAVE_DIRENT_H might still support embedded null bytes. Any wisdom on this?
malemburg commentedon Feb 10, 2003
Logged In: YES
user_id=38388
The file system does not need to support embedded \0 chars
even if it supports UTF-16. It only happens that your test
assumes
that you have one byte per characters encodings which may not
always be true. With UTF-16 your test will see lots of \0 bytes
but not necessarily ones which are ord(x)>=128.
I'm not sure whether other variable length encodings can result
in \0 bytes, e.g. the Asian ones.
There's also the possibility of the
encoding mapping the ASCII range to other non-ASCII characters,
e.g. ShiftJIS does this for the Yen sign.
If you absolutely want to use the simple test, I'd at least
restrict
the test to an ASCII isalnum(x) test and then try the
encode/decode
method I described if this test fails.
Note that isalnum() can be locale dependent on some
platforms, so
you have to hard-code it.
jvr commentedon Feb 10, 2003
Logged In: YES
user_id=92689
Ok, I went for your original suggestion: always convert to unicode and then try to convert to ascii. See new patch. Or should this use the default encoding? Hm.
malemburg commentedon Feb 10, 2003
Logged In: YES
user_id=38388
Good question. The default encoding would better fit
into the concept, I guess.
Instead of PyUnicode_AsASCIIString(v) you'd
have to use PyUnicode_AsEncodedString(v, NULL, "strict").
20 remaining items
gvanrossum commentedon Mar 4, 2003
Logged In: YES
user_id=6380
I haven't seen the code, but I have a complaint.
On Linux, when I have a file named '\xff' (i.e. its name is
the single byte with value 255), os.listdir(u'.') gives me a
UnicodeDecodeError.
Is that really progress?
jvr commentedon Mar 4, 2003
Logged In: YES
user_id=92689
Would you prefer the error be silenced and a byte string be used instead? If so, should there be a warning?
loewis commentedon Mar 4, 2003
Logged In: YES
user_id=21627
Guido's scenario was precisely the reason why Unix was left
out from consideration for PEP-277.
However, it is better than it sounds: There is a good chance
that invoking locale.setlocale(locale.LC_CTYPE, "") prior to
invoking listdir will overcome the problem, as the setlocale
call will set the file system encoding to the user's
preference. If \xff is a valid file name in the user's
preferred encoding, then listdir will succeed in converting
this file name to a Unicode string.
It might be useful to set the file system encoding on Unix
to the user's preferred encoding unconditionally (i.e. not
as a side effect of invoking setlocale). It might also be
useful to expose the file system encoding read-only for
inspection.
jvr commentedon Mar 4, 2003
Logged In: YES
user_id=92689
It would seem that even with a user's locale there's a chance os.listdir() fails when passed a unicode argument. I'm not sure it's reasonable for os.listdir() to fail at all (if the directory to be listed exists and we the right permissions).
If it's all too difficult to get right, I'm happy to put the listdir unicode support in a MacOSX switch. I know nothing about locales so I'm really not in a position to straighten this out. All I know is that if Py_FileSystemDefaultEncoding is known to be utf-8, it's just dumb _not_ to return unicode. You guys figure out the rest.
gvanrossum commentedon Mar 4, 2003
Logged In: YES
user_id=6380
The setlocale call indeed works.
I think I'd be happier if this was set by default, but I
don't know what other consequences there would be.
gvanrossum commentedon Mar 4, 2003
Logged In: YES
user_id=6380
Maybe the filesystem default encoding should be set to
Latin-1 by default (when nothing better is known about it)?
Then it's hard to imagine how the conversion could fail,
since every Latin-1 byte maps 1-1 to the corresponding
Unicode code point.
jvr commentedon Mar 4, 2003
Logged In: YES
user_id=92689
I think it would be better to simply return byte strings if the file system encoding isn't know. (This btw. was what my original patch did.)
loewis commentedon Mar 4, 2003
Logged In: YES
user_id=21627
I disagree with the last assertion: In *particular* if the
file system encoding is UTF-8, there is a good chance that
decoding will fail (unlike if it is latin-1; decoding will
then never fail - it may just produce mojibake).
OS X seems to make a guarantee to always return UTF-8 from
its low-level API, but I distrust this guarantee until I see
it with my own eyes :-) E.g. what happens if you mount an
NFS tree, and the NFS server gives file names in some other
encoding?
I see the following options:
only enable the code for OS X. I dislike this option, as
it essentially freezes the Unix status to non-Unicode (we
won't get further insights, the de jure status won't change,
de facto, all files will be encoded in the locale's encoding).
leave the code as-is, documenting the possibility of
exceptions.
add byte strings instead of Unicode strings into the
result for non-decodable strings. This gives a mixed-type
result, which is fine if you only pass the resulting file
names to stat() or open(), and will likely break the
application if it tries to display the file names somehow.
loewis commentedon Mar 4, 2003
Logged In: YES
user_id=21627
Setting the file system encoding on startup should be fine,
except that we need another setlocale/query/restore locale
sequence. This is, in principle, bad, as there is no
guarantee that the restore locale operation really produces
the original state, and may cause problems if other threads
are already running. In practice, it appears to work out
just fine, as we use such sequences already (e.g. to undo
the readline initialization).
jackjansen commentedon Mar 4, 2003
Logged In: YES
user_id=45365
I just did a test (created 254 files with all bytes except / and null in their names on a linux server, mounted the partition over NFS on MacOSX) and indeed MacOSX tries to interpret the bytes as UTF-8 and fails.
I know that conversion works for HFS and HFS+ volumes (which carry a filename encoding with them, or you have to specify it when mounting). I assume it works for AFP and SMB (which also carries encoding info, IIRC) but I can't test this. I haven't a clue about webdav and such.
Something to keep in mind is that we are really trying to solve someone else's problem: the inability of NFS and most unixen to handle file system encodings. If I'm on a latin-1 machine and I nfs-mount your latin-2 partition I will see garbage filenames.
jvr commentedon Mar 4, 2003
Logged In: YES
user_id=92689
Here's a note about file system encodings on OSX, including a few words about NFS: http://developer.apple.com/qa/qa2001/qa1173.html.
I propose to fall back to a byte string if conversion to unicode fails.
loewis commentedon Mar 4, 2003
Logged In: YES
user_id=21627
I only partially agree that this is somebody else's problem:
On Unix, it is always considered application responsibility
to interpret file names as characters if they need to -
hence the lack of a system-provided encoding strategy. So it
is the problem of Python or the Python application, and I
think we should try to shield the application from these
issues as good as we can.
Therefore, I'm in favour of jvr's latest proposal (use byte
strings as the last resort), hoping that the error case will
be unfrequent.
gvanrossum commentedon Mar 4, 2003
Logged In: YES
user_id=6380
On the one hand a user who isn't interested in encodings
shouldn't be passing a Unicode argument. On the other hand,
Unicode strings have a way of sneaking into your application
when you least suspect them. E.g. Tkinter returns them, so
does IDLE, and I see them used more and more in Zope 3.
FWIW, I like Just's "fall back to bytestrings" aproach.
jvr commentedon Mar 4, 2003
Logged In: YES
user_id=92689
I've committed the "fallback-to-byte-strings" behavior.
It's in posixmodule.c rev. 2.290.