os.listdir can return byte strings

@gvanrossum

BPO	3187
Nosy	@gvanrossum, @loewis, @birkenfeld, @amauryfa, @pitrou, @vstinner, @benjaminp, @djc
Files	posix_path_bytes.patch: Patch posixpath.join() to support bytes io_byte_filename.patch: open() allows bytes filename fnmatch_bytes.patch: Patch fnmatch.filter() to accept bytes filenames glob1_bytes.patch: Fix glob.glob() to accept invalid directory name listdir_encoding_warning.patch warn_at_the_end.patch raise_decoding_errors.patch force_unicode.patch getcwd_bytes.patch: getcwd() returns bytes if unicode conversion fails merge_os_getcwd_getcwdu.patch: Remove os.getcwdu(); os.getcwd(bytes=True) returns bytes os_getcwdb.patch: Fix getcwd() (use PyUnicode_Decode) and create getcwdb()->bytes python3_bytes_filename.patch: Patch for an initial support of bytes filename in Python3 setfsenc.diff python3_bytes_filename-3.patch: Patch for an initial support of bytes filename in Python3 (version 3) win32-bytes-filenames.patch macpath.patch library_os_doc.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/birkenfeld'
closed_at = <Date 2008-10-07.07:12:42.880>
created_at = <Date 2008-06-24.10:28:25.633>
labels = ['type-bug', 'docs']
title = 'os.listdir can return byte strings'
updated_at = <Date 2008-10-07.07:12:42.879>
user = 'https://bugs.python.org/HWJ'

bugs.python.org fields:

activity = <Date 2008-10-07.07:12:42.879>
actor = 'loewis'
assignee = 'georg.brandl'
closed = True
closed_date = <Date 2008-10-07.07:12:42.880>
closer = 'loewis'
components = ['Documentation']
creation = <Date 2008-06-24.10:28:25.633>
creator = 'HWJ'
dependencies = []
files = ['11212', '11213', '11215', '11216', '11549', '11550', '11581', '11630', '11632', '11652', '11655', '11658', '11663', '11680', '11685', '11693', '11721']
hgrepos = []
issue_num = 3187
keywords = ['patch']
message_count = 79.0
messages = ['68674', '68679', '68684', '68685', '68686', '68688', '68689', '70943', '70953', '71525', '71612', '71615', '71624', '71629', '71647', '71648', '71655', '71680', '71699', '71700', '71705', '71748', '71749', '71751', '71752', '71756', '71757', '71769', '71991', '72495', '73362', '73534', '73535', '73540', '73678', '73680', '73688', '73909', '73910', '73911', '73925', '73926', '73992', '73999', '74000', '74006', '74007', '74008', '74027', '74032', '74059', '74080', '74083', '74101', '74173', '74186', '74192', '74222', '74236', '74237', '74240', '74241', '74242', '74246', '74255', '74256', '74257', '74266', '74267', '74268', '74270', '74271', '74275', '74276', '74277', '74409', '74412', '74414', '74426']
nosy_count = 13.0
nosy_names = ['gvanrossum', 'loewis', 'georg.brandl', 'amaury.forgeotdarc', 'pitrou', 'vstinner', 'draghuram', 'benjamin.peterson', 'djc', 'HWJ', 'dlitz', 'zegreek', 'bboissin']
pr_nums = []
priority = 'critical'
resolution = 'accepted'
stage = None
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue3187'
versions = ['Python 3.0']

The script below produces 1664 lines of output before it bails out with
Traceback (most recent call last):
  File "WalkBug.py", line 5, in <module>
for Dir, SubDirs, Files in os.walk('/home/jarausch') :
  File "/usr/local/lib/python3.0/os.py", line 278, in walk
for x in walk(path, topdown, onerror, followlinks):
  File "/usr/local/lib/python3.0/os.py", line 268, in walk
if isdir(join(top, name)):
  File "/usr/local/lib/python3.0/posixpath.py", line 64, in join
if b.startswith('/'):
TypeError: expected an object with the buffer interface

=========================
file WalkBug.py:

#!/usr/local/bin/python3.0

import os

for Dir, SubDirs, Files in os.walk('/home/jarausch') :
  print("processing {0:d} files in {1}".format(len(Files),Dir))

Could you tell us what this 1665th line should be?
Maybe the 1665th directory has something special (a filename with spaces
or non-ascii chars...)

Can you try with an older version of python?

It's failing because he's giving a string to bytes.startswith when it
requires a byte string or such.

"he's giving a string"... the user simply called os.walk, which accepts
strings AFAIK.

We should discover what produced this bytestring. Does listdir() returns
a mixed list of strings and bytes?

It seems the conversion to unicode strings (PyUnicode vs PyBytes) was
not complete in os.listdir. See the attached patch.

The original problem seems to come from some Unix platform, but this
patch only handles two cases:

on win32, when the argument is a bytestring.
on OS/2.
And in both cases, the default (utf-8) conversion seems wrong. Something
like cp1252 (the ANSI code page for Western Windows) would be more sensible.

In the posix part of the function, there is the comment (2003-03-04):
/* fall back to the original byte string, as
discussed in patch bpo-683592 */
btw, I find the penultimate message of this other thread very pleasant,
in the py3k context... I suppose the conclusions would not be the same
today.

> Could you tell us what this 1665th line should be?
> Maybe the 1665th directory has something special (a filename with >>
> spaces or non-ascii chars...)

Yes, the next directory contains a filename with an iso-latin1 but non-
ascii character

> Can you try with an older version of python?
No problems - runs every night here

The patch (applied to SVN GMT 13:30) does NOT help.

Hmm, I suppose that while the filename is latin1-encoded,
Py_FileSystemDefaultEncoding is "utf-8" and therefore os.listdir fails
decoding the filename and falls back on returning a byte string.
It was acceptable in Python 2.x but is a very annoying problem in py3k
now that unicode and bytes objects can't be mixed together anymore. I'm
bumping this to critical, although there is probably no clean solution.

FWIW, I don't see a need to change macpath.py -- it's only used for
MacOS 9 and the occasional legacy app. OSX uses posixpath.py.

Committed r66779: test_ntpath now passes with the -bb option.

It seems that the Windows buildbots do not set -bb.

Thanks Amaury!

On to Georg for doc tweaks. Summary:

all the os.path functions now work on bytes as well, on all platforms
only on Unix (but not OSX) do we recommend using bytes
os.getcwdu() no longer exists
os.getcwdb() returns bytes
os.listdir(<str>) skips undecodable entries (previously it returned a
mixture of str and bytes instances)
open() accepts bytes as filename

Stuff that didn't change but that you might want to mention:

all the syscalls in os support bytes args; readlink() and listdir()
return bytes if the arg is bytes
getcwd() may raise UnicodeDecodeError

Martin already documented sys.setfilesystemencoding().

I have a patch for macpath.py nonetheless.
Tested on Windows (of course ;-) but all functions are pure text
manipulation, except realpath(). It was much easier than ntpath.py.

I also added tests for three functions which were not exercised at all.

Amaury, you're patch looks good.

Committed macpath.py in r66781.

Would it possible to close this issue since os.listdir() is fixed and
many other related functions (posix, posixpath, ntpath, macpath, etc.)
are also fixed? I propose to open new issues for new bugs since this
issue becomes a little big long :)

Eg. see new issues bpo-4035 and bpo-4036!

Would it possible to close this issue since os.listdir() is fixed and
many other related functions (posix, posixpath, ntpath, macpath, etc.)
are also fixed?

IIUC, these fixes are still not complete: they lack documentation
changes. Of course, it would have been better if the original patches
already contained the necessary documentation and test suite changes.
See msg74271 for what Guido considers the lacking documentation;
you may find that other aspects also need documentation.

As for test cases: it seems that those got waived, in the hurry.

Le Tuesday 07 October 2008 01:13:22 Martin v. Löwis, vous avez écrit :

IIUC, these fixes are still not complete: they lack documentation
changes. (...) Of course, it would have been better if the original patches
already contained the necessary documentation and test suite changes.

Most (or all) patches include new tests about bytes. Here is a patch for
os.rst documentation about listdir(), getcwdb() and readlink().

See msg74271 for what Guido considers the lacking documentation;
you may find that other aspects also need documentation.

I wrote a long document about bytes for filenames but not only. I'm still
waiting for some contributors or reviewers:
http://wiki.python.org/moin/Python3UnicodeDecodeError

As for test cases: it seems that those got waived, in the hurry.

Can you be more precise? Which tests have to be improved/rewritten?

Most (or all) patches include new tests about bytes. Here is a patch for
os.rst documentation about listdir(), getcwdb() and readlink().

Thanks! Committed as r66829.

I've added additional documentation in r66830, which should complete
Guido's list of things to be documented. So the issue can be closed
now.

> See msg74271 for what Guido considers the lacking documentation;
> you may find that other aspects also need documentation.

I wrote a long document about bytes for filenames but not only. I'm still
waiting for some contributors or reviewers:
http://wiki.python.org/moin/Python3UnicodeDecodeError

We should discuss that on python-dev, of course - the question is
whether additional documentation patches are needed in response to
this specific change.

> As for test cases: it seems that those got waived, in the hurry.

Can you be more precise? Which tests have to be improved/rewritten?

I was probably looking at the wrong patches (such as getcwd_bytes.patch,
merge_os_getcwd_getcwdu.patch, etc); I now see that the final patch did
have tests. I recommend that patches that get superseded by other
patches are removed from the issue. The won't be deleted; it's still
possible to navigate to them through the History at the bottom of the
issue.

Navigation Menu

Repository navigation

os.listdir can return byte strings #47437

85 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

os.listdir can return byte strings #47437

Description

Activity

HWJ commented on Jun 24, 2008

amauryfa commented on Jun 24, 2008

benjaminp commented on Jun 24, 2008

amauryfa commented on Jun 24, 2008

benjaminp commented on Jun 24, 2008

amauryfa commented on Jun 24, 2008

HWJ commented on Jun 24, 2008

pitrou commented on Aug 9, 2008

85 remaining items

gvanrossum commented on Oct 3, 2008

amauryfa commented on Oct 3, 2008

gvanrossum commented on Oct 3, 2008

amauryfa commented on Oct 3, 2008

benjaminp commented on Oct 3, 2008

amauryfa commented on Oct 3, 2008

vstinner commented on Oct 6, 2008

loewis commented on Oct 6, 2008

vstinner commented on Oct 6, 2008

loewis commented on Oct 7, 2008

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions