Repository navigation
[Python2] Use utf-8 in the import machinery on Windows to support unicode paths #43943
Description
| BPO | 1552880 |
|---|---|
| Nosy | @loewis, @brettcannon, @theller, @kristjanvalur, @vstinner, @ezio-melotti, @merwok |
| Files |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = <Date 2010-09-04.14:44:13.397>
created_at = <Date 2006-09-05.18:11:31.000>
labels = ['interpreter-core']
title = '[Python2] Use utf-8 in the import machinery on Windows to support unicode paths'
updated_at = <Date 2010-10-19.02:21:59.029>
user = 'https://github.com/kristjanvalur'bugs.python.org fields:
activity = <Date 2010-10-19.02:21:59.029>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2010-09-04.14:44:13.397>
closer = 'loewis'
components = ['Interpreter Core']
creation = <Date 2006-09-05.18:11:31.000>
creator = 'kristjan.jonsson'
dependencies = []
files = ['7526', '7527']
hgrepos = []
issue_num = 1552880
keywords = ['patch']
message_count = 23.0
messages = ['51081', '51082', '51083', '51084', '51085', '51086', '51087', '51088', '51089', '81636', '114818', '114820', '114830', '115283', '115284', '115329', '115354', '115553', '115575', '115683', '115686', '115691', '119109']
nosy_count = 10.0
nosy_names = ['loewis', 'nnorwitz', 'brett.cannon', 'anthonybaxter', 'theller', 'kristjan.jonsson', 'vstinner', 'ezio.melotti', 'eric.araujo', 'BreamoreBoy']
pr_nums = []
priority = 'normal'
resolution = 'out of date'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue1552880'
versions = ['Python 2.6']
Activity
kristjanvalur commentedon Sep 5, 2006
This patch modifies the import mechanism to fully
support unicode pathnames on Windows. It does this by
first converting each member of sys.path to utf-8.
strings are encoded using the current locale.
The whole of the import logic is then unchanged and
works on the utf-8 strings as though they were regular
ascii strings in the current locale.
Only when file operations are done, such as stat() and
open(), do we then convert from utf-8 back to unicode
and use the Windows unicode APIs for the job. This is
also done when initializing Module objects.
This approach has the benefit of being of having a low
impact on the importing logic, and is thus easy to
verify. There is however some overhead with the
conversions.
At CCP games we used this approach, backported to
python 2.3, to get unicode imports working for our
game, EVE Online, and thereby solving installation
issues in the far east.
This patch is submitted as demonstration code to the
python community. I would like to see unicode fully
supported in 2.6.
Cheers,
Kristján
loewis commentedon Sep 8, 2006
Logged In: YES
user_id=21627
What is the value of the __file__ attribute of a module when
this patch is used?
kristjanvalur commentedon Sep 9, 2006
Logged In: YES
user_id=1262199
From the top of my head, it is now unicode. I consider
trying to convert it back to the default encoding but
decided not to to keep the patch brief.
loewis commentedon Sep 9, 2006
Logged In: YES
user_id=21627
First: Do you want to continue to work on this, or do you
consider this just "demonstration code" (i.e. not
contributed for inclusion in Python), hoping that somebody
else implements this feature?
I think the behavior of __file__ must be more consistent
across platforms, and the selected behaviour must be
documented somewhere. Several definitions of "consistent
behavior" come to mind:
else Unicode
encoding, else Unicode.
The documentation needs to be updated in several places,
e.g. also for inspect.getfile.
I would expect that pydoc would also need to be updated.
Selecting from the options above: I believe 4 is most
compatible with previous versions; 1 and 2 are most
convenient to work with in applications like pydoc which
have to generate HTML (1 is easier to work with, 2 is more
compatible with previous versions).
kristjanvalur commentedon Sep 12, 2006
Logged In: YES
user_id=1262199
I submitted this mostly as a demonstration. I don't think
the approach is necessarily suitable for a final
implementation because of the use of utf-8 as an
intermediate representation and the price of the conversions
that keep happening. But perhaps this is the way to go, if
we consider utf-8 to be a stage-1 default file system
encoding for win32.
I also agree that 4 is probably the most sensible approach.
What about discrepancies between e.g. linux and windows
then, when including from a non-trivial path? On linux we
would get utf-8, on windows unicode?
this tends to lead to a kind of unicode-hell since a program
touched by one unicode object tends to have it percolating
down into every corner.
anthonybaxter commentedon Sep 12, 2006
Logged In: YES
user_id=29957
There's a variety of modules in the standard library that
reference __file__ - if it's potentially going to be a
unicode string, these are going to need to be checked, as
are their callers :-/
(Now that I've looked closer at some of the issues, I'm
extremely glad this didn't go into 2.5 final at this late stage)
loewis commentedon Sep 12, 2006
Logged In: YES
user_id=21627
krisvale: indeed, option 4 is platform dependent. Notice
that on Linux, the file system encoding won't necessarily be
UTF-8. Instead, the value depends on the locale, so it may
be latin-1, latin-9, gb2312, ... This makes it even more
dependent on the platform, and even the current user being
logged in (such is life with locale-based approaches; the
same is mostly true for Windows: "mbcs" can mean nearly
anything).
option 1) is Py3k-safe, where path names will be Unicode
strings always. As you say, Unicode is a virulent type, so
this approach would need a wide consensus.
I'm personally leaning towards option 2: it is nearly
backwards compatible, except for obscure cases where people
have mbcs-encodable entries in sys.path already, and it is
independent of manipulations of the system encoding.
I also think that processing of PYTHONPATH should take
Unicode into account, i.e. we should use _wgetenv to access
PYTHONPATH in 2.6. That would make the feature truly useful,
as then people could actually set sys.path to non-mbcs
directlories from the outside. Notice that W9x support can
be dropped in 2.6, so a W9x-compatible solution won't be
required.
In any case, I'd like to encourage you to continue working
on this issue. I, too, like to see it in 2.6, but I did so
ever since 2.1 or so (before PEP-277 was implemented), and
it was wishful thinking. Somebody has to take action, and it
is likely that it won't one of the past regular contributors
(or else they had contributed it long ago - although I think
Thomas Heller had something working at one point).
kristjanvalur commentedon Apr 17, 2007
I have uplodaed unicodepatch4.patch, which simplifies this a bit. __file__ and __path__ components are now stored in filesystemencoding if possible. and non-unicode paths are assumed to be in filesystemencoding. This minimizes the impact of the change.
File Added: Unicodeimport4.patch
nnorwitz commentedon Apr 19, 2007
Any function which is not static to a file, must be prefixed with Py or _Py. There are several lines which are over 80 columns and should be wrapped. Why is errno set in open_utf8, etc?
Indentation was messed up at least in one place in Objects/moduleobject.c on a DECREF line.
I can't provide any guidance on the windows specific code. Where do _wstat and _wfopen come from? There isn't a man page on my Unix box. I'm not sure if the exist in a library anywhere. I didn't see any changes to configure to verify if these exist or not. If Py_UNICODE_IMPORT, does that necessarily mean these APIs exist? (It's possible this code was inside an #if WINDOWS and I couldn't tell from the patch.)
kristjanvalur commentedon Feb 11, 2009
Ah, this one is still alive?
We still use this patch at CCP for our 2.x python. I'll give it some
more love to answer the issues raised.
Hm, is this still an issue with 3.x? Does the imput machinery use
unicode as the internal format when working with the import paths?
4 remaining items
vstinner commentedon Aug 31, 2010
utf-8 codec (in strict mode) rejects surrogates in python3, and so you doesn't support undecodable filenames (filenames decoded using surrogateescape error handler which produces surrogate characters). It may be possible if you use surrogateescape everywhere.
Manipulate encoded filenames is not trivial because it may quickly lead to mojibake if the encodings are different (eg. if sys.path contains a bytes filename, you have to be careful). Use utf-8 means that you have to decode and then reencode (to the filesystem encoding) a filename before passing it to a system call (eg. mkdir()). bpo-8611 problem is that Python3 doesn't work if the filesystem is *not* utf-8.
You solution is attractive because it is short, but I prefer to use directly the right solution to not patch Python twice: use unicode (with surrogates, PEP-383, for undecodable filenames) everywhere.
kristjanvalur commentedon Sep 1, 2010
I conffess that I didn't follow the utf-8/surrogate discussion.
But the utf-8 encoding can encode all valid unicode characters:
UTF-8 may only legally be used to encode valid Unicode scalar values. According to the Unicode standard the high and low surrogate halves used by UTF-16 (U+D800 through U+DFFF) and values above U+10FFFF are not legal Unicode values, and the UTF-8 encoding of them is an invalid byte sequence and should be treated as described above. (from wikipedia)
If we encounter surrogate halves when encoding (unicode) to utf-8, it means that we are really trying to decode utf-16 and reencode it as utf-8. (and that python is using 16 bits for its unicode chars). the utf--8 codec should be smart enough to merge the surrogates into a utf-32 char, and encode that.
Anyway, as you remark, my approach is a _patch_, designed to make python (2.x) work in an unicode environment, with the least amount of code change, for those willing to commit such a patch. In 3.x you may want to do things differently.
vstinner commentedon Sep 1, 2010
Yes, but in Python, U+DC80..D+DCFF range is used to store undecodable bytes.
Eg. 'abc\xff'.decode('ascii', 'surrogateescape') gives 'abc\udcff'.
Python 2.7 is out and I think it is too late to fix Python2. Anyway, Python2
uses bytes for sys.path or other paths, so the problem only occurs if the user
specifies unicode paths.
I choosed to rewrite the C code to manipulate unicode paths instead of byte
paths => bpo-9425
kristjanvalur commentedon Sep 2, 2010
That's an inventive way of breaking the unicode standard :)
Anyway, why would you worry about that? My patch doesn't use "surrogateescape" so there is no problem. There are only two places where I "decode":
If a _unicode_ input (sys.path) contains a valid surrogate pair, then the utf-8 encoder just encodes it.
But if it finds a lone surrogate as you describe (python special) then that represends an undecodable chacater, something that should have been covered earlier and something we know nothing about. Clearly, that makes that particular unicode sys.path component invalid.
(Hm, I notice that 2.7 happily encodes lone surrogates to utf-8)
vstinner commentedon Sep 4, 2010
Oh, I didn't see that the issue was specific to Python2. I updated the issue's title. If I understood correctly, the issue is also specific to Windows.
Do you know if your patch changes the public API? (break the compatibility)
--
FYI about Python3:
It is described in the PEP-383 and it does solve a real and common issue: store a filename that cannot be decoded with the filesystem encoding. The operation is reversible. In Python 3.2, there are os.fsdecode() and os.fsencode() functions. On UNIX/BSD, os.encode(os.fsdecode(x)) is x, if x is a bytes object.
The PEP-383 introduces the surrogateescape error handler which does create surrogates on decode, and convert back surrogates to bytes on encode.
In Python3, filenames are stored as unicode. On UNIX/BSD, if a filename cannot be decode, it is encoded with surrogates. To get a full unicode support in Python3, you have to support surrogates.
[-]Unicode Imports[/-][+][Python2] Use utf-8 in the import machinery on Windows to support unicode paths[/+][-]Unicode Imports[/-][+][Python2] Use utf-8 in the import machinery on Windows to support unicode paths[/+]loewis commentedon Sep 4, 2010
As this was never meant for inclusion in Python, and apparently confuses people, I'm closing it - it couldn't go into 2.x, anyway.
kristjanvalur commentedon Sep 6, 2010
Well, it was, originally, but it met with so little interest that I couldn't be bothered to polish it to inclusion standards. Anyway, there was the incompatibility problem of what to do with the __file__ attribute, and the fact that the patch was Windows only.
Do we have a place where we can put in working patches for people to use at their own risk, without going through all the hoops of a successful python.org checkin?
merwok commentedon Sep 6, 2010
There is no such place that I know of, sorry.
loewis commentedon Sep 6, 2010
Having patches in the tracker is fine to me. Even if the patch is closed, it's still available.
Of course, there are many ways to publish code on the net: you could post the patch to Rietveld, to the Python wiki, or publish an entire clone to bitbucket.
vstinner commentedon Oct 19, 2010
FYI, I finished my work on non-ascii filenames in Python 3.2 (bpo-8611, bpo-9425): Python 3.2 now suports any filename with any locale (filesystem) encoding.