Repository navigation
\N{...} neglects formal aliases and named sequences from Unicode charnames namespace #56962
Description
| BPO | 12753 |
|---|---|
| Nosy | @malemburg, @gvanrossum, @loewis, @terryjreedy, @abalkin, @ezio-melotti, @florentx |
| Superseder | |
| Files |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = 'https://github.com/ezio-melotti'
closed_at = <Date 2011-10-21.20:22:13.698>
created_at = <Date 2011-08-15.17:48:33.715>
labels = ['interpreter-core', 'type-feature', 'expert-unicode']
title = '\\N{...} neglects formal aliases and named sequences from Unicode charnames namespace'
updated_at = <Date 2013-06-23.22:32:03.145>
user = 'https://bugs.python.org/tchrist'bugs.python.org fields:
activity = <Date 2013-06-23.22:32:03.145>
actor = 'belopolsky'
assignee = 'ezio.melotti'
closed = True
closed_date = <Date 2011-10-21.20:22:13.698>
closer = 'ezio.melotti'
components = ['Interpreter Core', 'Unicode']
creation = <Date 2011-08-15.17:48:33.715>
creator = 'tchrist'
dependencies = []
files = ['22903', '23273', '23280', '23291', '23374']
hgrepos = []
issue_num = 12753
keywords = ['patch', 'needs review']
message_count = 37.0
messages = ['142136', '142145', '142502', '142506', '142507', '142508', '143043', '144679', '144681', '144703', '144708', '144716', '144738', '144739', '144757', '144758', '144760', '144779', '144783', '144802', '144803', '144825', '144827', '144832', '144836', '144839', '145254', '145263', '145327', '145401', '146034', '146036', '146075', '146114', '146129', '146135', '191737']
nosy_count = 10.0
nosy_names = ['lemburg', 'gvanrossum', 'loewis', 'terry.reedy', 'belopolsky', 'ezio.melotti', 'mrabarnett', 'flox', 'python-dev', 'tchrist']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = '4610'
type = 'enhancement'
url = 'https://bugs.python.org/issue12753'
versions = ['Python 3.3']
Activity
tchrist commentedon Aug 15, 2011
Unicode character names share a common namespace with formal aliases and with named sequences, but Python recognizes only the original name. That means not everything in the namespace is accessible from Python. (If this is construed to be an extant bug from than an absent feature, you probably want to change this from a wish to a bug in the ticket.)
This is a problem because aliases correct errors in the original names, and are the preferred versions. For example, ISO screwed up when they called U+01A2 LATIN CAPITAL LETTER OI. It is actually LATIN CAPITAL LETTER GHA according to the file NameAliases.txt in the Unicode Character Database. However, Python blows up when you try to use this:
This unfortunate, because the formal aliases correct egregious blunders, such as the Standard reading "BRAKCET" instead of "BRACKET":
There are only
In Perl, \N{...} grants access to the single, shared, common namespace of Unicode character names, formal aliases, and named sequences without distinction:
It is my suggestion that Python do the same thing. There are currently only 11 of these.
The third element in this shared namespace of name, named sequences, are multiple code points masquerading under one name. They come from the NamedSequences.txt file in the Unicode Character Database. An example entry is:
There are 418 of these named sequences as of Unicode 6.0.0. This shows that Perl can also access named sequences:
Since it is a single namespace, it makes sense that all members of that namespace should be accessible using \N{...} as a sort of equal-opportunity accessor mechanism, and it does not make sense that they not be.
Just makes sure you take only the approved named sequences from the NamedSequences.txt file. It would be unwise to give users access to the provisional sequences located in a neighboring file I shall not name :) because those are not guaranteed never to be withdrawn the way the others are, and so you would risk introducing an incompatibility.
If you look at the ICU UCharacter class, you can see that they provide a more
tchrist commentedon Aug 15, 2011
Here’s the right test file for the right ticket.
terryjreedy commentedon Aug 19, 2011
I verified that the test file raises the quoted SyntaxError on 3.2 on Win7. This:
>>> "\N{LATIN CAPITAL LETTER GHA}" SyntaxError: (unicode error) 'unicodeescape' codec can't decode bytes in position 0-27: unknown Unicode character nameis most likely a result of this:
>>> unicodedata.lookup("LATIN CAPITAL LETTER GHA") Traceback (most recent call last): File "<pyshell#1>", line 1, in <module> unicodedata.lookup("LATIN CAPITAL LETTER GHA") KeyError: "undefined character name 'LATIN CAPITAL LETTER GHA'"Although the lookup comes first in nametests.py, it is never executed because of the later SyntaxError.
The Reference for string literals says"
"\N{name} Character named name in the Unicode database"
The doc for unicodedata says
"This module provides access to the Unicode Character Database (UCD) which defines character properties for all Unicode characters. The data contained in this database is compiled from the UCD version 6.0.0.
The module uses the same names and symbols as defined by Unicode Standard Annex #44, “Unicode Character Database”."
http://www.unicode.org/reports/tr44/tr44-6.html
So the question is, what are the 'names' therein defined?
All such should be valid inputs to
"unicodedata.lookup(name) Look up character by name."
The annex refers to http://www.unicode.org/Public/6.0.0/ucd/
This contains NamesList.txt, derived from UnicodeData.txt. Unicodedata must be using just the latter. The ucd directory also contains NameAliases.txt, NamedSequences.txt, and the file of provisional named sequences.
As best I can tell, the annex plus files are a bit ambiguous as to 'Unicode character name'. The following quote seems neutral: "the Unicode Character Database (UCD), a collection of data files which contain the Unicode character code points and character names." The following: "Unicode character names constitute a special case. Formally, they are values of the Name property." points toward UnicodeData.txt, which lists the Name property along with others. However, "Unicode character name, as published in the Unicode names list," indirectly points toward including aliases. NamesList.txt says it contains the "Final Unicode 6.0 names list." (but one which "should not be parsed for machine-readable information". It includes all 11 aliases in NameAliases.txt.
My current opinion is that adding the aliases might be done in current releases. It certainly would serve the any user who does not know to misspell 'FTHORA' as 'FHTORA' for just one of the 17 'FTHORA' chars.
Adding named sequences is definitely a feature request. The definition of .lookup(name) would be enlarged to "Look up character by name, alias, or named sequence" with reference to the specific files. The meaning of \N{} would also have to be enlarged.
Minimal test code might be:
More what ;-)
I presume ICU =International Components for Unicode, icu-project.org/
"Offers a portable set of C/C++ and Java libraries for Unicode support, software internationalization (I18N) and globalization (G11N)."
[appears to be free, open source, and possibly usable within Python]
tchrist commentedon Aug 19, 2011
"Terry J. Reedy" <[email protected]> wrote
on Fri, 19 Aug 2011 22:50:58 -0000:
Yes, I think the 11 aliases pose no problem. It's amazing the trouble
you get into from having a fat-fingered amanuesis typing your laws
into indelible stone tablets.
But these do. The problem is bracketed character classes.
Yes, if you got named reference into the regex compiler as a raw
string, it could in theory rewrite
as
but that doesn't help if the sequence got replaced as a string escape.
At which point you have different behavior in the two lookalike cases.
If you ask how we do this in Perl, the answer is "poorly". It really only
works well in strings, not charclasses, although there is a proposal to do
a rewrite during compilation like I've spelled out above. Seems messy for
something that might(?) not get much use. But it would be nice for \N{} to
work to access the whole namespace without prejudice. I have a feeling
this may be a case of trying to keep one's cake and eating it too, as
the two goals seem to rule each other out.
More expressive set of lookup functions where it is clear which thing
you are getting. I believe the ICU regexes only support one-char returns
for \N{...}, not multis per the sequences. But I may not be looking
at the right docs for ICU; not sure.
Well, there are some Python bindings for ICU that I was eager to try out,
because I wanted to see whether I couild get at full/real Unicode collation
that way, but I had trouble getting the Python bindings to compile. Not
sure why. The documentation for the Python bindings isn't very um wordy,
and it isn't clear how tightly integrated it all is: there's talk about C++
strings that kind of scares me. :)
Hm, and maybe they are only for Python 2 not Python 3, which I try to do
all my Python stuff in because it seems like it has a better Unicode model.
--tom
mrabarnett commentedon Aug 19, 2011
For the "Line_Break" property, one of the possible values is "Inseparable", with 2 permitted aliases, the shorter "IN" (which is reasonable) and "Inseperable" (ouch!).
tchrist commentedon Aug 19, 2011
Matthew Barnett <[email protected]> wrote
on Fri, 19 Aug 2011 23:36:45 -0000:
Yeahy, I've shaken my head at that one, too.
It's one thing to make an alias for something you typo'd in the first
place, but to have something that's correct which you then make a typo
alias for is just encouraging bad/sloppy/wrong behavior.
And there's still set, which makes you wonder
why they couldn't spell at least *one* of them out:
You really have to look those up to realize they're two different things:
And that none of them have something like SB=Space or SB=Separator
so you know what you're talking about. Grrr.
--tom
gvanrossum commentedon Aug 26, 2011
+1 on the feature request.
ezio-melotti commentedon Sep 30, 2011
The attached patch changes Tools/unicode/makeunicodedata.py to create a list of names and codepoints taken from http://www.unicode.org/Public/6.0.0/ucd/NameAliases.txt and adds it to Modules/unicodename_db.h.
During the lookup the _getcode function at Modules/unicodedata.c:1055 loops over the 11 aliases and checks if any of those match.
The patch also includes tests for both unicodedata.lookup and \N{}.
The patch doesn't include changes for NamedSequences.txt.
loewis commentedon Sep 30, 2011
I propose to use a better lookup algorithm using binary search, and then integrate the NamedSequences into this as well. The search result could be a record
You would have two tables for these: one for the aliases, and one for the named sequences.
_getcode would continue to return a single char only, and thus not support named sequences. lookup could well return strings longer than 1, but only in 3.3.
I'm not sure that \N escapes should support named sequences: people rightfully expect that each escaped element in a string literal constitutes exactly one character.
ezio-melotti commentedon Sep 30, 2011
Leaving named sequences for unicodedata.lookup() only (and not for \N{}) makes sense.
The list of aliases is so small (11 entries) that I'm not sure using a binary search for it would bring any advantage. Having a single lookup algorithm that looks in both tables doesn't work because the aliases lookup must be in _getcode for \N{...} to work, whereas the lookup of named sequences will happen in unicodedata_lookup (Modules/unicodedata.c:1187).
I think we can leave the for loop over aliases in _getcode and implement a separate (and binary) search in unicodedata_lookup for the named sequences. Does that sound fine?
tchrist commentedon Sep 30, 2011
There are certainly advantages to that strategy: you don't have to
deal with [\N{sequence}] issues. If the argument to unicode.lookup()
and be any of name, alias, or sequence, that seems ok. \N{} should
still do aliases, though, since those don't have the complication that
sequences have.
You may wish unicode.name() to return the alias in preference, however.
That's what we do. And of course, there is no issue of sequences there.
The rest of this perhaps painfully long message is just elaboration
and icing on what I've said above.
--tom
If you mean, is it ok to add just the aliases and not the named sequences to
\N{}, it is certainly better than not doing so at all. Plus that way you do
*not* have to figure out what in the world to to do with [^a-c\N{sequence}],
since that would have be something like (?!\N{sequence})[^a-c]), which is
hardly obvious, especially if \N{sequence} actually starts with [a-c].
However, because the one namespace comprises all three of names,
aliases, and named sequences, it might be best to have a functional
(meaning, non-regex) API that allows one to do a fetch on the whole
namespace, or on each individual component.
The ICU library supports this sort of thing. In ICU4J's Java bindings,
we find this:
The first one obviously has a bug in its definition, as the English
doesn't scan. Looking at the full definition is even worse. Rather
than dig out the src jar, I looked at ICU4C, but its own bindings are
completely different. There you have only one function, with an enum to
say what namespace to access:
The UCharNameChoice enum tells what sort of thing you want:
Looking at the src for the Java is no more immediately illuminating,
but I think that "extended" may refer to a union of the old 1.0 names
with the current names.
Now I'll tell you what Perl does. I do this not to say it is "right",
but just to show you one possible strategy. I also am in the middle
of writing about this for the Camel, so it is in my head.
Perl does not provide the old 1.0 names at all. We don't have a Unicode
1.0 legacy to support, which makes this cleaner. However, we do provide
for the names of the C0 and C1 Control Codes, because apart from Unicode
1.0, they don't condescend to name the ASCII or Latin1 control codes.
We also provide for certain well known aliases from the Names file:
anything that says "* commonly abbreviated as ...", so things like LRO
and ZWJ and such.
Perl makes no distinction between anything in the namespace when using
the \N{} form for string and regex escapes. That means when you use
"\N{...}" or /\N{...}/, you don't know which it is, nor can you.
(And yes, the bracketed character class issue is annoying and unsolved.)
However, the "functional" API does make a slight distinction.
-- charnames::vianame() takes a name or alias (as a string) and returns a single
integer code point.
-- charnames::string_vianame() takes a string name, alias, *or* sequence,
and gives back a string.
-- charnames::viacode() takes an integer can gives back the official alias
if there is one, and the official name if there is not.
Consider
That was an error, and there is an official alias fixing it:
(That's FHTORA vs FTHORA.)
You may use either as the name, and if you reverse the code
point to name, you get the replacement alias.
% perl -mcharnames -wle 'printf "%04X\n", charnames::vianame("BYZANTINE MUSICAL SYMBOL FHTORA SKLIRON CHROMA VASIS")'
1D0C5
% perl -mcharnames -wle 'printf "%04X\n", charnames::vianame("BYZANTINE MUSICAL SYMBOL FTHORA SKLIRON CHROMA VASIS")'
1D0C5
% perl -mcharnames -wle 'print charnames::viacode(charnames::vianame("BYZANTINE MUSICAL SYMBOL FHTORA SKLIRON CHROMA VASIS"))'
BYZANTINE MUSICAL SYMBOL FTHORA SKLIRON CHROMA VASIS
So on round-tripping, I gave it the "wrong" one (the original) and it gave
me back the "right" one (the replacement).
Using the \N{} thing, it again doesn't matter:
% perl -mcharnames=:full -wle 'printf "%04X\n", ord "\N{BYZANTINE MUSICAL SYMBOL FHTORA SKLIRON CHROMA VASIS}"'
1D0C5
% perl -mcharnames=:full -wle 'printf "%04X\n", ord "\N{BYZANTINE MUSICAL SYMBOL FTHORA SKLIRON CHROMA VASIS}"'
1D0C5
The interesting thing is the named sequences. string_vianame() works just fine on those:
% perl -mcharnames -wle 'print length charnames::string_vianame("LATIN CAPITAL LETTER A WITH MACRON AND GRAVE")'
2
% perl -mcharnames -wle 'printf "U+%v04X\n", charnames::string_vianame("LATIN CAPITAL LETTER A WITH MACRON AND GRAVE")'
U+0100.0300
And that works fine with \N{} as well (provided you don't try charclasses):
% perl -mcharnames=:full -wle 'print "\N{LATIN CAPITAL LETTER A WITH MACRON AND GRAVE}"'
Ā̀
% perl -mcharnames=:full -wle 'print "\N{LATIN CAPITAL LETTER A WITH MACRON AND GRAVE}"' | uniquote -v
\N{LATIN CAPITAL LETTER A WITH MACRON}\N{COMBINING GRAVE ACCENT}
% perl -mcharnames=:full -wle 'print length "\N{LATIN CAPITAL LETTER A WITH MACRON AND GRAVE}"'
2
% perl -mcharnames=:full -wle 'printf "U+%v04X\n", "\N{LATIN CAPITAL LETTER A WITH MACRON AND GRAVE}"'
U+0100.0300
It's kinda sad that for \N{} and sequneces you can't just "do the right
thing" with strings and say that charclass stuff just isn't supported.
But my guess is that this simply won't work because you don't have
first class regexes. If you pass both of these to the regex engine,
they should behave the same (and would, assuming the regex compiler
knows about \N{} escapes):
However, that falls part if you do
Because the compiler will do the substitution early on the first
one but not the second. This seems a problem, eh? So I guess
you can't do it at all? Or could you document it? I think there
is no good solution here. Perl can and does actually do something
quite reasonable in the noncharclass case, but that is because we
know that we are compiling a regex in virtually all scenarios.
So you can do:
And it is just fine. The issue is that there are ways for you to get
yoruself into trouble if you do string-string stuff:
That works, but only accidentally, because of course U+0100.0300 contains
nothing but either U+0100 or U+0300.
This is not a solved problem.
I hope this helps.
--tom
19 remaining items
[-]\N{...} neglects formal aliases and named sequences from Unicode charnames namespace[/-][+]\N{...} neglects formal aliases and named sequences from Unicode charnames namespace[/+]ezio-melotti commentedon Oct 9, 2011
Here is a new patch that stores the names of aliases and named sequences in the Private Use Area.
To summarize a bit, this is what we want:
| 6.0.0 | 3.2.0 |
--------+-------+-------+
\N{...} | A | - |
.name | - | - |
.lookup | A,NS | - |
I.e., \N{...} should only support aliases, unicodedata.lookup should support aliases and named sequences, unicodedata.name doesn't support either, and when 3.2.0 is used nothing is supported.
The function calls involved for these 3 functions are:
\N{...} and .lookup:
_getcode
_cmpname
_getucname
_check_alias
.name:
_getucname
My patch adds an extra arg to _getcode and _getucname (I hope that's fine -- or are they public?).
_getcode is called by \N{...} and .lookup; both support aliases, so _getcode now resolves aliases by default. Since only .lookup wants named sequences, _getcode now accepts an extra 'with_named_seq' arg and looks up named sequences only when its value is 1. .lookup passes 1, gets the codepoint, and converts it to a sequence. \N{...} passes 0 and doesn't get named sequences.
_getucname is called by .name and indirectly (through _cmpname) by .lookup and \N{...}. Since _getcode takes care of deciding who gets aliases and sequences, _getucname now accepts an extra 'with_alias_and_seq' arg and looks up aliases and named sequences only when its value is 1. _cmpname passes 1, gets aliases and named sequences and then lets _getcode decide what to do with them. .name passes 0 and doesn't get aliases and named sequences.
All this happens on 6.0.0 only, when self != NULL (i.e. we are using 3.2.0) named sequences and aliases are ignored.
The patch doesn't include the changes to unicodename_db.h -- run makeunicodedata.py to get them.
I also added more tests to make sure that the names added in the PUA don't leak, and that ucd_3_2_0 is not affected.
tchrist commentedon Oct 9, 2011
Ezio Melotti <[email protected]> wrote
on Sun, 09 Oct 2011 13:21:00 -0000:
Looks good! Thanks!
--tom
[-]\N{...} neglects formal aliases and named sequences from Unicode charnames namespace[/-][+]\N{...} neglects formal aliases and named sequences from Unicode charnames namespace[/+]ezio-melotti commentedon Oct 11, 2011
(I had to re-upload the patch a couple of time to get the review button to work. Apparently if there are some conflicts rietveld fails to apply the patch, whereas hg is able to merge files without problems here. Sorry for the noise.)
loewis commentedon Oct 12, 2011
If you don't use git-style diffs, Rietveld will much better accommodate patches that don't apply to tip cleanly. Unfortunately, hg git-style diffs don't indicate the base revision, so Rietveld guesses that the base line is tip, and then fails if it doesn't apply exactly.
ezio-melotti commentedon Oct 20, 2011
If the latest patch is fine I'll commit it shortly.
tchrist commentedon Oct 20, 2011
Yes, it looks good. Thank you very much.
-tom
loewis commentedon Oct 21, 2011
LGTM
python-dev commentedon Oct 21, 2011
New changeset a985d733b3a3 by Ezio Melotti in branch 'default':
bpo-12753: Add support for Unicode name aliases and named sequences.
http://hg.python.org/cpython/rev/a985d733b3a3
ezio-melotti commentedon Oct 21, 2011
I committed the patch and the buildbots seem happy. Thanks for the report and the feedback!
Tom, about the problems you mentioned in msg144836, can you report it in a new issue or, if there are already issues about them, add a message there?
python-dev commentedon Oct 21, 2011
New changeset 329b96fe4472 by Ezio Melotti in branch 'default':
bpo-12753: fix compilation on Windows.
http://hg.python.org/cpython/rev/329b96fe4472
abalkin commentedon Jun 23, 2013
I believe that would be bpo-4610.