Repository navigation
Request for python casemapping functions to use full not simple casemaps per Unicode's recommendation #56945
Description
| BPO | 12736 |
|---|---|
| Nosy | @abalkin, @pitrou, @benjaminp, @ezio-melotti, @JimJJewett |
| Files |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = <Date 2012-01-11.23:23:51.338>
created_at = <Date 2011-08-11.21:39:44.387>
labels = ['interpreter-core', 'type-feature', 'expert-unicode']
title = "Request for python casemapping functions to use full not simple casemaps per Unicode's recommendation"
updated_at = <Date 2016-03-11.07:39:49.423>
user = 'https://bugs.python.org/tchrist'bugs.python.org fields:
activity = <Date 2016-03-11.07:39:49.423>
actor = 'benjamin.peterson'
assignee = 'none'
closed = True
closed_date = <Date 2012-01-11.23:23:51.338>
closer = 'benjamin.peterson'
components = ['Interpreter Core', 'Unicode']
creation = <Date 2011-08-11.21:39:44.387>
creator = 'tchrist'
dependencies = []
files = ['22883', '23051', '23055', '24171', '24190', '24199', '24204', '42121']
hgrepos = []
issue_num = 12736
keywords = ['patch']
message_count = 27.0
messages = ['141928', '143036', '143051', '143052', '143072', '143083', '143084', '143085', '143086', '143089', '143110', '143119', '143124', '143145', '143146', '143148', '150844', '150998', '151016', '151088', '151098', '151141', '151311', '151314', '261517', '261522', '261547']
nosy_count = 11.0
nosy_names = ['belopolsky', 'pitrou', 'benjamin.peterson', 'ezio.melotti', 'mrabarnett', 'Arfrever', 'Jean-Michel.Fauth', 'python-dev', 'tchrist', 'Jim.Jewett', '\xd0\x90\xd0\xbd\xd0\xb4\xd1\x80\xd0\xb5\xd0\xb9 \xd0\x91\xd0\xb0\xd0\xba\xd1\x81\xd0\xb0\xd0\xbb\xd1\x8f\xd1\x80']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = None
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue12736'
versions = ['Python 3.4', 'Python 3.5', 'Python 3.6']
Activity
tchrist commentedon Aug 11, 2011
Python's casemapping functions only use what Unicode calls simple casemaps. These are only appropriate for functions that operate on single characters alone, not for those that operate on strings. The reason for this is that you get much better results with full casemapping. Java, Ruby, and Perl all do full casemapping for their equivalent functions that do string mapping, and Python should, too.
I include a program that has a much of mappings and foldings both simple and full. Yes, it was machine-generated.
gvanrossum commentedon Aug 26, 2011
I presume this applies to builtin str methods like .lower(), right? I think it is a good thing to do for Python 3.3.
We'd need to define what should happen in edge cases, e.g. when (against all odds) a string happens to contain a lone surrogate or some other code point or sequence of code points that the Unicode standard considers illegal. I think it should not fail but just leave those code points alone.
Does this require us to import more data files from the Unicode standard? By itself that doesn't scare me.
Would this also affect .islower() and friends?
tchrist commentedon Aug 26, 2011
Guido van Rossum <[email protected]> wrote
on Fri, 26 Aug 2011 21:11:24 -0000:
Yes, the full casemaps are for upper, title, and lowercase. There is
also a full casefold and turkic case fold (which is full), but you
don't have a casefold function so I guess that doesn't matter.
Well, it's a funny thing. There are properties given for all
Unicode code points, even noncharacter code points. This
includes the casing properties, oddly enough.
From UnicodeData.txt, which has a few surrogate entries; notice
no casing is given:
And in SpecialCasing.txt, which does not have surrogates but does have
a default clause:
And in CaseFolding.txt, which also does not have surrogates but again does
have a default clause:
Taken all together, it follows that the surrogates have case{map,fold}s
back to themselves, since they have no case{map,fold}s listed.
It's ok to have arbitrary code points in memory, including surrogates and
the 66 noncharacters. It just isn't legal to have them in a UTF stream
for "open interchange", whatever that means.
One way or the other, yes, notably the SpecialCasing file for
casemapping and the CaseFolding file for casefolding (which you
should do anyway to fix re.I). But you can and should process the
new files into some tighter format optimized for your own lookups.
Oddly, Java doesn't provide for String methods that do full casing on
titlecase, even those they do do so on lowercase and uppercase. On
titlecase they only expose the simple casemaps via the Character class,
which are the ones from UnicodeData. They recognize that this is flaw,
but it was too late to fix it for JAva 7.
Well, it shouldn't, but .islower() and friends are already mistaken.
They seem to be checking for GC=Ll and such, but they need to be
checking the Unicode binary property Lowercase and such. Watch:
That's because the Roman numerals are GC=Nl but still have
case and change case. Similarly for the circled letters which
are GC=So but have case and change case. Plus there's U+0345,
the iota subscript, which is GC=Mn but has case and changes case.
I don't remember whether I've sent in my full test suite or not.
If I haven't yet, I should attach it to the bug report.
--tom
tchrist commentedon Aug 26, 2011
Here’s my casing test suite; I thought I sent it in but the mux file here isn’t the full thing.
It does several things, including letting you run it with regex vs re. It also checks for the islower, etc functions. It has both simple and full (and turkic) maps and folds in it, but is configured to only check the simple versions for now. The islower and isupper etc functions seem to be checking the wrong Unicode property.
Yes, it has my quaint Unixisms in it, because it needs to run with UTF-8 output, or you can't read what's going on.
tchrist commentedon Aug 27, 2011
Guido van Rossum <[email protected]> wrote
on Fri, 26 Aug 2011 21:11:24 -0000:
SHORT VERSION: (7 lines)
=======================================================================
MEDIUM VERSION: (87 lines)
I was initially confused with Python's islower() family because of the way
they are defined to operate on full strings. They don't check that
everything is lowercase even though they say they do.
< http://docs.python.org/py3k/library/stdtypes.html#sequence-types-str-bytes-bytearray-list-tuple-range
This is strange in several ways. Of lesser importance is that
strings can be considered lowercase even if they don't match
Another is that the result of calling str.lower() may not be .islower().
I'm not sure what these are particularly for, since I myself would just use
a regex to get finer-grained control. (I suppose that's because re doesn't
give access to the Unicode properties needed that this approach never
gained any traction in the Python community.)
However, the worst of this is that the documentation defines both cased
characters and lowercase characters *differently* from how Unicode does
defines those very same terms. This was quite confusing.
Unicode distinguishes Cased code points from Cased_Letter code points.
Python is using the Cased_Letter property but calling it Cased. Cased in
a proper superset of Cased_Letter. From the DerivedCoreProperties file in
the Unicode Character Database:
In the same way, the Lowercase and Uppercase properties are not the same as
the Lowercase_Letter and Uppercase_Letter properties. Rather, the former
are respectively proper supersets of the latter.
In all these, you almost always want the superset versions not the
restricted subset versions you are using. If it were in the regex engine,
the user could select either.
Java used to miss all these, too. But in 1.7, they updated their character
methods to use the properties that they'd all along said they were using:
< http://download.oracle.com/javase/7/docs/api/java/lang/Character.html#isLowerCase(char)
-> property Other_Lowercase as defined by the Unicode Standard.
(And yes, that's where Java uses "character" to mean "code unit"
not "code point", alas. No wonder people get confused)
I'm pretty sure that Python needs to either update its documentation to
match its code, update its code to match its documentation, or both. Java
chose to update the code to match the documentation, and this is the course
I would recommend if at all possible. If you say you are checking for
cased code points, then you should use the Unicode definition of cased code
points not your own, and if you say you are checking for lowercase code
points, then you should use the Unicode definition not your own. Both of
these require access to contributory properties from the UCD and not
just general categories alone.
--tom
=======================================================================
LONG VERSION: (222 lines)
Essential tools I use for inspecting Unicode code points and their
properties include
And over the course of the day, these get used a fair bit, too:
Here for example are (some of) the *non*-Letter code point that
are nonetheless considered lowercase or uppercase because
they have the Other_{Lower,Upper}case properties:
And here are (some of) the letters that are cased but which are
not Lu, Lt, or Ll (they're all Lm, in fact):
Perversely, here are some of the modifier letters which are *not* cased:
(Interesting how the commas sort as breath marks next to H.)
I cannot for the life of me figure out why Unicode deems these lowercase:
yet these *not* to be cased:
All I know is that the tables tell me.
Here's a fair assortment of cased and noncased, case-changing and
non-casing code points. The variation in binary properties is pretty wide.
Unicode also has a Case_Ignorable (CI) character property, which I haven't
thought much about but which might be useful.
I'm not sure if you should think about these when doing your isupper()
test; maybe you should. That way you wouldn't fail just because you had
a code point that was technically lowercase, like if someone used
"LEONARD MᶜCOY". That funny ᶜ wouldn't count as a spoiler then, so that
"Leonard MᶜCoy".upper().isupper() could be true, as the ᶜ wouldn't
change but wouldn't count, either. I haven't thought about this enough
though. I'm not used to full string-based isupper() functions, so my
instincts may be wrong here.
The only code point that is both CWCM and also CI is the notorious
Subscripts, superscripts, modifier letters, small capitals, and mathematical
letters *tend* to be cased code points that do not change when casemapped
or casefolded, although there are exceptions.
% uniprops 280 1a6
U+0280 <ʀ> \N{LATIN LETTER SMALL CAPITAL R}
\w \pL \p{LC} \p{L_} \p{L&} \p{Ll}
All Any Alnum Alpha Alphabetic Assigned InIPA_Extensions Cased Cased_Letter LC Changes_When_Casemapped CWCM Changes_When_Titlecased CWT Changes_When_Uppercased CWU Ll L Gr_Base Grapheme_Base Graph GrBase ID_Continue IDC ID_Start IDS IPA_Extensions Letter L_ Latin Latn Lowercase_Letter Lower Lowercase Print Word XID_Continue XIDC XID_Start XIDS X_POSIX_Alnum X_POSIX_Alpha X_POSIX_Graph X_POSIX_Lower
X_POSIX_Print X_POSIX_Word
U+01A6 <Ʀ> \N{LATIN LETTER YR}
\w \pL \p{LC} \p{L_} \p{L&} \p{Lu}
All Any Alnum Alpha Alphabetic Assigned InLatinExtendedB Cased Cased_Letter LC Changes_When_Casefolded CWCF Changes_When_Casemapped CWCM Changes_When_Lowercased CWL Changes_When_NFKC_Casefolded CWKCF Lu L Gr_Base Grapheme_Base Graph GrBase ID_Continue IDC ID_Start IDS Letter L_ Latin Latn Latin_Extended_B Uppercase_Letter Print Upper Uppercase Word XID_Continue XIDC XID_Start XIDS X_POSIX_Alnum X_POSIX_Alpha X_POSIX_Graph X_POSIX_Print X_POSIX_Upper X_POSIX_Word
That's right: the uppercase of LATIN LETTER SMALL CAPITAL R is LATIN LETTER
YR, and I don't know why. No other small capital -- which are all considered
lowercase -- changes when casemapped. Only this one alone.
Note that things like code points like U+00DF LATIN SMALL LETTER SHARP S
have these binary properties true because the normal/default sense of these
terms in Unicode is the full/string sense not the simple/character sense:
Those are true because the full uppercase map of "ß" is "SS"
and the full casefold of "ß" is "ss".
--tom
gvanrossum commentedon Aug 27, 2011
Thanks you very much. We should fix the behavior in 3.3 for sure. I'm
thinking that we may be able to backport the behavior fix to 2.7 and
3.2 as well, since it just makes the behavior generally "better" (and
for most folks it won't matter anyway).
I'm not sure where the somewhat odd rules for .islower() come from, I
think in part from the desire to have "".islower() be False but "a
b".islower() to be True. Intuitively, this means that .islower() means
both "there is at least one lower case character" and "there are no
upper case characters", but not "all characters are lowercase". I
forget what we do w.r.t. titlecase, but the intuitive meaning should
not change. Although personally I don't have much of an intuition for
what titlecase means (and why it's important), perhaps because I'm not
familiar with any language where there is a third case for some
letters.
tchrist commentedon Aug 27, 2011
Guido van Rossum <[email protected]> wrote
on Sat, 27 Aug 2011 16:15:33 -0000:
Neither am I. Even in "old-style" English with ae and oe, one wrote
ÆGYPT and ÆSIR all caps but Ægypt and Æsir in titlecase, not *Aegypt or
*Aesir. Similarly with ŒNOLOGY / Œnology / œnology, never *Oenology.
I believe that almost but not quite all the sticky situations with
Unicode casing involve compatibility characters for clean round-trips
with legacy encodings. Exceptions include the German sharp s (both of
them now) and the two Greek lowercase sigmas. Thank goodness we don't
use the long s in English anymore. What is it with s's, anyway? :)
Most of the titlecase letters are in Greek, with a few in Armenian.
I know no Armenian (their letters all look the same to me :), and the
folks I talked to about the Greek are skeptical. The German sharp s is
a red herring, because you can never have it as the first letter
(although it needn't be the last, as in Rußland). That's no more
possible than having the old legacy ff ligature appear at the beginning
of an English world.
In any event, there are only 129 total code points that are
"problematic" in terms of their case, where by problematic
I mean one or more of:
--- titlecase differs from uppercase
--- foldcase differs from lowercase
--- any of fold/lower/title/uppercase yields more than one code point
Of all these, it's the (now two!) sharp s's and the Turkic i that are the most annoying.
It's really quite a lot of trouble to go through for so few code points of so little
(perceived) use. But I suppose you never know what new ones they'll uncover, either.
Here are those 129 case-problematicals arranged in UCA order. Some of these
normilizations forms that decompose into graphemes with four code points (not shown).
There are a few other oddities, like the Kelvin sign and other "singletons", but these
are most of the trouble. They're all in the BMP; I guess we learned our lesson. :)
--tom
1: U+0345 ○ͅ COMBINING GREEK YPOGEGRAMMENI
fc=ι U+3B9 lc=○ͅ U+345 tc=Ι U+399 uc=Ι U+399
2: U+1E9A ẚ LATIN SMALL LETTER A WITH RIGHT HALF RING
fc=aʾ U+61.2BE lc=ẚ U+1E9A tc=Aʾ U+41.2BE uc=Aʾ U+41.2BE
3: U+01F3 dz LATIN SMALL LETTER DZ
fc=dz U+1F3 lc=dz U+1F3 tc=Dz U+1F2 uc=DZ U+1F1
4: U+01F2 Dz LATIN CAPITAL LETTER D WITH SMALL LETTER Z
fc=dz U+1F3 lc=dz U+1F3 tc=Dz U+1F2 uc=DZ U+1F1
5: U+01F1 DZ LATIN CAPITAL LETTER DZ
fc=dz U+1F3 lc=dz U+1F3 tc=Dz U+1F2 uc=DZ U+1F1
6: U+01C6 dž LATIN SMALL LETTER DZ WITH CARON
fc=dž U+1C6 lc=dž U+1C6 tc=Dž U+1C5 uc=DŽ U+1C4
7: U+01C5 Dž LATIN CAPITAL LETTER D WITH SMALL LETTER Z WITH CARON
fc=dž U+1C6 lc=dž U+1C6 tc=Dž U+1C5 uc=DŽ U+1C4
8: U+01C4 DŽ LATIN CAPITAL LETTER DZ WITH CARON
fc=dž U+1C6 lc=dž U+1C6 tc=Dž U+1C5 uc=DŽ U+1C4
9: U+FB00 ff LATIN SMALL LIGATURE FF
fc=ff U+66.66 lc=ff U+FB00 tc=Ff U+46.66 uc=FF U+46.46
10: U+FB03 ffi LATIN SMALL LIGATURE FFI
fc=ffi U+66.66.69 lc=ffi U+FB03 tc=Ffi U+46.66.69 uc=FFI U+46.46.49
11: U+FB04 ffl LATIN SMALL LIGATURE FFL
fc=ffl U+66.66.6C lc=ffl U+FB04 tc=Ffl U+46.66.6C uc=FFL U+46.46.4C
12: U+FB01 fi LATIN SMALL LIGATURE FI
fc=fi U+66.69 lc=fi U+FB01 tc=Fi U+46.69 uc=FI U+46.49
13: U+FB02 fl LATIN SMALL LIGATURE FL
fc=fl U+66.6C lc=fl U+FB02 tc=Fl U+46.6C uc=FL U+46.4C
14: U+1E96 ẖ LATIN SMALL LETTER H WITH LINE BELOW
fc=ẖ U+68.331 lc=ẖ U+1E96 tc=H̱ U+48.331 uc=H̱ U+48.331
15: U+0130 İ LATIN CAPITAL LETTER I WITH DOT ABOVE
fc=i̇ U+69.307 lc=i̇ U+69.307 tc=İ U+130 uc=İ U+130
16: U+01F0 ǰ LATIN SMALL LETTER J WITH CARON
fc=ǰ U+6A.30C lc=ǰ U+1F0 tc=J̌ U+4A.30C uc=J̌ U+4A.30C
17: U+01C9 lj LATIN SMALL LETTER LJ
fc=lj U+1C9 lc=lj U+1C9 tc=Lj U+1C8 uc=LJ U+1C7
18: U+01C8 Lj LATIN CAPITAL LETTER L WITH SMALL LETTER J
fc=lj U+1C9 lc=lj U+1C9 tc=Lj U+1C8 uc=LJ U+1C7
19: U+01C7 LJ LATIN CAPITAL LETTER LJ
fc=lj U+1C9 lc=lj U+1C9 tc=Lj U+1C8 uc=LJ U+1C7
20: U+01CC nj LATIN SMALL LETTER NJ
fc=nj U+1CC lc=nj U+1CC tc=Nj U+1CB uc=NJ U+1CA
21: U+01CB Nj LATIN CAPITAL LETTER N WITH SMALL LETTER J
fc=nj U+1CC lc=nj U+1CC tc=Nj U+1CB uc=NJ U+1CA
22: U+01CA NJ LATIN CAPITAL LETTER NJ
fc=nj U+1CC lc=nj U+1CC tc=Nj U+1CB uc=NJ U+1CA
23: U+017F ſ LATIN SMALL LETTER LONG S
fc=s U+73 lc=ſ U+17F tc=S U+53 uc=S U+53
24: U+1E9B ẛ LATIN SMALL LETTER LONG S WITH DOT ABOVE
fc=ṡ U+1E61 lc=ẛ U+1E9B tc=Ṡ U+1E60 uc=Ṡ U+1E60
25: U+00DF ß LATIN SMALL LETTER SHARP S
fc=ss U+73.73 lc=ß U+DF tc=Ss U+53.73 uc=SS U+53.53
26: U+1E9E ẞ LATIN CAPITAL LETTER SHARP S
fc=ss U+73.73 lc=ß U+DF tc=ẞ U+1E9E uc=ẞ U+1E9E
27: U+FB06 st LATIN SMALL LIGATURE ST
fc=st U+73.74 lc=st U+FB06 tc=St U+53.74 uc=ST U+53.54
28: U+FB05 ſt LATIN SMALL LIGATURE LONG S T
fc=st U+73.74 lc=ſt U+FB05 tc=St U+53.74 uc=ST U+53.54
29: U+1E97 ẗ LATIN SMALL LETTER T WITH DIAERESIS
fc=ẗ U+74.308 lc=ẗ U+1E97 tc=T̈ U+54.308 uc=T̈ U+54.308
30: U+1E98 ẘ LATIN SMALL LETTER W WITH RING ABOVE
fc=ẘ U+77.30A lc=ẘ U+1E98 tc=W̊ U+57.30A uc=W̊ U+57.30A
31: U+1E99 ẙ LATIN SMALL LETTER Y WITH RING ABOVE
fc=ẙ U+79.30A lc=ẙ U+1E99 tc=Y̊ U+59.30A uc=Y̊ U+59.30A
32: U+0149 ʼn LATIN SMALL LETTER N PRECEDED BY APOSTROPHE
fc=ʼn U+2BC.6E lc=ʼn U+149 tc=ʼN U+2BC.4E uc=ʼN U+2BC.4E
33: U+1F84 ᾄ GREEK SMALL LETTER ALPHA WITH PSILI AND OXIA AND YPOGEGRAMMENI
fc=ἄι U+1F04.3B9 lc=ᾄ U+1F84 tc=ᾌ U+1F8C uc=ἌΙ U+1F0C.399
34: U+1F8C ᾌ GREEK CAPITAL LETTER ALPHA WITH PSILI AND OXIA AND PROSGEGRAMMENI
fc=ἄι U+1F04.3B9 lc=ᾄ U+1F84 tc=ᾌ U+1F8C uc=ἌΙ U+1F0C.399
35: U+1F82 ᾂ GREEK SMALL LETTER ALPHA WITH PSILI AND VARIA AND YPOGEGRAMMENI
fc=ἂι U+1F02.3B9 lc=ᾂ U+1F82 tc=ᾊ U+1F8A uc=ἊΙ U+1F0A.399
36: U+1F8A ᾊ GREEK CAPITAL LETTER ALPHA WITH PSILI AND VARIA AND PROSGEGRAMMENI
fc=ἂι U+1F02.3B9 lc=ᾂ U+1F82 tc=ᾊ U+1F8A uc=ἊΙ U+1F0A.399
37: U+1F86 ᾆ GREEK SMALL LETTER ALPHA WITH PSILI AND PERISPOMENI AND YPOGEGRAMMENI
fc=ἆι U+1F06.3B9 lc=ᾆ U+1F86 tc=ᾎ U+1F8E uc=ἎΙ U+1F0E.399
38: U+1F8E ᾎ GREEK CAPITAL LETTER ALPHA WITH PSILI AND PERISPOMENI AND PROSGEGRAMMENI
fc=ἆι U+1F06.3B9 lc=ᾆ U+1F86 tc=ᾎ U+1F8E uc=ἎΙ U+1F0E.399
39: U+1F80 ᾀ GREEK SMALL LETTER ALPHA WITH PSILI AND YPOGEGRAMMENI
fc=ἀι U+1F00.3B9 lc=ᾀ U+1F80 tc=ᾈ U+1F88 uc=ἈΙ U+1F08.399
40: U+1F88 ᾈ GREEK CAPITAL LETTER ALPHA WITH PSILI AND PROSGEGRAMMENI
fc=ἀι U+1F00.3B9 lc=ᾀ U+1F80 tc=ᾈ U+1F88 uc=ἈΙ U+1F08.399
41: U+1F85 ᾅ GREEK SMALL LETTER ALPHA WITH DASIA AND OXIA AND YPOGEGRAMMENI
fc=ἅι U+1F05.3B9 lc=ᾅ U+1F85 tc=ᾍ U+1F8D uc=ἍΙ U+1F0D.399
42: U+1F8D ᾍ GREEK CAPITAL LETTER ALPHA WITH DASIA AND OXIA AND PROSGEGRAMMENI
fc=ἅι U+1F05.3B9 lc=ᾅ U+1F85 tc=ᾍ U+1F8D uc=ἍΙ U+1F0D.399
43: U+1F83 ᾃ GREEK SMALL LETTER ALPHA WITH DASIA AND VARIA AND YPOGEGRAMMENI
fc=ἃι U+1F03.3B9 lc=ᾃ U+1F83 tc=ᾋ U+1F8B uc=ἋΙ U+1F0B.399
44: U+1F8B ᾋ GREEK CAPITAL LETTER ALPHA WITH DASIA AND VARIA AND PROSGEGRAMMENI
fc=ἃι U+1F03.3B9 lc=ᾃ U+1F83 tc=ᾋ U+1F8B uc=ἋΙ U+1F0B.399
45: U+1F87 ᾇ GREEK SMALL LETTER ALPHA WITH DASIA AND PERISPOMENI AND YPOGEGRAMMENI
fc=ἇι U+1F07.3B9 lc=ᾇ U+1F87 tc=ᾏ U+1F8F uc=ἏΙ U+1F0F.399
46: U+1F8F ᾏ GREEK CAPITAL LETTER ALPHA WITH DASIA AND PERISPOMENI AND PROSGEGRAMMENI
fc=ἇι U+1F07.3B9 lc=ᾇ U+1F87 tc=ᾏ U+1F8F uc=ἏΙ U+1F0F.399
47: U+1F81 ᾁ GREEK SMALL LETTER ALPHA WITH DASIA AND YPOGEGRAMMENI
fc=ἁι U+1F01.3B9 lc=ᾁ U+1F81 tc=ᾉ U+1F89 uc=ἉΙ U+1F09.399
48: U+1F89 ᾉ GREEK CAPITAL LETTER ALPHA WITH DASIA AND PROSGEGRAMMENI
fc=ἁι U+1F01.3B9 lc=ᾁ U+1F81 tc=ᾉ U+1F89 uc=ἉΙ U+1F09.399
49: U+1FB4 ᾴ GREEK SMALL LETTER ALPHA WITH OXIA AND YPOGEGRAMMENI
fc=άι U+3AC.3B9 lc=ᾴ U+1FB4 tc=Άͅ U+386.345 uc=ΆΙ U+386.399
50: U+1FB2 ᾲ GREEK SMALL LETTER ALPHA WITH VARIA AND YPOGEGRAMMENI
fc=ὰι U+1F70.3B9 lc=ᾲ U+1FB2 tc=Ὰͅ U+1FBA.345 uc=ᾺΙ U+1FBA.399
51: U+1FB6 ᾶ GREEK SMALL LETTER ALPHA WITH PERISPOMENI
fc=ᾶ U+3B1.342 lc=ᾶ U+1FB6 tc=Α͂ U+391.342 uc=Α͂ U+391.342
52: U+1FB7 ᾷ GREEK SMALL LETTER ALPHA WITH PERISPOMENI AND YPOGEGRAMMENI
fc=ᾶι U+3B1.342.3B9 lc=ᾷ U+1FB7 tc=ᾼ͂ U+391.342.345 uc=Α͂Ι U+391.342.399
53: U+1FB3 ᾳ GREEK SMALL LETTER ALPHA WITH YPOGEGRAMMENI
fc=αι U+3B1.3B9 lc=ᾳ U+1FB3 tc=ᾼ U+1FBC uc=ΑΙ U+391.399
54: U+1FBC ᾼ GREEK CAPITAL LETTER ALPHA WITH PROSGEGRAMMENI
fc=αι U+3B1.3B9 lc=ᾳ U+1FB3 tc=ᾼ U+1FBC uc=ΑΙ U+391.399
55: U+03D0 ϐ GREEK BETA SYMBOL
fc=β U+3B2 lc=ϐ U+3D0 tc=Β U+392 uc=Β U+392
56: U+03F5 ϵ GREEK LUNATE EPSILON SYMBOL
fc=ε U+3B5 lc=ϵ U+3F5 tc=Ε U+395 uc=Ε U+395
57: U+1F94 ᾔ GREEK SMALL LETTER ETA WITH PSILI AND OXIA AND YPOGEGRAMMENI
fc=ἤι U+1F24.3B9 lc=ᾔ U+1F94 tc=ᾜ U+1F9C uc=ἬΙ U+1F2C.399
58: U+1F9C ᾜ GREEK CAPITAL LETTER ETA WITH PSILI AND OXIA AND PROSGEGRAMMENI
fc=ἤι U+1F24.3B9 lc=ᾔ U+1F94 tc=ᾜ U+1F9C uc=ἬΙ U+1F2C.399
59: U+1F92 ᾒ GREEK SMALL LETTER ETA WITH PSILI AND VARIA AND YPOGEGRAMMENI
fc=ἢι U+1F22.3B9 lc=ᾒ U+1F92 tc=ᾚ U+1F9A uc=ἪΙ U+1F2A.399
60: U+1F9A ᾚ GREEK CAPITAL LETTER ETA WITH PSILI AND VARIA AND PROSGEGRAMMENI
fc=ἢι U+1F22.3B9 lc=ᾒ U+1F92 tc=ᾚ U+1F9A uc=ἪΙ U+1F2A.399
61: U+1F96 ᾖ GREEK SMALL LETTER ETA WITH PSILI AND PERISPOMENI AND YPOGEGRAMMENI
fc=ἦι U+1F26.3B9 lc=ᾖ U+1F96 tc=ᾞ U+1F9E uc=ἮΙ U+1F2E.399
62: U+1F9E ᾞ GREEK CAPITAL LETTER ETA WITH PSILI AND PERISPOMENI AND PROSGEGRAMMENI
fc=ἦι U+1F26.3B9 lc=ᾖ U+1F96 tc=ᾞ U+1F9E uc=ἮΙ U+1F2E.399
63: U+1F90 ᾐ GREEK SMALL LETTER ETA WITH PSILI AND YPOGEGRAMMENI
fc=ἠι U+1F20.3B9 lc=ᾐ U+1F90 tc=ᾘ U+1F98 uc=ἨΙ U+1F28.399
64: U+1F98 ᾘ GREEK CAPITAL LETTER ETA WITH PSILI AND PROSGEGRAMMENI
fc=ἠι U+1F20.3B9 lc=ᾐ U+1F90 tc=ᾘ U+1F98 uc=ἨΙ U+1F28.399
65: U+1F95 ᾕ GREEK SMALL LETTER ETA WITH DASIA AND OXIA AND YPOGEGRAMMENI
fc=ἥι U+1F25.3B9 lc=ᾕ U+1F95 tc=ᾝ U+1F9D uc=ἭΙ U+1F2D.399
66: U+1F9D ᾝ GREEK CAPITAL LETTER ETA WITH DASIA AND OXIA AND PROSGEGRAMMENI
fc=ἥι U+1F25.3B9 lc=ᾕ U+1F95 tc=ᾝ U+1F9D uc=ἭΙ U+1F2D.399
67: U+1F93 ᾓ GREEK SMALL LETTER ETA WITH DASIA AND VARIA AND YPOGEGRAMMENI
fc=ἣι U+1F23.3B9 lc=ᾓ U+1F93 tc=ᾛ U+1F9B uc=ἫΙ U+1F2B.399
68: U+1F9B ᾛ GREEK CAPITAL LETTER ETA WITH DASIA AND VARIA AND PROSGEGRAMMENI
fc=ἣι U+1F23.3B9 lc=ᾓ U+1F93 tc=ᾛ U+1F9B uc=ἫΙ U+1F2B.399
69: U+1F97 ᾗ GREEK SMALL LETTER ETA WITH DASIA AND PERISPOMENI AND YPOGEGRAMMENI
fc=ἧι U+1F27.3B9 lc=ᾗ U+1F97 tc=ᾟ U+1F9F uc=ἯΙ U+1F2F.399
70: U+1F9F ᾟ GREEK CAPITAL LETTER ETA WITH DASIA AND PERISPOMENI AND PROSGEGRAMMENI
fc=ἧι U+1F27.3B9 lc=ᾗ U+1F97 tc=ᾟ U+1F9F uc=ἯΙ U+1F2F.399
71: U+1F91 ᾑ GREEK SMALL LETTER ETA WITH DASIA AND YPOGEGRAMMENI
fc=ἡι U+1F21.3B9 lc=ᾑ U+1F91 tc=ᾙ U+1F99 uc=ἩΙ U+1F29.399
72: U+1F99 ᾙ GREEK CAPITAL LETTER ETA WITH DASIA AND PROSGEGRAMMENI
fc=ἡι U+1F21.3B9 lc=ᾑ U+1F91 tc=ᾙ U+1F99 uc=ἩΙ U+1F29.399
73: U+1FC4 ῄ GREEK SMALL LETTER ETA WITH OXIA AND YPOGEGRAMMENI
fc=ήι U+3AE.3B9 lc=ῄ U+1FC4 tc=Ήͅ U+389.345 uc=ΉΙ U+389.399
74: U+1FC2 ῂ GREEK SMALL LETTER ETA WITH VARIA AND YPOGEGRAMMENI
fc=ὴι U+1F74.3B9 lc=ῂ U+1FC2 tc=Ὴͅ U+1FCA.345 uc=ῊΙ U+1FCA.399
75: U+1FC6 ῆ GREEK SMALL LETTER ETA WITH PERISPOMENI
fc=ῆ U+3B7.342 lc=ῆ U+1FC6 tc=Η͂ U+397.342 uc=Η͂ U+397.342
76: U+1FC7 ῇ GREEK SMALL LETTER ETA WITH PERISPOMENI AND YPOGEGRAMMENI
fc=ῆι U+3B7.342.3B9 lc=ῇ U+1FC7 tc=ῌ͂ U+397.342.345 uc=Η͂Ι U+397.342.399
77: U+1FC3 ῃ GREEK SMALL LETTER ETA WITH YPOGEGRAMMENI
fc=ηι U+3B7.3B9 lc=ῃ U+1FC3 tc=ῌ U+1FCC uc=ΗΙ U+397.399
78: U+1FCC ῌ GREEK CAPITAL LETTER ETA WITH PROSGEGRAMMENI
fc=ηι U+3B7.3B9 lc=ῃ U+1FC3 tc=ῌ U+1FCC uc=ΗΙ U+397.399
79: U+03D1 ϑ GREEK THETA SYMBOL
fc=θ U+3B8 lc=ϑ U+3D1 tc=Θ U+398 uc=Θ U+398
80: U+1FBE ι GREEK PROSGEGRAMMENI
fc=ι U+3B9 lc=ι U+1FBE tc=Ι U+399 uc=Ι U+399
81: U+1FD6 ῖ GREEK SMALL LETTER IOTA WITH PERISPOMENI
fc=ῖ U+3B9.342 lc=ῖ U+1FD6 tc=Ι͂ U+399.342 uc=Ι͂ U+399.342
82: U+0390 ΐ GREEK SMALL LETTER IOTA WITH DIALYTIKA AND TONOS
fc=ΐ U+3B9.308.301 lc=ΐ U+390 tc=Ϊ́ U+399.308.301 uc=Ϊ́ U+399.308.301
83: U+1FD3 ΐ GREEK SMALL LETTER IOTA WITH DIALYTIKA AND OXIA
fc=ΐ U+3B9.308.301 lc=ΐ U+1FD3 tc=Ϊ́ U+399.308.301 uc=Ϊ́ U+399.308.301
84: U+1FD2 ῒ GREEK SMALL LETTER IOTA WITH DIALYTIKA AND VARIA
fc=ῒ U+3B9.308.300 lc=ῒ U+1FD2 tc=Ϊ̀ U+399.308.300 uc=Ϊ̀ U+399.308.300
85: U+1FD7 ῗ GREEK SMALL LETTER IOTA WITH DIALYTIKA AND PERISPOMENI
fc=ῗ U+3B9.308.342 lc=ῗ U+1FD7 tc=Ϊ͂ U+399.308.342 uc=Ϊ͂ U+399.308.342
86: U+03F0 ϰ GREEK KAPPA SYMBOL
fc=κ U+3BA lc=ϰ U+3F0 tc=Κ U+39A uc=Κ U+39A
87: U+00B5 µ MICRO SIGN
fc=μ U+3BC lc=µ U+B5 tc=Μ U+39C uc=Μ U+39C
88: U+03D6 ϖ GREEK PI SYMBOL
fc=π U+3C0 lc=ϖ U+3D6 tc=Π U+3A0 uc=Π U+3A0
89: U+03F1 ϱ GREEK RHO SYMBOL
fc=ρ U+3C1 lc=ϱ U+3F1 tc=Ρ U+3A1 uc=Ρ U+3A1
90: U+1FE4 ῤ GREEK SMALL LETTER RHO WITH PSILI
fc=ῤ U+3C1.313 lc=ῤ U+1FE4 tc=Ρ̓ U+3A1.313 uc=Ρ̓ U+3A1.313
91: U+03C2 ς GREEK SMALL LETTER FINAL SIGMA
fc=σ U+3C3 lc=ς U+3C2 tc=Σ U+3A3 uc=Σ U+3A3
92: U+1F50 ὐ GREEK SMALL LETTER UPSILON WITH PSILI
fc=ὐ U+3C5.313 lc=ὐ U+1F50 tc=Υ̓ U+3A5.313 uc=Υ̓ U+3A5.313
93: U+1F54 ὔ GREEK SMALL LETTER UPSILON WITH PSILI AND OXIA
fc=ὔ U+3C5.313.301 lc=ὔ U+1F54 tc=Υ̓́ U+3A5.313.301 uc=Υ̓́ U+3A5.313.301
94: U+1F52 ὒ GREEK SMALL LETTER UPSILON WITH PSILI AND VARIA
fc=ὒ U+3C5.313.300 lc=ὒ U+1F52 tc=Υ̓̀ U+3A5.313.300 uc=Υ̓̀ U+3A5.313.300
95: U+1F56 ὖ GREEK SMALL LETTER UPSILON WITH PSILI AND PERISPOMENI
fc=ὖ U+3C5.313.342 lc=ὖ U+1F56 tc=Υ̓͂ U+3A5.313.342 uc=Υ̓͂ U+3A5.313.342
96: U+1FE6 ῦ GREEK SMALL LETTER UPSILON WITH PERISPOMENI
fc=ῦ U+3C5.342 lc=ῦ U+1FE6 tc=Υ͂ U+3A5.342 uc=Υ͂ U+3A5.342
97: U+03B0 ΰ GREEK SMALL LETTER UPSILON WITH DIALYTIKA AND TONOS
fc=ΰ U+3C5.308.301 lc=ΰ U+3B0 tc=Ϋ́ U+3A5.308.301 uc=Ϋ́ U+3A5.308.301
98: U+1FE3 ΰ GREEK SMALL LETTER UPSILON WITH DIALYTIKA AND OXIA
fc=ΰ U+3C5.308.301 lc=ΰ U+1FE3 tc=Ϋ́ U+3A5.308.301 uc=Ϋ́ U+3A5.308.301
99: U+1FE2 ῢ GREEK SMALL LETTER UPSILON WITH DIALYTIKA AND VARIA
fc=ῢ U+3C5.308.300 lc=ῢ U+1FE2 tc=Ϋ̀ U+3A5.308.300 uc=Ϋ̀ U+3A5.308.300
100: U+1FE7 ῧ GREEK SMALL LETTER UPSILON WITH DIALYTIKA AND PERISPOMENI
fc=ῧ U+3C5.308.342 lc=ῧ U+1FE7 tc=Ϋ͂ U+3A5.308.342 uc=Ϋ͂ U+3A5.308.342
101: U+03D5 ϕ GREEK PHI SYMBOL
fc=φ U+3C6 lc=ϕ U+3D5 tc=Φ U+3A6 uc=Φ U+3A6
102: U+1FA4 ᾤ GREEK SMALL LETTER OMEGA WITH PSILI AND OXIA AND YPOGEGRAMMENI
fc=ὤι U+1F64.3B9 lc=ᾤ U+1FA4 tc=ᾬ U+1FAC uc=ὬΙ U+1F6C.399
103: U+1FAC ᾬ GREEK CAPITAL LETTER OMEGA WITH PSILI AND OXIA AND PROSGEGRAMMENI
fc=ὤι U+1F64.3B9 lc=ᾤ U+1FA4 tc=ᾬ U+1FAC uc=ὬΙ U+1F6C.399
104: U+1FA2 ᾢ GREEK SMALL LETTER OMEGA WITH PSILI AND VARIA AND YPOGEGRAMMENI
fc=ὢι U+1F62.3B9 lc=ᾢ U+1FA2 tc=ᾪ U+1FAA uc=ὪΙ U+1F6A.399
105: U+1FAA ᾪ GREEK CAPITAL LETTER OMEGA WITH PSILI AND VARIA AND PROSGEGRAMMENI
fc=ὢι U+1F62.3B9 lc=ᾢ U+1FA2 tc=ᾪ U+1FAA uc=ὪΙ U+1F6A.399
106: U+1FA6 ᾦ GREEK SMALL LETTER OMEGA WITH PSILI AND PERISPOMENI AND YPOGEGRAMMENI
fc=ὦι U+1F66.3B9 lc=ᾦ U+1FA6 tc=ᾮ U+1FAE uc=ὮΙ U+1F6E.399
107: U+1FAE ᾮ GREEK CAPITAL LETTER OMEGA WITH PSILI AND PERISPOMENI AND PROSGEGRAMMENI
fc=ὦι U+1F66.3B9 lc=ᾦ U+1FA6 tc=ᾮ U+1FAE uc=ὮΙ U+1F6E.399
108: U+1FA0 ᾠ GREEK SMALL LETTER OMEGA WITH PSILI AND YPOGEGRAMMENI
fc=ὠι U+1F60.3B9 lc=ᾠ U+1FA0 tc=ᾨ U+1FA8 uc=ὨΙ U+1F68.399
109: U+1FA8 ᾨ GREEK CAPITAL LETTER OMEGA WITH PSILI AND PROSGEGRAMMENI
fc=ὠι U+1F60.3B9 lc=ᾠ U+1FA0 tc=ᾨ U+1FA8 uc=ὨΙ U+1F68.399
110: U+1FA5 ᾥ GREEK SMALL LETTER OMEGA WITH DASIA AND OXIA AND YPOGEGRAMMENI
fc=ὥι U+1F65.3B9 lc=ᾥ U+1FA5 tc=ᾭ U+1FAD uc=ὭΙ U+1F6D.399
111: U+1FAD ᾭ GREEK CAPITAL LETTER OMEGA WITH DASIA AND OXIA AND PROSGEGRAMMENI
fc=ὥι U+1F65.3B9 lc=ᾥ U+1FA5 tc=ᾭ U+1FAD uc=ὭΙ U+1F6D.399
112: U+1FA3 ᾣ GREEK SMALL LETTER OMEGA WITH DASIA AND VARIA AND YPOGEGRAMMENI
fc=ὣι U+1F63.3B9 lc=ᾣ U+1FA3 tc=ᾫ U+1FAB uc=ὫΙ U+1F6B.399
113: U+1FAB ᾫ GREEK CAPITAL LETTER OMEGA WITH DASIA AND VARIA AND PROSGEGRAMMENI
fc=ὣι U+1F63.3B9 lc=ᾣ U+1FA3 tc=ᾫ U+1FAB uc=ὫΙ U+1F6B.399
114: U+1FA7 ᾧ GREEK SMALL LETTER OMEGA WITH DASIA AND PERISPOMENI AND YPOGEGRAMMENI
fc=ὧι U+1F67.3B9 lc=ᾧ U+1FA7 tc=ᾯ U+1FAF uc=ὯΙ U+1F6F.399
115: U+1FAF ᾯ GREEK CAPITAL LETTER OMEGA WITH DASIA AND PERISPOMENI AND PROSGEGRAMMENI
fc=ὧι U+1F67.3B9 lc=ᾧ U+1FA7 tc=ᾯ U+1FAF uc=ὯΙ U+1F6F.399
116: U+1FA1 ᾡ GREEK SMALL LETTER OMEGA WITH DASIA AND YPOGEGRAMMENI
fc=ὡι U+1F61.3B9 lc=ᾡ U+1FA1 tc=ᾩ U+1FA9 uc=ὩΙ U+1F69.399
117: U+1FA9 ᾩ GREEK CAPITAL LETTER OMEGA WITH DASIA AND PROSGEGRAMMENI
fc=ὡι U+1F61.3B9 lc=ᾡ U+1FA1 tc=ᾩ U+1FA9 uc=ὩΙ U+1F69.399
118: U+1FF4 ῴ GREEK SMALL LETTER OMEGA WITH OXIA AND YPOGEGRAMMENI
fc=ώι U+3CE.3B9 lc=ῴ U+1FF4 tc=Ώͅ U+38F.345 uc=ΏΙ U+38F.399
119: U+1FF2 ῲ GREEK SMALL LETTER OMEGA WITH VARIA AND YPOGEGRAMMENI
fc=ὼι U+1F7C.3B9 lc=ῲ U+1FF2 tc=Ὼͅ U+1FFA.345 uc=ῺΙ U+1FFA.399
120: U+1FF6 ῶ GREEK SMALL LETTER OMEGA WITH PERISPOMENI
fc=ῶ U+3C9.342 lc=ῶ U+1FF6 tc=Ω͂ U+3A9.342 uc=Ω͂ U+3A9.342
121: U+1FF7 ῷ GREEK SMALL LETTER OMEGA WITH PERISPOMENI AND YPOGEGRAMMENI
fc=ῶι U+3C9.342.3B9 lc=ῷ U+1FF7 tc=ῼ͂ U+3A9.342.345 uc=Ω͂Ι U+3A9.342.399
122: U+1FF3 ῳ GREEK SMALL LETTER OMEGA WITH YPOGEGRAMMENI
fc=ωι U+3C9.3B9 lc=ῳ U+1FF3 tc=ῼ U+1FFC uc=ΩΙ U+3A9.399
123: U+1FFC ῼ GREEK CAPITAL LETTER OMEGA WITH PROSGEGRAMMENI
fc=ωι U+3C9.3B9 lc=ῳ U+1FF3 tc=ῼ U+1FFC uc=ΩΙ U+3A9.399
124: U+0587 և ARMENIAN SMALL LIGATURE ECH YIWN
fc=եւ U+565.582 lc=և U+587 tc=Եւ U+535.582 uc=ԵՒ U+535.552
125: U+FB14 ﬔ ARMENIAN SMALL LIGATURE MEN ECH
fc=մե U+574.565 lc=ﬔ U+FB14 tc=Մե U+544.565 uc=ՄԵ U+544.535
126: U+FB15 ﬕ ARMENIAN SMALL LIGATURE MEN INI
fc=մի U+574.56B lc=ﬕ U+FB15 tc=Մի U+544.56B uc=ՄԻ U+544.53B
127: U+FB17 ﬗ ARMENIAN SMALL LIGATURE MEN XEH
fc=մխ U+574.56D lc=ﬗ U+FB17 tc=Մխ U+544.56D uc=ՄԽ U+544.53D
128: U+FB13 ﬓ ARMENIAN SMALL LIGATURE MEN NOW
fc=մն U+574.576 lc=ﬓ U+FB13 tc=Մն U+544.576 uc=ՄՆ U+544.546
129: U+FB16 ﬖ ARMENIAN SMALL LIGATURE VEW NOW
fc=վն U+57E.576 lc=ﬖ U+FB16 tc=Վն U+54E.576 uc=ՎՆ U+54E.546
mrabarnett commentedon Aug 27, 2011
There are some oddities in Unicode case-folding.
Under full case-folding, both "\N{LATIN CAPITAL LETTER SHARP S}" and "\N{LATIN SMALL LETTER SHARP S}" fold to "ss", which means that those codepoints match each other.
However, under simple case-folding, they fold to themselves, which means that those codepoints _don't_ match each other.
pitrou commentedon Aug 27, 2011
Trying to disprove you a bit:
http://ecx.images-amazon.com/images/I/51G6CH9XFFL.\_SL500_AA300_.jpg
http://ecx.images-amazon.com/images/I/51k7TmosPdL.\_SL500_AA300_.jpg
http://ecx.images-amazon.com/images/I/518UzMeLFCL.\_SL500_AA300_.jpg
but classical typographies seem to write either the uppercase Œ or the lowercase œ.
That said, I wonder why Unicode even includes ligatures like ff. Sounds like mission creep to me (and horrible annoyances for people like us).
ezio-melotti commentedon Aug 28, 2011
FTR, with the latest Python 3.2/3.3 (narrow) I get:
Total failures: 58 / 500 ( 12%)
Total successes: 442 / 500 ( 88%)
and with the latest Python 3.2/3.3 (wide) I get:
Total failures: 52 / 500 ( 10%)
Total successes: 448 / 500 ( 90%)
4 remaining items
pitrou commentedon Aug 29, 2011
It's not only "typographically speaking", it's really a spelling error,
even in hand-written text :-)
tchrist commentedon Aug 29, 2011
Antoine Pitrou <[email protected]> wrote
on Mon, 29 Aug 2011 13:21:06 -0000:
Sure, and so too is omitting an accent mark or diaeresis. But—alas!—you’ll
never convince most monoglot anglophones of that, the ones who keep wanting to
strip them from résumé, façade, châteaux, crème brûlée, fête, tête-à-tête,
à la française, or naïveté, not to mention José, jalapeño, the erstwhile
American Secretary of State Federico Peña, or nearby Cañon City, Colorado,
where I have family. I think œnonlogy has survived solely on its rarity,
and the Encyclopædia Britannica is that way because the ligat(ur)ed letter
is in their actual trademark.
Cell phone users sending text messages have long suffered the grievous
injuries to their language(s) that naked ASCII imparts, but this is
nothing like the crossdressing nightmare called Greeklish, also variously
known as Grenglish, Latinoellinika/Λατινοελληνικά, or ASCII Greek.
I especially enjoy George Baloglou’s "Byzantine" Grenglish, wherein:
Οι Ατρείδες => Oi Atpeides instead of the Atreïdes
Terrible though the depredations upon the French language that may
have been committed by ASCII, surely these go even further. :)
--tom
Μῆνιν ἄειδε, θεὰ, Πηληϊάδεω Ἀχιλῆος Mhnin aeide, 8ea, 5hlhiadeo Axilhos
οὐλομένην, ἣ μυρί’ Ἀχαιοῖς ἄλγε’ ἔθηκε, oulomenhn, 'h mupi’ Axaiois alge’ e8hke,
πολλὰς δ’ ἰφθίμους ψυχὰς Ἄϊδι προῒαψεν nollas d’ iph8imous yuxas Aidi npoiayen
ἡρώων, αὐτοὺς δὲ ἑλώρια τεῦχε κύνεσσιν 'hpoon, autous de elopia teuxe kuneccin
οἰωνοῖσί τε πᾶσι· Διὸς δ’ ἐτελείετο βουλή· oionoici te naci· Dios d’ eteleieto boulh·
ἐξ οὗ δὴ τὰ πρῶτα διαστήτην ἐρίσαντε eks o'u dh ta npota diacththn epicante
Ἀτρεΐδης τε ἄναξ ἀνδρῶν καὶ δῖος Ἀχιλλεύς. Atpeidhs te anaks andpon kai dios Axilleus.
benjaminp commentedon Jan 8, 2012
Here is a patch. I only dealt with case mappings and not titlecase. Doing titlecase properly requires word segmentation, which I think should be another patch/issue. This patch fixes swapcase(), capitalize(), upper(), and lower(). It does not include the changes to Objects/unicodetype_db.h because those are huge. Regenerate the database if you want to test it. Please review.
benjaminp commentedon Jan 10, 2012
New patch. I implemented it the way Antoine desired. It seems rather inefficient to be copying around so much data...
benjaminp commentedon Jan 10, 2012
__ap__'s implementation method is about 2x faster than mine.
benjaminp commentedon Jan 11, 2012
New patch with title casing mappings added.
python-dev commentedon Jan 11, 2012
New changeset f7e05d205a52 by Benjamin Peterson in branch 'default':
use full unicode mappings for upper/lower/title case (bpo-12736)
http://hg.python.org/cpython/rev/f7e05d205a52
jimjjewett commentedon Jan 12, 2012
The currently applied patch ( http://hg.python.org/cpython/rev/f7e05d205a52 ) left some dead code in unicodeobject.c
function fixup ( http://hg.python.org/cpython/file/f7e05d205a52/Objects/unicodeobject.c#l9386 ) has a shortcut for when the fixer doesn't make any actual changes. The removed fixers (like fixupper ) returned 0 rather than maxchar to indicate that. The only remaining fixer, fix_decimal_and_space_to_ascii (line 8839), does not. (I think fix_decimal_and_space_to_ascii *should* add a touched flag, but until it does, the shortcut dedup code is dead.)
Also, around line 10502, there is an #if 0 section with code that relied on one of the removed fixers; is it time to remove that section?
jimjjewett commentedon Jan 16, 2012
Why was the delta-processing removed from the casing functions?
As best I can tell, the whole point of going through multiple levels of indirection (courtesy splitbins) is to maximize compression and minimize the amount of cache that unicode might occupy.
By using deltas, only one record is needed for each combination of (upper - lower, upper - title), which is generally only one or two combinations per script.
Without deltas, nearly every cased letter needs its own record, and the index tables also get bigger. (It seems to be about 2.6 times as large, but cache effects may be worse, since letters from the same script will no longer be in the same record or the same index chain.)
If it is a concern about not enough room for flags, then the decimal/digit chars could be combined. They are always the same, unless the number isn't decimal (in which case the flag is enough).
python-dev commentedon Jan 16, 2012
New changeset 03ea95e3b497 by Benjamin Peterson in branch 'default':
delta encoding of upper/lower/title makes a glorious return (bpo-12736)
http://hg.python.org/cpython/rev/03ea95e3b497
ghost commentedon Mar 10, 2016
A same problem with the unicode case mapping is still present in the Python 3.4.3. You can repeat the bug with this code, for instance:
'ΰ'.upper().lower() == 'ΰ'
The case swapping is strangelly leads to character replacement:
b'\xce\xb0' → b'\xcf\x85\xcc\x88\xcc\x81'
ghost commentedon Mar 10, 2016
Interestingly, the bug is still reproducible in version 3.5.1, but fixed in 2.7.6.
benjaminp commentedon Mar 11, 2016
The full case mappings do not preserve normalization form.