Include more chars in the decimal codec

@malemburg

BPO	6632
Nosy	@malemburg, @loewis, @rhettinger, @terryjreedy, @mdickinson, @abalkin, @pitrou, @ericvsmith, @ezio-melotti, @skrah, @ambv
Superseder	bpo-10581: Review and document string format accepted in numeric data type constructors

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2013-06-12.05:01:33.994>
created_at = <Date 2009-08-03.16:34:44.245>
labels = ['interpreter-core', 'type-feature', 'expert-unicode']
title = 'Include more chars in the decimal codec'
updated_at = <Date 2013-06-12.05:01:33.988>
user = 'https://github.com/ezio-melotti'

bugs.python.org fields:

activity = <Date 2013-06-12.05:01:33.988>
actor = 'belopolsky'
assignee = 'none'
closed = True
closed_date = <Date 2013-06-12.05:01:33.994>
closer = 'belopolsky'
components = ['Interpreter Core', 'Unicode']
creation = <Date 2009-08-03.16:34:44.245>
creator = 'ezio.melotti'
dependencies = []
files = []
hgrepos = []
issue_num = 6632
keywords = []
message_count = 21.0
messages = ['91225', '91232', '91236', '91304', '91358', '91360', '91362', '91395', '92984', '93004', '93056', '93059', '113440', '113619', '123584', '124923', '185449', '190877', '190881', '190948', '191010']
nosy_count = 13.0
nosy_names = ['lemburg', 'loewis', 'rhettinger', 'terry.reedy', 'mark.dickinson', 'belopolsky', 'ggenellina', 'pitrou', 'eric.smith', 'ezio.melotti', 'skrah', 'lukasz.langa', 'tchrist']
pr_nums = []
priority = 'normal'
resolution = 'duplicate'
stage = None
status = 'closed'
superseder = '10581'
type = 'enhancement'
url = 'https://bugs.python.org/issue6632'
versions = ['Python 3.3']

The decimal codec only handles characters in the Nd (Number, decimal)
Unicode category and whitespaces a. It is used by int(), float(),
complex() and indirectly by Decimal(), Fraction() and possibly others.
This works well only for plain digits (e.g. int(u'１２３')) but it
doesn't work for all the other characters used to represent numbers, like:

plus or minus sign, e.g. int(u'＋１２３') or int(u'－１２３')
decimal point, e.g. float(u'１．２３')
2.1 some languages/alphabets use other chars (e.g. a comma or other
symbols) instead of the decimal point.
exponential notation, e.g. float(u'１ｅ５')
the 'j' in complex numbers, e.g. complex(u'３ｊ')
the 'x' and 'p' in hexadecimal floats, e.g.
float.fromhex(u'０ｘ１．７ｐ３')
5.1 hex floats also uses hexadecimal digits, see 6.3
digits > 9 for numbers with a base > 10, e.g. int(u'７Ｆ', 16)
6.1 not all the alphabets have the equivalent of the letters a-z
6.2 afaik there are no standards that specify how to deal with
digits >9
6.3 in the Unicode FAQ [b] there's a link to a table [c] that says
"Code points not listed in this file are not hexadecimal
digits." This is not a standard though, and even if in the
UCD [d] there's a file [e] where the numbers with the Hex_Digit
property are defined, it doesn't say that *only* these numbers
are valid hex digits. Also it doesn't say anything about
different bases.
Python currently accepts int(u'１０', 16), int(u'७', 16)
(U+096D - DEVANAGARI DIGIT SEVEN) and even int(u'７F', 16)
(with a normal F it works, with a fullwidth Ｆ it fails).
6.4 UTS Rename Doc/README.txt to Doc/README.rst #18 [f] includes in the property 'xdigit' [g] (hexadecimal
digit) all the chars defined in [c] and also all the chars with
a Nd category. This also is not a standard, and it doesn't
give indications about the valid hex digits and how int()
should behave.
6.5 if possible re and int() should agree. Any string that matches
/^[[:xdigit:]]+$/ should work fine with int(s, 16) and vice
versa. See also bpo-6561 [h] and bpo-2636 [i].
possibly others

For all the chars listed in the points 1-5 there's no way, AFAIK, to
know their equivalents in other alphabets (if they exist at all) and
since (apparently) there's no standard that specifies how to handle
them, they should be kept out.
This will also avoid a number of problems, e.g. 2.1.

The fullwidth forms are an exception though: they seem to be the only
set of characters with a direct equivalent for all these chars, and they
are also the only non-ascii chars included in the list of chars with the
Unicode Hex_Digits property.

Including all the necessary chars from this range in the decimal codec
seems to me the best thing to do. The chars listed in the points 1-5
should all be implemented and they should work everywhere. The regex
used by Decimal/Fraction should be updated as well, since the decimal
codec is not accessible from Python (maybe it should be accessible, but
this is another issue).

Point 6 is a slightly different issue, even if it can be partially
solved if the fullwidth forms will be included. One of the possible
options is to limit the valid chars used by int() with bases > 10 only
to the characters listed in [c], but this won't be backward-compatible
with existing code and forward-compatible with [[:xdigit:]].
OTOH if we keep the current behavior it will be possible to express the
digits from 0 to 9 using several alphabets, but all the digits > 9 will
be limited to [a-fA-F] (and possibly [ａ－ｆＡ－Ｆ]).
For example, '7F' in the devanagari alphabet will result in a mix of
devanagari numbers and ascii letters, i.e. int(u'७F', 16) (this already
works in Python).

under 'Decimal Encoder'
[b]: http://unicode.org/faq/casemap_charprop.html#13
[c]: http://unicode.org/faq/hex-digit-values.txt - [0-9a-fA-
F０－９ａ－ｆＡ－Ｆ]
[d]: http://unicode.org/Public/UNIDATA/UCD.html#UCD_Files - PropList.txt
section
[e]: http://unicode.org/Public/UNIDATA/PropList.txt
[f]: http://unicode.org/reports/tr18/ - UTS #18: Unicode Regular Expressions
[g]: http://unicode.org/reports/tr18/#Compatibility_Properties - xdigit row
[h]: http://bugs.python.org/issue6561#msg90878 point (1) about int() and re
[i]: http://bugs.python.org/issue2636#msg65513 point 8) will introduce
[[:xdigit:]]

(Thanks to Mark Dickinson and Adam Olsen for pointing out some of these
issues.)

The bit that most convinces me that *some* change is desirable is that
(with py3k notation), int('７', 16) is legal but int('Ｆ', 16) is not.

In an ideal world one might hope that the set of characters accepted by
int(s, 16) would be the same as those characters with the Unicode
Hex_Digits property, but currently there's a mismatch for two different
reasons... (1) fullwidth hex digits have property 'Hex_Digit' but aren't
accepted, and (2) non-European decimal digits (e.g. Devanagari digits,
etc.) don't have property 'Hex_Digit' but are accepted by int.

It's tempting to suggest that int and float should be modified to reject
*any* decimal digits other than '0' through '9', and possibly their
fullwidth variants. (Jean-Paul Calderone already advanced this argument
on #python-dev a few days ago; essentially saying, if I understood him
correctly, that dealing with localization shouldn't be part of the job
of int or float.)

Ezio Melotti wrote:

New submission from Ezio Melotti <[email protected]>:

The decimal codec only handles characters in the Nd (Number, decimal)
Unicode category and whitespaces [a]. It is used by int(), float(),
complex() and indirectly by Decimal(), Fraction() and possibly others.
This works well only for plain digits (e.g. int(u'１２３')) but it
doesn't work for all the other characters used to represent numbers, like:
[...]

In general, Python has always stuck to the Unicode standard
for these things (as well as others like casing, etc.).

If the Unicode standard adopts a scheme for dealing with these
issues, we should include support for it.

Implementing something based on non-standards now and breaking
that support later on in order to implement the true standards
is not such a good idea.

There is work underway to define a standard for locale specific
formatting of numbers, dates, etc.:

http://cldr.unicode.org/

Here's the TR with the data format specification:

http://www.unicode.org/reports/tr35/tr35-12.html

I'm sure that the information gathered in that project will
sooner or later be folded back into the standard Unicode character
database. Once that's done we can then use that information to
e.g. determine the characters that make up a sign, decimal
point, etc.

On the specific point of:

2.1 some languages/alphabets use other chars (e.g. a comma or other
symbols) instead of the decimal point.

I think it's not the job of the float() constructor to support it.
Depending on the country, the comma has different meanings when put in a
number (thousands separator or decimal separator). Ditto for the point,
but using a point as decimal separator is the accepted standard for
non-localized computer I/O.

More generally, I think the fact that int(), float() et al. support
non-ASCII decimal digits should be seen as a convenience rather than a
willingness to accomodate the broadest set possible of inputs. Which
means, we should only add support for new formats only if it's sensible,
safe and non-ambiguous to do so.

I also agree with Marc-André's argument that the Unicode spec should be
a good guide here.

Python's current support for localization in int and float seems largely
accidental, as far as I can tell. But I appreciate the argument that
despite the current inconsistencies, we shouldn't add extra support
without a standard to base it on.

I'm not sure how relevant TR35 is to this situation; that seems to be
about localization, and I don't really understand why it should be the
job of int and float to deal with localization (even though they
currently do, to some extent). I'd even argue for removing support[*]
for anything other than ASCII digits from int and float, except that
that would likely break existing applications, and annoy people. Out of
curiosity, I asked on #python about this, and found that there are
people working with CJK alphabets who find it convenient that int
currently accepts fullwidth digits. I don't know whether there's anyone
who cares that int and float currently accept e.g., Devanagari digits.

I guess I'm +0.2 for preserving the status quo, for now.

([*] I'm aware that I'm being a bit inconsistent here, since I was
recently arguing that the Decimal type should accept non-European
decimal digits partly based on the fact that int and float do. But in
the case of Decimal there's an underlying standard that recommends
acceptance of these digits, and compliance with that standard has
generally taken precedence over consistency with Python's other numeric
types.)

Mark Dickinson wrote:

Python's current support for localization in int and float seems largely
accidental, as far as I can tell.

Not at all. The support for non-ASCII numeric characters
in int() and float() is by design - however, I did not look
at things like decimal points, minus/plus signs, etc. at the
time and only included support for numeric values associated
with a number of code points, such as ½ for 1/2.

But I appreciate the argument that
despite the current inconsistencies, we shouldn't add extra support
without a standard to base it on.

I'm not sure how relevant TR35 is to this situation; that seems to be
about localization, and I don't really understand why it should be the
job of int and float to deal with localization (even though they
currently do, to some extent). I'd even argue for removing support[*]
for anything other than ASCII digits from int and float, except that
that would likely break existing applications, and annoy people. Out of
curiosity, I asked on #python about this, and found that there are
people working with CJK alphabets who find it convenient that int
currently accepts fullwidth digits. I don't know whether there's anyone
who cares that int and float currently accept e.g., Devanagari digits.

My suggestion is to wait for the Unicode locale project to collect
locale based information on numeric formatting. I'm sure that some
of this information will be entered back into the Unicode database
in form of code point properties for e.g. decimal points, signs, etc.

[Mark Dickinson]

Python's current support for localization in int and float seems
largely accidental, as far as I can tell.

[MAL]

Not at all. [...]

Apologies; 'accidental' was a poor choice of word here.

however, I did not look
at things like decimal points, minus/plus signs, etc. at the
time and only included support for numeric values associated
with a number of code points, such as ½ for 1/2.

I'm less concerned about decimal points and the like, and more bothered by
the fact that e.g., int(x, 16) accepts some, but not all, characters with
the Hex_Digit property. This seems counter to the intent of the Unicode
standard.

[MAL]

My suggestion is to wait for the Unicode locale project to collect
locale based information on numeric formatting. [...]

Sounds fine to me.

Mark Dickinson wrote:

I'm less concerned about decimal points and the like, and more bothered by
the fact that e.g., int(x, 16) accepts some, but not all, characters with
the Hex_Digit property. This seems counter to the intent of the Unicode
standard.

int()/float() use the decimal codec for numbers - this only supports
base-10 numbers. For hex numbers, we'd need a new hex codec (only
the encoder part, actually), otherwise, int('a') would start to return
10.

Any takers ?

int()/float() use the decimal codec for numbers - this only supports
base-10 numbers. For hex numbers, we'd need a new hex codec (only
the encoder part, actually), otherwise, int('a') would start to return
10.

That's not true. PyUnicode_EncodeDecimal could happily accept hexdigits,
and int(u'a') would still be rejected. In fact, PyUnicode_EncodeDecimal
*already* accepts arbitrary Latin-1 characters, whether they represent
digits or not. I suppose this is to support non-decimal bases, so it
would only be consequential to widen this to all characters that
reasonably have the Hex_Digit property (although I'm unsure which ones
are excluded at the moment).

Martin v. Löwis wrote:

Martin v. Löwis <[email protected]> added the comment:

> int()/float() use the decimal codec for numbers - this only supports
> base-10 numbers. For hex numbers, we'd need a new hex codec (only
> the encoder part, actually), otherwise, int('a') would start to return
> 10.

That's not true. PyUnicode_EncodeDecimal could happily accept hexdigits,
and int(u'a') would still be rejected. In fact, PyUnicode_EncodeDecimal
*already* accepts arbitrary Latin-1 characters, whether they represent
digits or not. I suppose this is to support non-decimal bases, so it
would only be consequential to widen this to all characters that
reasonably have the Hex_Digit property (although I'm unsure which ones
are excluded at the moment).

The codec currently doesn't look at the base at all - and shouldn't
need to:

It simply converts input characters that have a decimal digit value
associated with them, to the usual ASCII digits in preparation
for parsing them using the standard number parsing tools we have in
Python.

This is to support number representations using non-ASCII code
points for digits (e.g. Japanese or Sanskrit numbers)

http://sp.cis.iwate-u.ac.jp/sp/lessonj/doc/numbers.html
http://veda.wikidot.com/sanskrit-numbers

All other Latin-1 characters are passed through as-is, so you
can already use the codec to e.g. prepare parsing of hex
values.

Also note that we already have a hex codec in Python 2.x
which converts between the hex representations of a string
and its regular form. This was removed in 3.x for some reason
I don't understand (probably just an oversight).

The codec currently doesn't look at the base at all - and shouldn't
need to:

It simply converts input characters that have a decimal digit value
associated with them, to the usual ASCII digits in preparation
for parsing them using the standard number parsing tools we have in
Python.

Right. And as such, it shouldn't stop with digit 9, but continue into
digits a, b, c, and so on, as appropriate.

This is to support number representations using non-ASCII code
points for digits (e.g. Japanese or Sanskrit numbers)

Notice that it also supports bases other than 10:

80

So calling it "decimal" is a misnomer.

Also note that we already have a hex codec in Python 2.x
which converts between the hex representations of a string
and its regular form. This was removed in 3.x for some reason
I don't understand (probably just an oversight).

The hex codec doesn't have to do anything with number conversions;
nor does it have to do with character encodings. To introduce it was
a mistake in Python 2.x which has been fixed in 3.x (by removing
it and other similar "codecs", such as rot13).

Martin v. Löwis wrote:

Martin v. Löwis <[email protected]> added the comment:

> The codec currently doesn't look at the base at all - and shouldn't
> need to:
>
> It simply converts input characters that have a decimal digit value
> associated with them, to the usual ASCII digits in preparation
> for parsing them using the standard number parsing tools we have in
> Python.

Right. And as such, it shouldn't stop with digit 9, but continue into
digits a, b, c, and so on, as appropriate.

I don't think that's needed. The codec already passes those
through as-is.

> This is to support number representations using non-ASCII code
> points for digits (e.g. Japanese or Sanskrit numbers)

Notice that it also supports bases other than 10:

80

So calling it "decimal" is a misnomer.

Not really: _PyUnicode_ToDecimalDigit() is used for the
conversion and that API explicitly only returns integer
values for code points that map to the digits 0-9 - at
least that's how it was originally written (see the code
in Python 1.6 which makes this explicit).

If it returns values outside that range, that's a bug
and needs to be fixed, since it would cause the codec
to fail. It is designed to only work on digits, not
arbitrary decimals.

> Also note that we already have a hex codec in Python 2.x
> which converts between the hex representations of a string
> and its regular form. This was removed in 3.x for some reason
> I don't understand (probably just an oversight).

The hex codec doesn't have to do anything with number conversions;
nor does it have to do with character encodings. To introduce it was
a mistake in Python 2.x which has been fixed in 3.x (by removing
it and other similar "codecs", such as rot13).

That's your particular view of things. It's not mine and never
was the basis of the codec design.

Codecs in Python are open to work on arbitrary types and
it's well possible to have codecs that return the same type
as their input.

The hex codec in Python 2.x is a very useful and handy
codec and it's used a lot.

It should be added back again - after all, even by your
restrictive view of codecs in Python only serving as a way to
do character encodings, it is a valid character encoding -
that of Latin-1 code points to a two-byte HEX representation
and vice-versa.

Just like rot-13 and most of the others that were apparently
removed (uu, base64, quoted-printable, zip, bz2).

BTW: I noticed that idna and punycode were not removed...
even though they fall into the same category as the hex
codec.

I guess we should have a discussion about this on python-dev.

I believe this is covered by the PEP-3003 3.2 change moratorium.

The moratorium only includes builtins and certainly does not apply to pure python modules (other implementations see those benefits immediately and it does not interfere with their getting caught-up).

FWIW, I'm +1 on the basic idea.

I wish I discovered this issue when I was working on bpo-10557! Chances are I wouldn't have started a long python-dev thread rehashing the same issues as I see discussed here.

In any case, in bpo-10557, I replaced core uses of PyUnicode_EncodeDecimal() with PyUnicode_TransformDecimalToASCII() which transforms unicode to unicode. There should not be any change in what digits builtin number types accept, but the C-API "codec" is defined to pass through anything that is not "Nd" while replacing Nd characters with the corresponding 0-9 digit.

String to string codecs including hex have been readded in r86934. See bpo-7475.

I wonder if the issues raised here can be neatly addressed by applying NFKC normalization before string to number conversion. This will convert full-width variants to ASCII and also eliminate digit/decimal differences. For example superscript and subscript variants will be converted to ASCII. Note that NFKC normalization is already applied to identifiers, so its effect should be familiar to users.

Alexander, can this be closed or is there still something that should be done?

I am changing the title slightly to include '\N{MINUS SIGN}' in the scope of this issues. See [1]:

"Unless anyone can point me to a case where \N{MINUS SIGN} should not be treated as a (duh) minus sign, we should go and try to make life easier for our users by adopting at least a few of such characters." (Łukasz Langa)

[1] http://mail.python.org/pipermail/python-ideas/2013-June/021243.html

As a design principle, "accept what's unambiguous in any locale" is reasonable, but it is hard to apply consistently. I would agree that the status quo is hard to defend. After a long discussion, it has been accepted that fullwidth digits should be accepted and now float(u'１２３') is valid, but not float('＋１２３'), float('－１２３') or float('１２⒊'). The last example is

>>> '\N{FULLWIDTH DIGIT ONE}\N{FULLWIDTH DIGIT TWO}\N{DIGIT THREE FULL STOP}'
'１２⒊'

All these variations can be neatly addressed by applying NFKC or NFKD normalization to unicode data before conversion:

>>> float(unicodedata.normalize('NFKD', '＋１２３'))
123.0
>>> float(unicodedata.normalize('NFKD', '－１２３'))
-123.0
>>> float(unicodedata.normalize('NFKC', '１２⒊'))
123.0

This would even allow parsing fullwidth hexadecimal numbers:

>>> float.fromhex(unicodedata.normalize('NFKC', '０ｘ⒈７ｐ３'))
11.5
>>> int(unicodedata.normalize('NFKC', '７Ｆ'), 16)
127

but would not help with the MINUS SIGN.

Allowing '\N{MINUS SIGN}' is particularly attractive because arguably unicode text should prefer it to ambiguous '\N{HYPHEN-MINUS}', but on the same token fractions.Fraction() should accept '\N{FRACTION SLASH}' in addition to the legacy '\N{SOLIDUS}'.

Overall, I think this situation calls for a PEP-size proposal and discussion about handling unicode numerical data throughout stdlib rather that a case by case discussion of the various quirks in the curent version.

[In the light of the current discussion on python-ideas regarding adding support for the Unicode minus sign]

I'm +1 on adding support for the minus code point, since
it's the correct correspondent to the plus code point in Unicode.
The traditional ASCII "-" is a compromise between a mathematical
minus and a hyphen.

https://en.wikipedia.org/wiki/Hyphen-minus

While we're at it, we should probably also include the
FULLWIDTH PLUS SIGN (U+FF0B) and SUPERSCRIPT PLUS SIGN (U+207A)
as alternative for the plus sign, and additionally the
SUPERSCRIPT MINUS (U+207B) as alternative for the minus
sign.

http://en.wikipedia.org/wiki/Minus_sign

I'm also +1 on adding support for other mathematical symbols used in parsing numbers, but only if their purpose is clearly defined in the Unicode database (i.e. accepting an "e" in some other script for exponential notation should not be allowed).

This is essentially a duplicate of bpo-10581, so I am closing this and will summarize the situation there.

Navigation Menu

Repository navigation

Include more chars in the decimal codec #50881

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Include more chars in the decimal codec #50881

Description

Activity

ezio-melotti commented on Aug 3, 2009

mdickinson commented on Aug 3, 2009

malemburg commented on Aug 3, 2009

pitrou commented on Aug 5, 2009

mdickinson commented on Aug 6, 2009

malemburg commented on Aug 6, 2009

mdickinson commented on Aug 6, 2009

malemburg commented on Aug 7, 2009

loewis commented on Sep 22, 2009

malemburg commented on Sep 22, 2009

loewis commented on Sep 24, 2009

malemburg commented on Sep 24, 2009

terryjreedy commented on Aug 9, 2010

rhettinger commented on Aug 11, 2010

abalkin commented on Dec 7, 2010

abalkin commented on Dec 30, 2010

ezio-melotti commented on Mar 28, 2013

abalkin commented on Jun 9, 2013

abalkin commented on Jun 10, 2013

malemburg commented on Jun 11, 2013

abalkin commented on Jun 12, 2013

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions