setlocale error message is confusing

@malemburg

BPO	3067
Nosy	@malemburg, @loewis, @terryjreedy, @vstinner, @ezio-melotti, @merwok, @skrah, @akheron
Files	issue3067.patch issue3067_v2.patch issue3067_v3.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2011-11-05.08:28:25.795>
created_at = <Date 2008-06-09.12:08:13.592>
labels = ['easy', 'type-bug', 'docs']
title = 'setlocale error message is confusing'
updated_at = <Date 2011-11-05.08:28:25.793>
user = 'https://bugs.python.org/vincentchute'

bugs.python.org fields:

activity = <Date 2011-11-05.08:28:25.793>
actor = 'petri.lehtinen'
assignee = 'docs@python'
closed = True
closed_date = <Date 2011-11-05.08:28:25.795>
closer = 'petri.lehtinen'
components = ['Documentation']
creation = <Date 2008-06-09.12:08:13.592>
creator = 'vincent.chute'
dependencies = []
files = ['23439', '23447', '23452']
hgrepos = []
issue_num = 3067
keywords = ['patch', 'easy']
message_count = 27.0
messages = ['67861', '67862', '82511', '112650', '138619', '138715', '138718', '138719', '138721', '138739', '138756', '145789', '145818', '145827', '145852', '145855', '145859', '145862', '145864', '145866', '147030', '147032', '147060', '147069', '147070', '147075', '147076']
nosy_count = 12.0
nosy_names = ['lemburg', 'loewis', 'terry.reedy', 'vstinner', 'ezio.melotti', 'eric.araujo', 'vincent.chute', 'skrah', 'docs@python', 'nailor', 'python-dev', 'petri.lehtinen']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue3067'
versions = ['Python 2.7', 'Python 3.2', 'Python 3.3']

import locale
locale.setlocale( locale.LC_ALL, u'ja_JP.utf8')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.5/locale.py", line 475, in setlocale
    locale = normalize(_build_localename(locale))
  File "/usr/lib/python2.5/locale.py", line 383, in _build_localename
    language, encoding = localetuple
ValueError: too many values to unpack

The problem is line 473:
if locale and type(locale) is not type(""):

Replacing this with
if locale and not isinstance(locale, basestring):
fixes the problem.

I have confirmed this exists on trunk

http://svn.python.org/view/python/trunk/Lib/locale.py?rev=63824&view=markup

(63824 is the latest)
where the line in question is now 475

FWIW the type("") is gone in Py3, now it is:
"if locale and not isinstance(locale, _builtin_str):"
where "from builtins import str as _builtin_str"
(http://svn.python.org/view/python/branches/py3k/Lib/locale.py?view=markup)

However Py3.0 now raises the same error when the second arg is a byte
string:
>>> locale.setlocale(locale.LC_ALL, b'ja_JP.utf8')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Programs\Python30\lib\locale.py", line 500, in setlocale
    locale = normalize(_build_localename(locale))
  File "C:\Programs\Python30\lib\locale.py", line 408, in _build_localename
    language, encoding = localetuple
ValueError: too many values to unpack

On Py3, locale.setlocale() should allow only unicode strings and reject
byte strings.

The docs say that the locale arg should be None, tuple, or string, so I take that to mean that Unicode should be OK for 2.x, and that would help porting to 3.x. If bytes are rejected in 3.x, there should be TypeError raised, not ValueError, as is still the case in 3.1.2.

After more thought and investigation, I have changed my opinions on this issue.

Allowing unicode string for locale in 2.7:
Since the module predates unicode strings (it is in 1.5) and since the locale string is passed to a C function, 'string' in the doc can just as well be taken to mean ascii byte string only, as the code requires. As far as I know, unicode is never needed. Allowing such could be considered a feature addition, which is not allowed for 2.7. So I would reject the OP's request (and have hence changed the title).

Expected failure cases could be added to test_locale.py.

Options for locale name:
As I remember, multiple assignments in 1.5, as in

def _build_localename(localetuple):
    language, encoding = localetuple

required a tuple on the right and was called 'tuple unpacking'.
Now, any iterable producing 2 items works; Rather than change to code to check that 'localetuple' really is a tuple (which could break code and the principle of duck-typing), I think the doc should be updated to

"If locale is specified, it may be a None, a string, or an iterable producing two strings, language code and encoding."

This is not a feature addition but a recognition of a new feature added versions ago. The parameter name in the private function should then be shortened to just 'locale'.

Test cases with non-tuples could be added if not present now.

Exception message:
The current message arises from setlocale assuming that a 'locale' that is neither None or a string is a valid iterable for a call to _build_localename. A strings of the wrong type produces too many items and hence the obscure message.

Python is known to be inconsistent in its usage of ValueError versus TypeError for builtin function args. Guido has said to leave inconsistencies rather than break code. So I retract the ValueError to TypeError suggestion.

The accompanying messages, however, can be improved. The lines above that fails could be wrapped with
try:
language, encoding = locale
except ValueError:
raise ValueError("Locale must be None, a string, or an iterable of two strings -- language code, encoding -- not {}".format(locale))

The scope of the wrapper could be extended to the entire function so that failure of
return language + '.' + encoding
would also be caught. Failure would happen if locale produced 2 non-strings items, so that the double assignment 'worked', but the string concatenation failed.

A complication: the doc says
"exception locale.Error
Exception raised when setlocale() fails.

locale.setlocale(category, locale=None) 
...If the modification of the locale fails, the exception Error is raised."

So it seems that either a) the wrapper above should raise Error instead, or the doc could more clearly say "Exception raised when the locale passed to setlocale is not recognized."

"Since the module predates unicode strings (it is in 1.5) and since the locale string is passed to a C function, 'string' in the doc can just as well be taken to mean ascii byte string only, as the code requires."

My only comment is that generally it doesn't seem reasonable to me that developer should need to investigate the history and implementation of a function in order to understand the documentation correctly.

On Py3, locale.setlocale() should allow only unicode strings
and reject byte strings.

I agree and it is the current behaviour (of Python 3.3). I don't see any use case of a byte strings in locale.setlocale() with Python 3.3, so I remove Python 3 from the versions of this issue.

New changeset d370d609d09b by Victor Stinner in branch '2.7':
Close bpo-3067: locale.setlocale() accepts a Unicode locale.
http://hg.python.org/cpython/rev/d370d609d09b

Thanks for clarification! I see the problem now. So if I get this
correctly we should change the _build_localename to raise TypeError?

Yes, that's what I'm proposing.

If the given locale is in wrong format, we'll get TypeError, but if
it's valid type but otherwise invalid locale (like 'en'), we'll get
ValueError (or more specifically locale.Error).

Ideally, yes. Notice that it will be difficult to produce a TypeError
for u"en", unless you explicitly test for Unicode objects.

Uploaded a new patch that raises TypeError

New changeset 931ae170e51c by Petri Lehtinen in branch '3.2':
Issue bpo-3067: Fix the error raised by locale.setlocale()
http://hg.python.org/cpython/rev/931ae170e51c

New changeset d90d88380aca by Petri Lehtinen in branch 'default':
Issue bpo-3067: Fix the error raised by locale.setlocale()
http://hg.python.org/cpython/rev/d90d88380aca

Terry: Do you still think there's need for a doc update?

Yes. I think in locale.rst (assuming that is the name)
'''
exception locale.Error
Exception raised when setlocale() fails.

locale.setlocale(category, locale=None) 
If *locale* is specified, it may be a string, a tuple of the form (language code, encoding), or None. If it is a tuple, it is converted to a string using the locale aliasing engine.
'''
should be changed to
'''
exception locale.Error 
Exception raised when the locale passed to setlocale() is not recognized.

locale.setlocale(category, locale=None) 
If *locale* is specified, it may be a None, a string, or an iterable of two strings, language code and encoding. String pairs are converted to a single string using the locale aliasing engine.
'''
where language code and encoding are gray shaded as they are now.

If *locale* is specified, it may be a None, a string, or an iterable of two strings, language code and encoding. String pairs are converted to a single string using the locale aliasing engine.

What about the possible None value then? Do you think that mentions to
it be dropped?

I don't think so, because setlocale(locale.LC_ALL, None) is an
explicit way of saying "Return me the current value", especially
because the function's name is SETlocale, which doesn't make it
explicit.

If None is not dropped, the ", language code and encoding" should
maybe be in parentheses insteead: "to strings (language code and
encoding), or None..."

Yes, parentheses would be better.

New changeset 34c9465f5023 by Petri Lehtinen in branch '2.7':
Issue bpo-3067: Enhance the documentation and docstring of locale.setlocale()
http://hg.python.org/cpython/rev/34c9465f5023

New changeset 98806dd03506 by Petri Lehtinen in branch '3.2':
Issue bpo-3067: Enhance the documentation and docstring of locale.setlocale()
http://hg.python.org/cpython/rev/98806dd03506

New changeset 8a27920efffe by Petri Lehtinen in branch 'default':
Issue bpo-3067: Enhance the documentation and docstring of locale.setlocale()
http://hg.python.org/cpython/rev/8a27920efffe

I decided to restructure the documentation of setlocale() a bit and I think it's better now overall. It includes Terry's suggestions.

I think this issue can now be closed. Thanks for the report and patches!

Navigation Menu

Repository navigation

setlocale error message is confusing #47317

11 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

setlocale error message is confusing #47317

Description

Activity

vincentchute commented on Jun 9, 2008

vincentchute commented on Jun 9, 2008

ezio-melotti commented on Feb 20, 2009

terryjreedy commented on Aug 3, 2010

terryjreedy commented on Jun 18, 2011

vincentchute commented on Jun 20, 2011

vstinner commented on Jun 20, 2011

python-dev commented on Jun 20, 2011

11 remaining items

loewis commented on Oct 18, 2011

nailor commented on Oct 18, 2011

python-dev commented on Nov 4, 2011

akheron commented on Nov 4, 2011

terryjreedy commented on Nov 5, 2011

akheron commented on Nov 5, 2011

terryjreedy commented on Nov 5, 2011

python-dev commented on Nov 5, 2011

akheron commented on Nov 5, 2011

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions