Repository navigation
json library is decoding/encoding when it should not #57421
Description
| BPO | 13212 |
|---|---|
| Nosy | @rhettinger, @etrepum, @pitrou, @vstinner, @ezio-melotti, @merwok, @florentx, @akheron |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = 'https://github.com/etrepum'
closed_at = <Date 2014-11-27.19:34:05.235>
created_at = <Date 2011-10-18.18:00:31.817>
labels = ['type-bug', 'library']
title = 'json library is decoding/encoding when it should not'
updated_at = <Date 2014-11-27.19:34:05.234>
user = 'https://bugs.python.org/thinred'bugs.python.org fields:
activity = <Date 2014-11-27.19:34:05.234>
actor = 'serhiy.storchaka'
assignee = 'bob.ippolito'
closed = True
closed_date = <Date 2014-11-27.19:34:05.235>
closer = 'serhiy.storchaka'
components = ['Library (Lib)']
creation = <Date 2011-10-18.18:00:31.817>
creator = 'thinred'
dependencies = []
files = []
hgrepos = []
issue_num = 13212
keywords = []
message_count = 21.0
messages = ['145847', '145853', '145888', '145895', '145916', '145917', '145995', '146142', '146144', '146147', '146152', '168941', '168960', '170158', '170194', '213991', '218642', '219142', '219143', '219401', '231772']
nosy_count = 11.0
nosy_names = ['rhettinger', 'bob.ippolito', 'pitrou', 'vstinner', 'ezio.melotti', 'eric.araujo', 'cvrebert', 'flox', 'petri.lehtinen', 'thinred', 'hac.man']
pr_nums = []
priority = 'high'
resolution = 'wont fix'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue13212'
versions = ['Python 2.7', 'Python 3.2', 'Python 3.3']
Activity
thinred commentedon Oct 18, 2011
Hi there!
According to http://www.ietf.org/rfc/rfc4627.txt . The JSON object is either a list or a dictionary with other elements in it. However Python's implementation took liberty to parse also basic types, like integers, strings, floats, etc. For example:
It wouldn't be a problem if other implementations would adhere to this behavior, but this is not true. Take Ruby for example:
irb(main):007:0> require 'rubygems';
irb(main):008:0* require 'json';
irb(main):009:0* JSON.parse '42'
JSON::ParserError: 743: unexpected token at '42'
from /var/lib/gems/1.8/gems/json-1.6.1/lib/json/common.rb:148:in
parse' from /var/lib/gems/1.8/gems/json-1.6.1/lib/json/common.rb:148:inparse'from (irb):9
from �:0
irb(main):010:0> JSON.generate(3.141592)
JSON::GeneratorError: only generation of JSON objects or arrays allowed
from /var/lib/gems/1.8/gems/json-1.6.1/lib/json/common.rb:216:in
generate' from /var/lib/gems/1.8/gems/json-1.6.1/lib/json/common.rb:216:ingenerate'from (irb):10
from �:0
Also http://jsonlint.com/ considers it an error.
I know this is not very important, but I've ran into this today when trying to parse JSON in Ruby which was produced in Python.
I've made a patch for that, but I still need to fix tests and there are many of them. I first would like to know if it is going to be fixed, then I will complete the patch.
Thanks for Python, anyway!
Tomek
rhettinger commentedon Oct 18, 2011
I don't see any way of changing this without breaking lots of code that currently works fine. It may be possible to add a strict-mode (off by default) or to add warnings.
ezio-melotti commentedon Oct 19, 2011
I wonder if there are concrete use cases for loading/dumping basic types.
I see two options here:
thinred commentedon Oct 19, 2011
It wouldn't be a problem if 'json' library would be a thing specific to Python. But JSON is used to gule together different technologies and programming languages. I'm afraid that the default behavior should be 1).
rhettinger commentedon Oct 19, 2011
I'm thinking that we should offer a strict-option (off by default) and leave it at that. No need to break existing code. AFAICT, the OP is the only one to have had problems with this in-practice (the JSON module or its ancestor simplejson has been around for a long time).
Am marking this as a high priority but want to leave it open for a good while to get more comments before taking any action.
pitrou commentedon Oct 19, 2011
Agreed that a strict parameter would be a good idea to implement this.
Since it's technically a new feature (and not that important), I think only 3.3 should receive it.
ezio-melotti commentedon Oct 20, 2011
I'm not sure adding a strict option is a good idea. HTMLParser got a strict argument and it's causing more problems than it solves. If another problem comes up later we would either have to change the behavior for strict and break code that by then is using strict=True or add yet another really_strict argument.
merwok commentedon Oct 21, 2011
ezio-melotti commentedon Oct 21, 2011
rhettinger commentedon Oct 21, 2011
Assigning back to me (the current maintainer of the module).
1 remaining item
thinred commentedon Aug 23, 2012
Take a look at http://bugs.python.org/issue14674.
A proposed documentation patch specifically discusses this issue.
akheron commentedon Aug 23, 2012
The strict flag sounds good to me, too. I don't think it should be called "strict" though, as that would imply that we comply with RFC 4627 strictly (which is not true without passing allow_nan=False for dump() or passing a parse_constant function for load()). How about "encode_any" and "decode_any" or "any_value"?
Deprecation doesn't sound good to me. If the feature is already there, it probably has its users and use cases.
hacman commentedon Sep 10, 2012
I think it's worth pointing out that both Firefox and Chrome support the non-standard JSON that Python supports (serializing and deserializing basic types). I'm guessing that communicating with web browsers is the vast majority of JSON IPC. That is to say, supporting the de-facto standard implemented by web browsers may be better than adhering to the exact specifications of the RFC.
Maybe someone else wants to check what IE, Safari, Opera, and the various phone browsers allow as that might influece the discussion one way or another.
merwok commentedon Sep 10, 2012
What if the strict mode implied both of these things?
FWIW, I’m +1 for a doc section about how to achieve strict mode with special arguments and callbacks (if the recent doc patch does not already have that), and +0 on adding it to the code itself.
cvrebert commentedon Mar 18, 2014
The docs added by that patch do indeed cover this:
http://docs.python.org/2/library/json.html#standard-compliance
cvrebert commentedon May 16, 2014
Note that, per the new JSON RFC 7159 (https://tools.ietf.org/html/rfc7159 ), top-level non-collection values are now de-jure permissible in JSON:
etrepum commentedon May 26, 2014
As Chris Rebert mentioned, the JSON standards have adopted this (unsurprising) behavior. Ruby hasn't, and I doubt Crockford has, but I think they're in the minority at this point. JavaScript's own JSON implementation works the same way json/simplejson does.
At best, I think it's probably worth a mention in the documentation, but not worth changing any code over.
etrepum commentedon May 26, 2014
In other words, I would consider this to be fixed by the documentation change made elsewhere.
cvrebert commentedon May 30, 2014
Okay, so can this issue be closed in light of the existing docs and bpo-21514 then?
cvrebert commentedon Nov 27, 2014
Ping!
Seems like this should be closed since the new RFC explicitly legalizes the feature in question and since the docs explicitly warn about the interoperability of the feature.