Repository navigation
Serialiser in ElementTree returns unicode strings in Py3k #52295
Description
| BPO | 8047 |
|---|---|
| Nosy | @malemburg, @birkenfeld, @scoder, @bitdancer, @florentx |
| Files |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = <Date 2011-10-29.02:37:45.468>
created_at = <Date 2010-03-03.07:15:23.877>
labels = ['expert-XML', 'easy', 'type-bug', 'library', 'docs']
title = 'Serialiser in ElementTree returns unicode strings in Py3k'
updated_at = <Date 2011-10-29.02:37:45.467>
user = 'https://github.com/scoder'bugs.python.org fields:
activity = <Date 2011-10-29.02:37:45.467>
actor = 'flox'
assignee = 'none'
closed = True
closed_date = <Date 2011-10-29.02:37:45.468>
closer = 'flox'
components = ['Documentation', 'Library (Lib)', 'XML']
creation = <Date 2010-03-03.07:15:23.877>
creator = 'scoder'
dependencies = []
files = ['18286']
hgrepos = []
issue_num = 8047
keywords = ['easy']
message_count = 47.0
messages = ['100333', '100342', '100345', '100349', '100350', '100513', '100572', '100582', '100633', '100634', '100649', '100846', '100857', '100868', '100877', '100880', '100883', '100884', '100887', '100890', '100891', '100895', '100896', '100898', '100900', '100902', '100903', '100907', '100915', '100916', '100919', '100923', '100929', '100930', '100931', '100932', '100936', '101050', '101052', '101427', '101487', '101488', '101490', '112165', '113296', '113307', '146593']
nosy_count = 6.0
nosy_names = ['lemburg', 'effbot', 'georg.brandl', 'scoder', 'r.david.murray', 'flox']
pr_nums = []
priority = 'normal'
resolution = 'out of date'
stage = 'needs patch'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue8047'
versions = ['Python 3.1']
Activity
scoder commentedon Mar 3, 2010
The xml.etree.ElementTree package in the Python 3.x standard library breaks compatibility with existing ET 1.2 code. The serialiser returns a unicode string when no encoding is passed. Previously, the serialiser was guaranteed to return a byte string. By default, the string was 7-bit ASCII compatible.
This behavioural change breaks all code that relies on the default behaviour of ElementTree. Since there is no longer a default encoding in Python 3, unicode strings are incompatible with byte strings, which means that the result of the serialisation can no longer be written to a file, for example.
XML is well defined as a stream of bytes. Redefining it as a unicode string *by default* is hard to understand at best.
Finally, it would have been good to look at the other ET implementation before introducing such a change. The lxml.etree package has had support for serialising XML into a unicode string for years, and does so in a clear, safe and explicit way. It requires the user to pass the 'unicode' (Py3 'str') type as encoding parameter, e.g.
which is explicit enough to make it clear that this is different from a normal encoding.
bitdancer commentedon Mar 3, 2010
I'm not an ElementTree user, but that spelling (etree.tostring(encode=str), or even etree.tostring(encode=unicode)) strikes me as horrible. You don't encode to unicode, you *decode* to unicode. Thus the current Python3 interface works the way I'd expect: if I don't specify an encoding, I get unicode. If I do specify an encoding, I get encoded bytes. In the general case the fact that you can no longer get away with being sloppy about what encoding a byte stream is in, the way you could in Python2, is a feature of Python3, not a bug.
If anything, having 'tostring' return bytes is broken, given its name. But I think we fudge that by claiming it is returning a 'byte string' when given an encoding.
That said, I'm not sure how much, if at all, my opinion counts :)
scoder commentedon Mar 3, 2010
I agree that the lxml API is somewhat clumsy here. I just mentioned it to show that there are already ways to do it in a backwards compatible way, so this change does two things: it breaks existing code, and it does so in a way that is incompatible with other existing implementations. That's what *I* would call horrible.
Also, this is absolutely not a feature that is restricted to Py3, so what's the equivalent feature in the standard library of Py2 going to be, and how much code will it break for the Py2 series?
bitdancer commentedon Mar 3, 2010
My understanding is that backward compatibility, while nice to retain, was not considered a stopper for cleaning up interfaces in py3. Exactly how considered this change was, I have no idea, but as I said it does make sense to me. As for 2.x, what's there is what's there, as far as I can see. Florent could speak to whether or not that API is likely to change in 2.7, but I doubt it will.
florentx commentedon Mar 3, 2010
With ET 1.3, the serializer ElementTree.write() should output bytes only. And the default encoding is still US-ASCII.
The new behaviour is specific to the 3.x branch (since 3.0, r56841).
Even if it is not fully backward compatible, I don't find this behavior shocking: it is a rule of Python 3 to avoid implicit encoding/decoding.
pitrou commentedon Mar 6, 2010
I don't know what compatibility you are talking about. Py3k deliberately breaks compatibility with many 2.x behaviours that were considered defective or suboptimal.
scoder commentedon Mar 7, 2010
It has been brought up several times that ET is special in the stdlib in that it is an externally maintained package. Correct me if I'm wrong, but the rules seem to be: features come outside, adaptation to Py3 can happen inside. What we are talking about here is a new feature that makes sense for both Py2 and Py3. We are not talking about a bug fix, neither is this an adaptation to Py3. It is a new feature that was added inside of the standard library and that is not compatible with the external libraries that are supposed to implement the same interface, namely, ElementTree and lxml.etree.
pitrou commentedon Mar 7, 2010
As Florent said, it is a rule of py3k to avoid implicit encoding/decoding. The fact that it could have made sense for 2.x as well is not relevant, since the change was only done in py3k (and for good reason: we normally try not to break compatibility without prior notice).
In any case, I have trouble understanding your concern here. Do you think the change is bad? Is it really that difficult to support it in lxml?
scoder commentedon Mar 8, 2010
Antoine, in the same comment, you say that it was not backported to Py2 in order to prevent breaking existing code, and then you ask if it's difficult to support in lxml. ;-)
Supporting the same behaviour in lxml would either mean that it breaks existing code in Py2 (when making the API consistent), or that you can safely (and correctly) write the return value to a file in Py2, but that you can't do the same in Py3 (when adopting the change only in Py3).
Previously, in ElementTree, serialising without an explicit encoding was a way to get a byte encoded serialisation without an XML declaration header, so I expect there to be code that depends on this. Since ElementTree 1.3 uses the same keyword argument as lxml for this feature, I assume that Florent's patches provide at least an alternative here, even if it requires users to adapt their code.
I just wish this backwards incompatible feature had been advertised at the time, or at least *documented* in any way. Even the latest 3.2-dev docs still state that the default encoding of the serialiser is US-ASCII, not a word about *ever* returning a unicode string, especially not by default, and totally not the required big fat warning that writing to a file will fail with mysterious errors if no encoding is specified.
florentx commentedon Mar 8, 2010
With ET 1.3, you should have an explicit keyword argument "xml_declaration":
# ----
if xml_declaration or (xml_declaration is None and
encoding not in ("utf-8", "us-ascii")):
if method == "xml":
write("<?xml version='1.0' encoding='%s'?>\n" % encoding)
# ----
In ET 1.2.6, the same snippet looks like:
# ----
pitrou commentedon Mar 8, 2010
Le Mon, 08 Mar 2010 09:01:19 +0000,
Stefan Behnel <[email protected]> a écrit :
I meant breaking existing *user* code. Besides, the fact that
compatibility is broken doesn't mean third-party code difficult to fix;
hence my question.
Sorry, I don't understand this. Are you saying it's impossible
for you to define two different behaviours based on the current Python
version? What's bad with
"""if sys.version_info() >= (3, 0, 0): # blah"""
This doesn't seem to be documented. The doc simply says
"""encoding is the output encoding (default is US-ASCII)""".
In other words, undocumented (and untested) behaviour has been "broken"
when porting to 3.0, which is the version which deliberately broke
compatibility for documented things. I guess we can live with it ;)
Ok, perhaps some documentation changes are in order :-)
(I wonder why the default was US-ASCII, though. Sounds a bit braindead)
28 remaining items
effbot commentedon Mar 21, 2010
Hmm. I'm not entirely sure about giving False a meaning when None has traditionally had a different (and documented) meaning. And sleeping on it hasn't convinced me in either direction :-(
(well, I'd say no, but the compatibility argument is somewhat tempting)
I'm not that concerned by changing the default for write -- 3.x users with utf-8 as the default output encoding will get different output, but still perfectly valid XML. 3.x users with non-utf-8 default encodings will get valid XML also in cases where it didn't work before.
tostring() is more problematic, but I'm leaning towards Guido's torpedoes approach there -- changing the default output to bytestrings is more likely to cause code to blow up than cause bad output, and you can trivially make your program backwards compatible by adding an extra check/decode after the call. Supporting unicode for lxml.etree compatibility is fine with me, but I think it might make sense to support the string "unicode" as well (as a pseudo-encoding -- it's pretty clear to me that nobody will ever define a real character encoding with that name :-).
Have you posted/can you post the patch to riedveld, btw? I have some questions about the code that are independent of the encoding decision.
florentx commentedon Mar 22, 2010
http://codereview.appspot.com/664043 (patch against 3.x)
IIUC, the changes proposed (for 3.2) are:
==> fallback to 'US-ASCII' encoding (instead of Unicode)
==> serialize to Unicode
And it changes the behavior of :
For 2.x we could add the options for Unicode output:
scoder commentedon Mar 22, 2010
The reason I chose the unicode type over a 'unicode' string name at the time was that I wanted to make a clear distinction to show that this is not just selecting a different codec but that it changes the output type.
I don't really care either way, though, given that this reads a lot less well in Py3. If ET supports both, lxml will follow.
Stefan
malemburg commentedon Mar 22, 2010
Stefan Behnel wrote:
There's always the possibility of adding a new official codec
called 'unicode' which converts Unicode to Unicode as no-op.
This may also be useful to have in other situations where you
want to signal a special case for Unicode input or output.
florentx commentedon Jul 31, 2010
Patch updated here, and on Rietveld too.
http://codereview.appspot.com/664043
Rules (as discussed):
(compatible with 2.7 and lxml.etree)
(compatible with lxml.etree)
For 2.7, no change planned.
For 3.1, do we keep the current behavior?
scoder commentedon Aug 8, 2010
I would suggest fixing the tostring() behaviour also in a future 3.1.x bug fix release. After all, the current behaviour means that 3.0 and 3.1 would behave different from any other (released or future) Python version here.
florentx commentedon Aug 8, 2010
Done for 3.2 with r83851.
Still opened, if someone wants to propose a patch for 3.1.
florentx commentedon Oct 29, 2011
3.1 is no longer in scope for this issue.