Repository navigation
Badly formed XML using etree and utf-16 #45279
Description
| BPO | 1767933 |
|---|---|
| Nosy | @amauryfa, @florentx, @serhiy-storchaka |
| Dependencies | |
| Files |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = <Date 2012-07-15.03:02:42.829>
created_at = <Date 2007-08-05.15:01:57.000>
labels = ['expert-XML', 'type-bug']
title = 'Badly formed XML using etree and utf-16'
updated_at = <Date 2012-07-18.06:49:34.570>
user = 'https://bugs.python.org/bugok'bugs.python.org fields:
activity = <Date 2012-07-18.06:49:34.570>
actor = 'serhiy.storchaka'
assignee = 'effbot'
closed = True
closed_date = <Date 2012-07-15.03:02:42.829>
closer = 'python-dev'
components = ['XML']
creation = <Date 2007-08-05.15:01:57.000>
creator = 'bugok'
dependencies = ['6472']
files = ['12009', '18211', '25386', '25652', '26300', '26316', '26377', '26425']
hgrepos = []
issue_num = 1767933
keywords = ['patch']
message_count = 34.0
messages = ['32587', '32588', '32589', '75864', '75866', '75875', '99394', '111533', '111608', '111611', '111631', '111635', '117864', '159491', '161046', '161222', '161236', '163739', '163756', '164713', '164858', '164918', '165003', '165350', '165366', '165415', '165492', '165508', '165673', '165675', '165714', '165723', '165740', '165746']
nosy_count = 11.0
nosy_names = ['effbot', 'nnorwitz', 'amaury.forgeotdarc', 'bugok', 'rurwin', 'eli.bendersky', 'flox', 'BreamoreBoy', 'Richard.Urwin', 'python-dev', 'serhiy.storchaka']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue1767933'
versions = ['Python 2.7', 'Python 3.2', 'Python 3.3']
Activity
bugok commentedon Aug 5, 2007
Hello,
The bug occurs when writing an XML file using the UTF-16 encoding.
The problem is that the etree encodes every string to utf-16 by itself - meaning, inserting the 0xfffe BOM before every string (tag, text, attribute name, etc.), causing a badly formed utf=16 strings.
A possible solution, which was offered by a co-worker of mine, was to use a utf-16 writer (from codecs.getwriter('utf-16') to write the file.
Best,
BugoK.
nnorwitz commentedon Aug 7, 2007
Fredrik, could you take a look at this?
effbot commentedon Aug 7, 2007
ET's standard serializer currently only supports ASCII-compatible encodings. See e.g.
The best workaround for ET 1.2 (Python 2.5) is probably to serialize as "utf-8" and transcode:
rurwin commentedon Nov 14, 2008
This is a bug in two halves.
Not all characters in the file are UTF-16. The initial xml header
isn't, and the individual < > etc characters are not. This is just a
matter of extending the methodology to encode all characters and not
just the textual bits. There is no work-around except a five-minute hack
of the ElementTree.write() method.
Every write has a BOM, so corrupting the file in a manner analogous
to bug 555360. This is a result of using string.encode() and is a
well-known feature. It can be worked around by using UTF-16LE or
UTF-16BE which do not prepend a BOM, but then the file doesn't have any
BOM. A complete solution would be to rewrite ElementTree.write() to use
a different encoding methodology such as StreamWriter.
I have made the above hack and work-around for my own use, and I can
report that it produces perfect UTF-16.
amauryfa commentedon Nov 14, 2008
Would you provide a patch?
rurwin commentedon Nov 14, 2008
Here is a patch of my quick hack, more for interest than any suggestion
it gets used. Although it does produce good output so long as you avoid
the BOM.
The full solution is beyond my (very weak) Python skills. The character
encoding is tied in with XML character substitution (& etc. and
hexadecimal representation of multibyte characters). I could disentangle
it, but I probably wouldn't produce optimal Python, or indeed anything
that wouldn't inspire mirth and/or incredulity.
NB. The workaround suggested by Fredrik Lundh doesn't solve our
particular problems, since the downsize to UTF-8 causes the multi-byte
characters to be represented in hex. Our software doesn't read those. (I
know that's our problem.)
florentx commentedon Feb 16, 2010
Could you provide a test case, so we can check if the upgrade proposed on bpo-6472 solves this issue?
BreamoreBoy commentedon Jul 25, 2010
@richard: Could you provide a test case for this, or do you consider it beyond your Python capabilities allowing for your comments on msg75875?
RichardUrwin commentedon Jul 26, 2010
I can't produce an automated test, for want of time, but here is a demonstrator.
Grab the example XHTML from http://docs.python.org/library/xml.etree.elementtree.html#elementtree-objects or use some tiny ASCII-encoded xml file. Save it as "file.xml" in the same folder as bug-test.py attached here.
Execute bug-test.xml
file.xml is read and then written in UTF-16. The output file is then read and dumped to stdout as a byte-stream.
To be correct UTF-16, the output should start with 255 254, which should never occur in the rest of the file.
The rest of the output (including the first line) should alternate zeros with ASCII character codes.
The file output.xml should be loadable in a UTF16-capable text editor (eg jEdit), be recognised as UTF-16 and be identical in terms of content to file.xml
13 remaining items
serhiy-storchaka commentedon Jul 8, 2012
Here is a patch with using context management (as Eli advised). This
makes error handling much safer and probably makes the code a little
easier. Several new tests are added.
elibendersky commentedon Jul 13, 2012
Thanks, this looks much better. I've reviewed the _4 patch with some minor comments.
elibendersky commentedon Jul 13, 2012
Serhiy, can you also take a look at bpo-9458 - it may be related?
serhiy-storchaka commentedon Jul 13, 2012
Patch updated with some comments.
python-dev commentedon Jul 15, 2012
New changeset 6120cf695574 by Eli Bendersky in branch 'default':
Close bpo-1767933: Badly formed XML using etree and utf-16. Patch by Serhiy Storchaka, with some minor fixes by me
http://hg.python.org/cpython/rev/6120cf695574
serhiy-storchaka commentedon Jul 15, 2012
Thank you, Eli.
However changes to tostring() and tostringlist() break the invariant b"".join(tostringlist(element, 'utf-16')) == tostring(element, 'utf-16'). You should add followed methods to DataStream:
Note, that monkey-patched version is faster.
Benchmark results:
tostring() with BytesIO:
$ ./python -m timeit -s "import xml.etree.ElementTree as ET; e=ET.XML('<root/>')" "ET.tostring(e, 'utf-16')"
1000 loops, best of 3: 268 usec per loop
$ ./python -m timeit -s "import xml.etree.ElementTree as ET; e=ET.XML('<root>'+'<child/>'*100+'</root>' )" "ET.tostring(e, 'utf-16')"
100 loops, best of 3: 4.63 msec per loop
tostring() with monkey-patching:
$ ./python -m timeit -s "import xml.etree.ElementTree as ET; e=ET.XML('<root/>')" "ET.tostring(e, 'utf-16')"
1000 loops, best of 3: 263 usec per loop
$ ./python -m timeit -s "import xml.etree.ElementTree as ET; e=ET.XML('<root>'+'<child/>'*100+'</root>' )" "ET.tostring(e, 'utf-16')"
100 loops, best of 3: 3.84 msec per loop
tostringlist() with DataStream class:
$ ./python -m timeit -s "import xml.etree.ElementTree as ET; e=ET.XML('<root/>')" "ET.tostringlist(e, 'utf-16')"
1000 loops, best of 3: 624 usec per loop
$ ./python -m timeit -s "import xml.etree.ElementTree as ET; e=ET.XML('<root>'+'<child/>'*100+'</root>' )" "ET.tostringlist(e, 'utf-16')"
100 loops, best of 3: 4.09 msec per loop
tostringlist() with monkey-patching:
$ ./python -m timeit -s "import xml.etree.ElementTree as ET; e=ET.XML('<root/>')" "ET.tostringlist(e, 'utf-16')"1000 loops, best of 3: 259 usec per loop
$ ./python -m timeit -s "import xml.etree.ElementTree as ET; e=ET.XML('<root>'+'<child/>'*100+'</root>' )" "ET.tostringlist(e, 'utf-16')"
100 loops, best of 3: 3.81 msec per loop
elibendersky commentedon Jul 17, 2012
Fixed the invariant violation in changeset 64ff90e07d71
I'll review the performance difference separately
elibendersky commentedon Jul 17, 2012
I posted a message to python-dev about the performance issue
python-dev commentedon Jul 17, 2012
New changeset 51978f89e5ed by Eli Bendersky in branch 'default':
Optimize tostringlist by taking the stream class outside the function. It's now 2x faster on short calls. Related to bpo-1767933
http://hg.python.org/cpython/rev/51978f89e5ed
serhiy-storchaka commentedon Jul 17, 2012
How about porting this to 3.2? The main difficulty I see with the tests, which significantly differ in 3.2 and 3.3.
elibendersky commentedon Jul 18, 2012
Frankly, I don't think the problem is serious enough to warrant a backport to 3.2, given that 3.3 gonna be out in a few weeks. The issue was open for 5 years without anyone seriously complaining :)
serhiy-storchaka commentedon Jul 18, 2012
Python 3.2 currently shipped in last Ubuntu LTS and will be in production at least next 5 years. I think it will be main Python version for many years.
Here is a compound patch (changesets 6120cf695574, 64ff90e07d71, 51978f89e5ed and 63ba0c32b81a) for Python 3.2 without tests. It is almost same as for 3.3, except using manual finalizing instead ExitStack.