Repository navigation
xml.etree.ElementTree.ElementTree.write(): encoding handling problems #53703
Description
| BPO | 9458 |
|---|---|
| Nosy | @vstinner, @merwok, @florentx, @dabrahams, @serhiy-storchaka |
| Files |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = <Date 2012-07-15.03:20:58.401>
created_at = <Date 2010-08-02.21:02:04.633>
labels = ['expert-XML', 'type-bug', 'docs']
title = 'xml.etree.ElementTree.ElementTree.write(): encoding handling problems'
updated_at = <Date 2012-07-15.03:20:58.398>
user = 'https://bugs.python.org/kune'bugs.python.org fields:
activity = <Date 2012-07-15.03:20:58.398>
actor = 'eli.bendersky'
assignee = 'docs@python'
closed = True
closed_date = <Date 2012-07-15.03:20:58.401>
closer = 'eli.bendersky'
components = ['Documentation', 'XML']
creation = <Date 2010-08-02.21:02:04.633>
creator = 'kune'
dependencies = []
files = ['18349', '18350']
hgrepos = []
issue_num = 9458
keywords = ['patch']
message_count = 12.0
messages = ['112550', '112568', '112653', '112658', '148280', '158903', '158905', '164714', '164745', '165378', '165493', '165494']
nosy_count = 10.0
nosy_names = ['effbot', 'vstinner', 'eric.araujo', 'kune', 'eli.bendersky', 'flox', 'docs@python', 'dabrahams', 'python-dev', 'serhiy.storchaka']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'test needed'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue9458'
versions = ['Python 2.7', 'Python 3.2', 'Python 3.3']
Activity
kune commentedon Aug 2, 2010
If one wants to use the encoding parameter of ElementTree.write() the file must be opened with "wb". Without encoding parameter normal files can be used, but the should be opened with the encoding "UTF-8", because otherwise this may create an error.
Probably comparable problems exist with the parser side of things.
merwok commentedon Aug 3, 2010
Is this a behavior bug or a doc bug?
kune commentedon Aug 3, 2010
I believe handling of TextIOWrapper streams is broken in xml.etree.ElementTree.ElementTree.write().
First example:
assert sys.stdout.encoding == "UTF-8"
element_tree.write(sys.stdout, encoding="UTF-8")
print()
I don't think that write a tree into a stream with the correct encoding should generate any problem at all.
The output looks like this:
Example 2:
import sys
from xml.etree import ElementTree
The first ö umlaut generates an UnicodeEncodeError here, while the method could use XML character references. One could argue this, but the method could take care of the problem.
Third example:
import sys
from xml.etree import ElementTree
with open("bug3.xml", "w", encoding="ISO-8859-1",
errors="xmlcharrefreplace") as f:
element_tree.write(f, xml_declaration=True)
This creates finally an ISO-8859-1 encoded XML file, but without XML declaration. Didn't we request one?
Example 4: Try to do the right thing.
import sys
from xml.etree import ElementTree
with open("bug4.xml", "w", encoding="ISO-8859-1",
errors="xmlcharrefreplace") as f:
element_tree.write(f, encoding="ISO-8859-1", xml_declaration=True)
Here we get the same exception as example 1 of course.
All the files can be found in the tar container below.
[-]xml.etree.ElementTree.write(): encoding handling problems[/-][+]xml.etree.ElementTree.ElementTree.write(): encoding handling problems[/+]kune commentedon Aug 3, 2010
Here is a patch that handles all 4 examples in the last comment correctly and survives the Python test suite on Linux (Ubuntu 9.04 x86-64).
merwok commentedon Nov 24, 2011
Thanks for the patch. The examples in your message need to be converted to a patch that applies to 3.2 or 2.7, so that we can reproduce the bug before fixing it.
dabrahams commentedon Apr 21, 2012
These bugs are annoying. How does one convert a set of examples into a patch? Do you mean you want these to become test cases?
merwok commentedon Apr 21, 2012
Yes. See the devguide if you need more info.
elibendersky commentedon Jul 6, 2012
Please make sure that the patch(es) apply cleanly to 3.3, since this is the version I'll be focusing on.
dabrahams commentedon Jul 6, 2012
I won't get to this, FYI.
serhiy-storchaka commentedon Jul 13, 2012
ElementTree write works with two kinds of output -- binary and text. The difference between them is only determined by encoding argument. If encoding is "unicode", then output is text, else it is binary. There is no other way for filename or general file-like object to determine kind of output. If these are not explained in the documentation, then the documentation should be improved.
The patch can cause data corruption because direct writing to underlying file by fileno conflicts with TextIOBase/BufferedIOBase internal buffering. And not every file-like object have fileno. With patch the behavior becomes less obvious and will lead to confusion.
I don't see a behavior bug which should be fixed.
Only one thing can be enhanced -- error diagnostic in some corner cases. When we can determines that file object is instance of RawIOBase or TextIOBase and it is conflicts with encoding argument value, it will be helpful for novices to raise a descriptive exception. This is of course not eliminate all causes for confusing.
python-dev commentedon Jul 15, 2012
New changeset 51b5ee7cfa3b by Eli Bendersky in branch 'default':
Issue bpo-9458: clarify the documentation of ElementTree.write with regards to the type of the stream expected for a given encoding
http://hg.python.org/cpython/rev/51b5ee7cfa3b
elibendersky commentedon Jul 15, 2012
I agree with Serhiy that this is more of a documentation/understanding issue than a real bug. I've clarified the doc of ElementTree.write a bit to make it explicit what stream is expected for 'write'.