Repository navigation
email.parser: impossible to read messages encoded in a different encoding #48911
Description
| BPO | 4661 |
|---|---|
| Nosy | @warsaw, @pitrou, @vstinner, @bitdancer |
| Files |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = 'https://github.com/bitdancer'
closed_at = <Date 2010-12-12.18:03:17.612>
created_at = <Date 2008-12-14.16:55:45.574>
labels = ['type-bug', 'expert-email']
title = 'email.parser: impossible to read messages encoded in a different encoding'
updated_at = <Date 2014-10-03.01:07:07.673>
user = 'https://bugs.python.org/dato'bugs.python.org fields:
activity = <Date 2014-10-03.01:07:07.673>
actor = '[email protected]'
assignee = 'r.david.murray'
closed = True
closed_date = <Date 2010-12-12.18:03:17.612>
closer = 'r.david.murray'
components = ['email']
creation = <Date 2008-12-14.16:55:45.574>
creator = 'dato'
dependencies = []
files = ['18962', '19078', '19102', '19110', '19113', '19155', '19160', '19161', '36788']
hgrepos = []
issue_num = 4661
keywords = ['patch']
message_count = 25.0
messages = ['77807', '89508', '89509', '91737', '95293', '117118', '117119', '117120', '117774', '117856', '117857', '117893', '117897', '117900', '118160', '118175', '118191', '118192', '118197', '118198', '118199', '118201', '118204', '123843', '228289']
nosy_count = 9.0
nosy_names = ['barry', 'anadelonbrin', 'pitrou', 'vstinner', 'r.david.murray', 'dato', 'tercero12', 'meatballhat', '[email protected]']
pr_nums = []
priority = 'high'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue4661'
versions = ['Python 3.2']
Activity
dato commentedon Dec 14, 2008
Currently, email.parser/feedparser can only parse messages that come
as a string, or from a file opened in text mode.
Email messages, however, can contain 8bit characters in any encoding
other than the local one (yet still be valid e-mails, of course), so I
think a method is needed to have the parser be able to receive bytes.
At the moment, and as far as I can see, it's not possible to parse
some perfectly valid messages with python 3.0.
I don't think it's appropriate to ask that files be opened with the
proper encoding, and then for the parser to read them. First, it is
not possible to know what encoding that would be without parsing the
message. And second, a message could contain parts in different
encoding, and many mailboxes with random messages most certainly do.
Also, message objects will need a way to return a bytes repreentation,
for the reasons explained above, and particularly if one wants to
write back the message without modifying it.
bitdancer commentedon Jun 18, 2009
Is there any use case for feedparser accepting strings as input that
isn't a design error waiting to bite the programmer?
warsaw commentedon Jun 18, 2009
dato: We've started some branches that try to address this, by exposing
both a read-a-buncha-bytes interface and a read-a-string interface.
rdm: As it turns out, yes. There are use cases for reading a string
containing only ascii bytes.
In general, this is part of the big tricky problem in fixing the email
package for Python 3.x. We had great debates about this at Pycon with
no resolution, and I think everyone has just been too busy to engage on
this since then. :(
email-sig is the best place to try to rally some effort.
AlexQuinn commentedon Aug 19, 2009
This bug also prevents the cgi module from handling POST data with
multipart/form-data. Consequently, 3.x cannot be readily used to write
web apps for uploading files. See bpo-4953:
http://bugs.python.org/issue4953
tercero12 commentedon Nov 15, 2009
Just an update for people interested:
The email team has a goal of fixing the email module in time for the 3.2
release. There is the possibility of having to change some interfaces.
See this document: http://wiki.python.org/moin/Email%20SIG/DesignThoughts
bitdancer commentedon Sep 21, 2010
OK, I'm not entirely sure I want to post this, but....
Antoine and I were having a conversation about nntplib and email and I noted that unicode as an email transmission channel acts as if it required 7bit clean data. That is, that there's no way to use unicode as an 8bit data transmission channel. Antoine pointed out that there is PEP-383, and that he is using that in his nntplib update to tunnel 8bit data (if there is any) from and back to the nntp server. I said I couldn't do that with email because I not only needed to transmit the data, I also needed to *parse* it.
Antoine pointed out that you can in fact parse a header even if it has surrogateescape code points in it.
So I started thinking about that. In point of fact, from the point of view of an email parser, non-ASCII bytes are pretty much opaque. They don't affect the semantics of the parsing. Either they are invalid data (in headers), or they are opaque content data (8bit Content-Transfer-Encoding).
So...I came up with a horrible little hack, which is attached here as a patch. This is horrible because it is a perversion of the Python3 desire to make a clean separation between bytes and strings. The only thing it really has to recommend it is that it works: it allows email5 (the version of email currently in Python3) to read wire-format messages and parse them into valid message structures.
The patch is a proof of concept and is far from complete. It handles only message bodies (but those are the most important) and has no doc updates and only one test. If this approach is deemed worth considering, I will flesh out the tests and make sure the corner cases are handled correctly, and write docs with lots of notes about why this is perverse and email6 will make it all better :)
I feel bad about posting this both because it is an ugly hack and because it will likely slow down email6 development (because it will make email5 mostly work). But making email5 mostly work in 3.2 seems like a case where practicality beats purity.
The essence of the hack is as follows: Given binary data we encode it to ASCII using the surrogateescape error handler. Then, when a message body is retrieved we check to see if there are any surrogates in it, and if there are we encode it back to ASCII using surrogateescape, thereby recovering the original bytes. For "Content-Transfer-Encoding: 8bit" parts we can then try to decode it using the declared charset, or ASCII with the replace error handler if the charset isn't known. But in any case the original binary data is accessible by using 'decode=True' in the call to get_payload. (NB for those not familiar with the API: decode=True refers to decoding the Content-Transfer-Encoding, *not* decoding to unicode...which means after CTE decoding you end up with a byte string).
For headers, which are not supposed to have 8bit data in them, the best we can do is re-decode them with ASCII/replace, but at least it will be possible to parse the messages. (The current patch doesn't do this.)
Another thing missing from the current patch is the generator side. But since the binary data for the message content is now available, it should be possible to have a generator that outputs binary.
Note that in this patch I've introduced new functions/methods for getting binary string data in, but for file input one needs to open the file as text using ASCII encoding and the surrogateescape error handler.
I've only done minimal testing on this (obviously), and so I may find a showstopper somewhere along the way, but so far it seems to work, and logically it seems like it should work.
I don't know if that makes me happy or sad :)
pitrou commentedon Sep 21, 2010
A couple of comments:
str(self.get_param('charset', 'ascii'))supposed to achieve? does get_param() return a bytes object?bitdancer commentedon Sep 21, 2010
The 'str' around get_param shouldn't be there, that was left over from an earlier version of the patch.
I use surrogateescape rather than latin1 because using surrogateescape with ascii encoding gives me a reliable way to know whether or not the original source was bytes containing non-ascii chars. (If it was bytes containing only ascii chars, then it doesn't matter what the original source was.)
bitdancer commentedon Oct 1, 2010
New version of the patch that adds many more tests, and handles non-ASCII bytes in header values by changing them to '?'s when the header value is retrieved as a string. I think I'm half done. Still to do: generate_bytes, and the doc updates.
By the way, another important reason to use surrogateescape rather than latin1 is that if I miss something and the byte-containing-strings escape, it will be obvious that that is what happened. Otherwise we're back in Python2 bytes/string conflation land.
I of course make no promises about performance. And there is an issue there in that every header value access is now wrapped in an additional function call and a regex test, at a minimum, whether there are bytes present in the input or not :(
bitdancer commentedon Oct 2, 2010
New version of patch including a BytesGenerator.
6 remaining items
pitrou commentedon Oct 8, 2010
There are a couple of things I don't understand:
+* :class:`~email.generator.Generator` will convert message bodies that
+ have a :mailheader:`ContentTransferEncoding` of 8bit and a known charset to
+ instead have a :mailheader:`CotnentTransferEncoding` of ``QuotedPrintable``.
Why so?
+* All operations are on unicode strings. Text inputs must be strings,
+ text outputs are strings.
This is a rather strange statement given that you are adding bytes-consuming and bytes-producing functions.
bitdancer commentedon Oct 8, 2010
Generator converts 8bit bodies into 7bit bodies by applying an appropriate 7bit CTE. The reason it does this is that the output of Generator will often be passed to some other Python library function (most often smtplib) that can only handle ASCII unicode input. That is, Generator now produces a 7bit clean message that can be put on the wire by encoding it to ascii. This means that RFC-compliant bytes input can be successfully transmitted onward using Generator and smtplib, whereas if Generator produced non-ASCII unicode it would not be possible to pass a message with an 8bit CTE on to smtplib.
The statement about string input and output is a description of email *5.0*, the existing email package in 3.0 and 3.1, before my patch. The differences between 4.0 and 5.0 were never previously added to the docs, so I had to add them in order to then describe the differences between 5.0 and 5.1.
pitrou commentedon Oct 8, 2010
What do you mean, ASCII *unicode* input? Any low-level network library
should accept bytes when arbitrary data is possible.
Enforcing 7-bit means things like binary attachments can grow larger for
no real reason. Also, raw message bodies become less readable (which
obviously is very minor issue).
Ah, my bad. Sorry.
bitdancer commentedon Oct 8, 2010
Even if smtplib accepted bytes (it currently does not), *Generator* is still producing unicode, and should produce valid unicode and still insofar as possible preserve the meaning of the original message. This means unicode acts as if it is an SMTP server that does not support the 8bit capability, so we must convert to 7bit clean CTEs.
If smtplib later grows the ability to accept bytes, BytesGenerator can be used to feed it. I've clarified that BytesGenerator does not do the 7bit transform, and made some other doc tweaks.
pitrou commentedon Oct 8, 2010
That sounds like a critical failure.
bitdancer commentedon Oct 8, 2010
I can only fix one package at a time :)
And in case it isn't clear, the "Generator produces ASCII-only unicode", which is in many ways a rather strange API, is one of the chief motivations for email6.
bitdancer commentedon Oct 8, 2010
Here is the final pre-alpha patch. This one includes the BytesFeedParser class and a test.
Unless there are objections I'd like to commit this. Believing the code needs a more thorough review would be a valid objection :)
bitdancer commentedon Oct 8, 2010
After RM approval on irc, committed in r85322, with some additional doc fixes but no code changes relative to the last patch posted here.
I'm leaving this open because I still want to try to improve the handling of non-ascii bytes in headers when decoding them to unicode.
bitdancer commentedon Dec 12, 2010
I've opened a bpo-10686 to address improving the RFC conformance by using unknown-8bit encoded words for 8bit bytes in headers.
yuzhaogetcwdcom commentedon Oct 3, 2014
BytesParser.parse uses TextIOWrapper which by default translates universal newlines to '\n'. This breaks binary payload.
Fix the problem by disabling the translation.