multipart/related header causes false positive StartBoundaryNotFoundDefect and MultipartInvariantViolationDefect

@warsaw

BPO	36226
Nosy	@warsaw, @bitdancer, @vadmium, @maxking, @tzickel, @grische
PRs	gh-80407: Fix multipart false positive header defects #12214

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2019-03-07.15:31:17.399>
labels = ['3.7', 'type-bug', 'library']
title = 'multipart/related header causes false positive StartBoundaryNotFoundDefect and MultipartInvariantViolationDefect'
updated_at = <Date 2019-09-04.15:58:20.261>
user = 'https://github.com/grische'

bugs.python.org fields:

activity = <Date 2019-09-04.15:58:20.261>
actor = 'ned.deily'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2019-03-07.15:31:17.399>
creator = 'cschmidbauer'
dependencies = []
files = []
hgrepos = ['381']
issue_num = 36226
keywords = ['patch']
message_count = 5.0
messages = ['337395', '337401', '337444', '337661', '351114']
nosy_count = 6.0
nosy_names = ['barry', 'r.david.murray', 'martin.panter', 'maxking', 'tzickel', 'cschmidbauer']
pr_nums = ['12214']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue36226'
versions = ['Python 3.7']

The current implementation of multipart/related in urllib triggers header defects even though the headers are valid:
[StartBoundaryNotFoundDefect(), MultipartInvariantViolationDefect()]

The example header is valid according to RFC 2387 (https://tools.ietf.org/html/rfc2387):

Content-Type: multipart/related; boundary="==="

Both defects are triggered by the fact that httplib only passes on headers to the underlying email parser, while the email parser assumes to receive a full message. The simple fix is to tell the underlying email parser that we are only passing the header: 0a89fc15c93c271eb08e46e2cda9a72adb97d633

The second issue is related, but independent: The underlying email parser checks if the parsed message is of type multipart by checking of the object "root" is of type list. As we only passed the header (and set headersonly=True), the check does makes no sense anymore at this point, creating a false positive: fdc7c47b77e330a36255fd00dc36accd72824e5b

Apologies, here are the correct commit IDs:

8928543

a82e662

Probably the same as bpo-29353. I remember than enabling "headersonly" can create inconsistencies in the message object. But I don't remember the details.

According to bpo-29991 (another duplicate), my patch for bpo-24363 might help. But I don't think I got much enthusiasm reviewing that (and I don't have time to spend on it now).

@martin.panter: I see a relation to bpo-29353, but I don't see why this report here is a duplicate. Could you elaborate on this?

bpo-29991 contains parts of what I reported here, but it is closed "resolved" and refers back to 29353.

I also tried your patch "policy-flag.patch" and it did not help in the regard of the bug here and tests which are included in the PR.

It should be noted that this causes a big headache for users of requests / urllib3 / etc... as those print on each multipart response a logging warning based on this bug, and it might cause people to go try debugging valid code:

home-assistant/core#17042

Azure/azure-storage-python#167

and others....