Inconsistent/undocumented urlsplit/urlparse behavior on invalid inputs #73837
Description
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = <Date 2017-05-16.05:50:14.977>
created_at = <Date 2017-02-25.19:45:27.149>
labels = ['3.7', 'type-bug', 'library', 'docs']
title = 'Inconsistent/undocumented urlsplit/urlparse behavior on invalid inputs'
updated_at = <Date 2017-05-16.05:50:14.976>
user = 'https://bugs.python.org/vfaronov'bugs.python.org fields:
activity = <Date 2017-05-16.05:50:14.976>
actor = 'orsenthil'
assignee = 'docs@python'
closed = True
closed_date = <Date 2017-05-16.05:50:14.977>
closer = 'orsenthil'
components = ['Documentation', 'Library (Lib)']
creation = <Date 2017-02-25.19:45:27.149>
creator = 'vfaronov'
dependencies = []
files = []
hgrepos = []
issue_num = 29651
keywords = []
message_count = 6.0
messages = ['288577', '288959', '291640', '293748', '293750', '293751']
nosy_count = 5.0
nosy_names = ['rhettinger', 'orsenthil', 'docs@python', 'vfaronov', 'Howie Benefiel']
pr_nums = ['1128', '1596', '1597']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue29651'
versions = ['Python 2.7', 'Python 3.5', 'Python 3.6', 'Python 3.7']
Activity
vfaronov commentedon Feb 25, 2017
There is a problem with the standard library's urlsplit and urlparse functions, in Python 2.7 (module urlparse) and 3.2+ (module urllib.parse).
The documentation for these functions [1] does not explain how they behave when given an invalid URL.
One could try invoking them manually and conclude that they tolerate anything thrown at them:
Without studying the source code, it is impossible to know that there is a very narrow class of inputs on which they raise ValueError [2]:
>>> urlparse('http://[') Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/usr/lib/python3.5/urllib/parse.py", line 295, in urlparse splitresult = urlsplit(url, scheme, allow_fragments) File "/usr/lib/python3.5/urllib/parse.py", line 345, in urlsplit raise ValueError("Invalid IPv6 URL") ValueError: Invalid IPv6 URLThis could be viewed as a documentation issue. But it could also be viewed as an implementation issue. Instead of raising ValueError on those square brackets, urlsplit could simply consider them *invalid* parts of an RFC 3986 reg-name, and lump them into netloc, as it already does with other *invalid* characters:
>>> urlparse('http://\0\0æí\n/') ParseResult(scheme='http', netloc='\x00\x00æí\n', path='/', params='', query='', fragment='')Note that the raising behavior was introduced in Python 2.7/3.2.
See also bpo-8721 [3].
[1] https://docs.python.org/3/library/urllib.parse.html
[2] https://github.com/python/cpython/blob/e32ec93/Lib/urllib/parse.py#L406-L408
[3] http://bugs.python.org/issue8721
rhettinger commentedon Mar 4, 2017
A note in the docs would be useful. This API is far too well established to make any behavioral changes at this point.
howinator commentedon Apr 14, 2017
I'm going to make a note in the documentation. I should have a PR for it in about 1 day.
orsenthil commentedon May 16, 2017
New changeset f6e863d by Senthil Kumaran (Howie Benefiel) in branch 'master':
bpo-29651 - Cover edge case of square brackets in urllib docs (bpo-1128)
f6e863d
orsenthil commentedon May 16, 2017
New changeset 72e5aa1 by Senthil Kumaran in branch '3.5':
bpo-29651 - Cover edge case of square brackets in urllib docs (bpo-1128) (bpo-1597)
72e5aa1
orsenthil commentedon May 16, 2017
New changeset 75b8a54 by Senthil Kumaran in branch '3.6':
bpo-29651 - Cover edge case of square brackets in urllib docs (bpo-1128) (bpo-1596)
75b8a54