Repository navigation
[regression] reading from a urllib2 file descriptor happens byte-at-a-time #46853
Description
| BPO | 2601 |
|---|---|
| Nosy | @mhammond, @akuchling, @doko42, @gpshead, @pitrou |
| Superseder |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = 'https://github.com/akuchling'
closed_at = <Date 2008-04-30.05:56:14.812>
created_at = <Date 2008-04-08.21:15:29.632>
labels = ['release-blocker', 'library', 'performance']
title = '[regression] reading from a urllib2 file descriptor happens byte-at-a-time'
updated_at = <Date 2008-04-30.05:59:01.255>
user = 'https://github.com/doko42'bugs.python.org fields:
activity = <Date 2008-04-30.05:59:01.255>
actor = 'mhammond'
assignee = 'akuchling'
closed = True
closed_date = <Date 2008-04-30.05:56:14.812>
closer = 'mhammond'
components = ['Library (Lib)']
creation = <Date 2008-04-08.21:15:29.632>
creator = 'doko'
dependencies = []
files = []
hgrepos = []
issue_num = 2601
keywords = []
message_count = 10.0
messages = ['65219', '65488', '65503', '65504', '65517', '65538', '65539', '65540', '65545', '65990']
nosy_count = 7.0
nosy_names = ['mhammond', 'akuchling', 'nnorwitz', 'doko', 'gregory.p.smith', 'pitrou', 'schmir']
pr_nums = []
priority = 'release blocker'
resolution = 'duplicate'
stage = None
status = 'closed'
superseder = '2632'
type = 'performance'
url = 'https://bugs.python.org/issue2601'
versions = ['Python 2.6', 'Python 2.5']
Activity
doko42 commentedon Apr 8, 2008
r61009 on the 2.5 branch
calling .read() on a socket object wrapped with makefile().
causes a regression compared to 2.4.5 and 2.5.2:
When reading from urllib2 file descriptor, python will read the data a
byte at a time regardless of how much you ask for. python versions up to
2.5.2 will read the data in 8K chunks.
This has enough of a performance impact that it increases download time
for a large file over a gigabit LAN from 10 seconds to 34 minutes. (!)
Trivial/obvious example code:
... and then strace it to see the recv()'s chugging along, one byte at a
time.
pitrou commentedon Apr 14, 2008
See bpo-2632 for more discussion of what is probably the same issue.
nnorwitz commentedon Apr 15, 2008
Bumping the priority. I'd like to see this fixed before the next
release. What version(s) does this problem apply to: 2.5, 2.6, 3.0?
schmir commentedon Apr 15, 2008
quoting http://bugs.python.org/issue1389051: "Applied to 2.6 trunk in
rev. 61008 and to 2.5-maint in rev. 61009."
I don't know about py3k...
akuchling commentedon Apr 15, 2008
It was applied to 2.5-maint after 2.5.2 was released, BTW, so the change
isn't in any stable released version, only the 2.6 alphas.
nnorwitz commentedon Apr 16, 2008
So if the fix was applied to 2.5 branch and 2.6 (3.0 should have
picked up from 2.6 automatically), can we close this bug?
gpshead commentedon Apr 16, 2008
I don't think the fix was acceptable. Now python spins consuming all
cpu trying to read trivial amounts of data one byte at a time...
See the discusson at the end of http://bugs.python.org/issue1092502 as
well as a recent python-dev thread:
http://mail.python.org/pipermail/python-dev/2008-April/078613.html
gpshead commentedon Apr 16, 2008
or else i'm missing something here in the maze of three bugs talking
about the same issue..
which revisions fixed the introduced performance issue?
schmir commentedon Apr 16, 2008
me and amk are talking about the commit that introduced this bug (which
was meant as a fix for another bug).
neal seems to think that this commit is the fix to this bug itself.
and gregory, you are now confused :)
hope it's clear now.
mhammond commentedon Apr 30, 2008
For those trying to follow along at home: best I can tell we have 3
other issues on this: bpo-1092502 and bpo-1389051 are dupes of an initial bug,
but the fix for those bugs caused regressions reported in this bug and
in bpo-2632. To try and reduce confusion I'm closing this as a dupe of
bpo-2632 which has a patch for review.