Repository navigation
zipimport is a bit slow #52991
Description
| BPO | 8745 |
|---|---|
| Nosy | @brettcannon, @rhettinger, @ericsnowcurrently, @serhiy-storchaka |
| Files |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = 'https://github.com/serhiy-storchaka'
closed_at = <Date 2013-02-16.21:54:44.637>
created_at = <Date 2010-05-18.06:08:09.415>
labels = ['interpreter-core', 'performance']
title = 'zipimport is a bit slow'
updated_at = <Date 2013-02-16.21:54:44.635>
user = 'https://bugs.python.org/Goplat'bugs.python.org fields:
activity = <Date 2013-02-16.21:54:44.635>
actor = 'serhiy.storchaka'
assignee = 'serhiy.storchaka'
closed = True
closed_date = <Date 2013-02-16.21:54:44.637>
closer = 'serhiy.storchaka'
components = ['Interpreter Core']
creation = <Date 2010-05-18.06:08:09.415>
creator = 'Goplat'
dependencies = []
files = ['17387', '25823', '28116', '28117', '28599', '28612']
hgrepos = []
issue_num = 8745
keywords = ['patch']
message_count = 14.0
messages = ['105954', '105960', '106191', '162300', '174934', '176367', '176376', '178602', '178828', '179226', '179268', '180312', '182230', '182241']
nosy_count = 8.0
nosy_names = ['brett.cannon', 'rhettinger', 'ysj.ray', 'Goplat', 'catalin.iacob', 'python-dev', 'eric.snow', 'serhiy.storchaka']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'performance'
url = 'https://bugs.python.org/issue8745'
versions = ['Python 3.4']
Activity
Goplat commentedon May 18, 2010
Reading the list of files in a .zip takes a while because several seeks are done for each entry, which (on Windows at least) flushes stdio's buffer and forces a system call on the next read. For large .zips the effect on startup speed is noticeable, being perhaps 50ms per thousand files. Changing the read_directory function to read the central directory entirely sequentially would cut this time by more than half.
ysjray commentedon May 18, 2010
When I perform some test on debian-5.0, I see the timing results almost the same before and after apply your patch(I modified the patch to against the trunk).
Could you give some test result on windows? I can't see the speedups on debian-5.0.
Goplat commentedon May 20, 2010
Zipping up the Lib directory from the python source (1735 files) as a test, it took on average 0.10 sec to read the zip before, 0.04 sec after.
(To test the time taken by zipimport isolated from other startup costs, instead of actually getting the zip in the import path, I just ran
cataliniacob commentedon Jun 4, 2012
I updated Goplat's patch to the default branch.
It now needs to read 4 dummy bytes instead of 6 since an extra PyMarshal_ReadShortFromFile was added to the default branch in the mean time. I added an explicit dummy buffer instead of reading the dummy bytes into name (for cleanness and because name would overflow on hypothetical platforms where MAXPATHLEN + 5 < 8). Also added tests for the loop that skips the rest of the header by creating some zips with file comments; without the extra test, commenting out the loop didn't fail test_zipimport.
Running Goplat's test in msg106191 on Windows I get 0.032 sec before and 0.015 sec after. On Linux I see no significant difference.
AFAIK Mercurial (for example) ships with a zipped stdlib on Windows and they care quite a lot about startup time. Can this make it into 3.3?
serhiy-storchaka commentedon Nov 5, 2012
I suggest to use fseek(fp, relative_offset, SEEK_CUR). It doesn't force a system call (at least on Linux) and may be a little faster than fread() or multiple getc().
cataliniacob commentedon Nov 25, 2012
I tried Serhiy's suggestion in msg174934, see attached attempt-fseek-seek_cur.patch updated to current default. It makes no difference relative to the default branch for my test stdlib.zip, dummy reads still work better on Windows.
This makes sense if you follow the CRT's implementation, fseek calls _fseek_nolock which always calls _flush, regardless whether SEEK_CUR is used or not. This is the case with both VS2008 and VS2010. So this patch is a workaround for poor fseek performance in Microsoft's CRT, it doesn't cause performance issues on Linux but saves quite some milliseconds of startup time so I think it's worth the tradeoff.
I'll also upload zipimport_speedup-v3.patch updated to apply to current default and with error handling for failing freads since the fseeks that the patch replaces have gained error handling on the default branch in the mean time. The timings remain the same on my Windows machine: around 30ms for default branch, around 15ms with patch.
serhiy-storchaka commentedon Nov 25, 2012
I think some clarifying comments will be good in the code. In particular about the
dummyvariable.Perhaps getc() requires error handling too.
serhiy-storchaka commentedon Dec 30, 2012
Catalin, are you going to continue?
cataliniacob commentedon Jan 2, 2013
Yes Serhiy, I was planning to, lack of time followed by (just finished) vacation lead to a stall. Thanks for following up on this.
I should soon, probably this weekend, have an updated version addressing your comments.
cataliniacob commentedon Jan 6, 2013
Attached v4 of patch with error checking for getc and some more comments.
A real world example of the speedup is Calibre (http://calibre-ebook.com/) which on Windows comes with its own code, its dependecies and Python's stdlib all bundled in a 40MB zip. With the patch the time to create a zipimporter instance from that zip drops from around 60ms to around 15ms.
serhiy-storchaka commentedon Jan 7, 2013
In general, the patch LGTM, however I can't try it on Windows, and on Linux it has no any performance effect. Can anyone try it on Windows?
I have re-uploaded the patch for review after converting it from UTF-16 and CRLF.
serhiy-storchaka commentedon Jan 20, 2013
Sorry, Brian. Raymond is an initiator of bpo-17004.
python-dev commentedon Feb 16, 2013
New changeset 088a14031998 by Serhiy Storchaka in branch 'default':
Issue bpo-8745: Small speed up zipimport on Windows. Patch by Catalin Iacob.
http://hg.python.org/cpython/rev/088a14031998
serhiy-storchaka commentedon Feb 16, 2013
Thank you for contribution.