Repository navigation
Fix telco benchmark #70472
Description
| BPO | 26284 |
|---|---|
| Nosy | @brettcannon, @pitrou, @vstinner, @skrah, @1st1 |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = 'https://github.com/vstinner'
closed_at = <Date 2016-09-13.09:05:39.744>
created_at = <Date 2016-02-04.14:28:01.253>
labels = ['type-bug', 'performance']
title = 'Fix telco benchmark'
updated_at = <Date 2016-09-13.09:05:39.742>
user = 'https://github.com/skrah'bugs.python.org fields:
activity = <Date 2016-09-13.09:05:39.742>
actor = 'vstinner'
assignee = 'vstinner'
closed = True
closed_date = <Date 2016-09-13.09:05:39.744>
closer = 'vstinner'
components = ['Benchmarks']
creation = <Date 2016-02-04.14:28:01.253>
creator = 'skrah'
dependencies = []
files = []
hgrepos = []
issue_num = 26284
keywords = []
message_count = 8.0
messages = ['259569', '259789', '260743', '260748', '273922', '273925', '273976', '276218']
nosy_count = 5.0
nosy_names = ['brett.cannon', 'pitrou', 'vstinner', 'skrah', 'yselivanov']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'needs patch'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue26284'
versions = ['Python 3.6']
Activity
skrah commentedon Feb 4, 2016
The telco benchmark is unstable. It needs some of Victor's changes from bpo-26275 and probably a larger data set:
http://speleotrove.com/decimal/expon180-1e6b.zip is too big for
_pydecimal, but the one that is used is probably too small for
_decimal.
skrah commentedon Feb 7, 2016
Unfortunately, replacing io.BytesIO(data) with indexing does not make the benchmark faster or more stable on my machine.
BTW, string conversion of the result is actually a crucial part of
the benchmark, it was taken out in http://bugs.python.org/file41802/telco_haypo.py.
[-]FIx telco benchmark[/-][+]Fix telco benchmark[/+]brettcannon commentedon Feb 23, 2016
If you think the string conversion should go back in, Stefan, feel free to put it back (unless Victor wants to say why he took it out).
vstinner commentedon Feb 23, 2016
Ah, I didn't check. I expected BytesIO.read() to be slower than bytes string slicing.
vstinner commentedon Aug 30, 2016
I worked on a new version of the benchmark suite, it is now called performance and moved to GitHub:
https://github.com/python/performance
In performance 0.1.2, bm_telco.py uses BytesIO for input and six.StringIO for output. The output is just one number per line, it's different than http://bugs.python.org/file41802/telco_haypo.py output.
I don't know which benchmark is "right".
I believe that bm_telco.py of performance is now much more stable thanks to the perf module which runs the benchmark in multiple processes (20 by default). It helps to get a better distribution of samples.
Example:
22.7 ms: 18 #####################################################
23.0 ms: 27 ###############################################################################
23.4 ms: 7 ####################
23.7 ms: 1 ###
24.1 ms: 2 ######
24.4 ms: 1 ###
24.7 ms: 0 |
25.1 ms: 1 ###
25.4 ms: 0 |
25.8 ms: 0 |
26.1 ms: 0 |
26.5 ms: 0 |
26.8 ms: 0 |
27.1 ms: 0 |
27.5 ms: 0 |
27.8 ms: 0 |
28.2 ms: 0 |
28.5 ms: 0 |
28.9 ms: 1 ###
29.2 ms: 1 ###
29.5 ms: 1 ###
Total duration: 15.1 sec
Start date: 2016-08-30T18:09:49
End date: 2016-08-30T18:10:07
Raw sample minimum: 182 ms
Raw sample maximum: 237 ms
Number of runs: 20
Total number of samples: 60
Number of samples per run: 3
Number of warmups per run: 1
Loop iterations per sample: 8
Minimum: 22.8 ms (-2%)
Median +- std dev: 23.2 ms +- 1.4 ms
Mean +- std dev: 23.6 ms +- 1.4 ms
Maximum: 29.7 ms (+28%)
Median +- std dev: 23.2 ms +- 1.4 ms
The histogram helps to see that there is no such "minimum", but more a gaussian curve. The perf module uses the median of samples rather than the minimum, it also displays and computes the standard deviation by default.
vstinner commentedon Aug 30, 2016
@Stefan: Can you please check bm_telco.py and maybe propose a pull request if something is wrong?
skrah commentedon Aug 30, 2016
Wow, on my machine this is very stable, great.
The output should be like
http://www.bytereef.org/software/mpdecimal/benchmarks/telco.py ,
but printing one number only should be okay. The important thing is that some decimal is printed at all to test the formatting speed.
So LGTM.
vstinner commentedon Sep 13, 2016
Sorry, I'm not sure if you want to make any change to bm_telco.py of the performance. If yes, please open an issue at https://github.com/python/performance
Hum, it looks like you are happy, so I now close this issue :-) I'm slowly trying to move discussions related to benchmarking to the performance module on GitHub.