Repository navigation
testPythonOrg() of test_robotparser fails with the new www.python.org website #64918
Description
| BPO | 20719 |
|---|---|
| Nosy | @jcea, @vstinner, @ned-deily |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = <Date 2015-03-18.13:27:54.919>
created_at = <Date 2014-02-21.09:20:19.667>
labels = []
title = 'testPythonOrg() of test_robotparser fails with the new www.python.org website'
updated_at = <Date 2015-03-18.13:27:54.918>
user = 'https://github.com/vstinner'bugs.python.org fields:
activity = <Date 2015-03-18.13:27:54.918>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2015-03-18.13:27:54.919>
closer = 'vstinner'
components = []
creation = <Date 2014-02-21.09:20:19.667>
creator = 'vstinner'
dependencies = []
files = []
hgrepos = []
issue_num = 20719
keywords = []
message_count = 9.0
messages = ['211836', '211840', '211841', '211852', '211853', '211889', '211898', '212538', '238433']
nosy_count = 4.0
nosy_names = ['jcea', 'vstinner', 'ned.deily', 'python-dev']
pr_nums = []
priority = 'normal'
resolution = 'out of date'
stage = None
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue20719'
versions = ['Python 3.4']
Activity
vstinner commentedon Feb 21, 2014
I read somewhere that python.org has a new website and the new website has gzip compression enabled which makes some urllib tests failing. It's probably related.
http://buildbot.python.org/all/builders/SPARC%20Solaris%2010%20%28cc%2C%2064b%29%20%5BSB%5D%203.x/builds/1750/steps/test/logs/stdio
======================================================================
ERROR: testPythonOrg (test.test_robotparser.NetworkTestCase)
----------------------------------------------------------------------
vstinner commentedon Feb 21, 2014
http://buildbot.python.org/all/builders/x86%20FreeBSD%206.4%203.x/builds/4531/steps/test/logs/stdio
======================================================================
ERROR: testPythonOrg (test.test_robotparser.NetworkTestCase)
----------------------------------------------------------------------
vstinner commentedon Feb 21, 2014
http://buildbot.python.org/all/builders/x86%20Windows%20Server%202003%20%5BSB%5D%203.x/builds/2166/steps/test/logs/stdio
======================================================================
ERROR: testPythonOrg (test.test_robotparser.NetworkTestCase)
----------------------------------------------------------------------
ned-deily commentedon Feb 21, 2014
It looks like the new python.org web server configuration was just changed to no longer gzip robots.txt so the test is no longer failing for me.
ned-deily commentedon Feb 21, 2014
... or, more likely, that a robots.txt file is now in place.
vstinner commentedon Feb 21, 2014
If I check HTTP headers of http://www.python.org/robots.txt using a small Python script sending "GET /robots.txt HTTP/1.0" and "Host: www.python.org" (but no Accept-Encoding header): I still see "Content-Encoding: gzip".
It looks like a bug in the HTTP server serving www.python.org, because my client didn't send "Accept-Encoding: gzip, deflate".
The RFC 2616 (HTTP/1.1) says "If no Accept-Encoding field is present in a request, the server MAY assume that the client will accept any content coding."
http://www.w3.org/Protocols/rfc2616/rfc2616.html
See also:
"HTTP/1.1 (unlike HTTP/1.0) carefully specifies the Accept-Encoding header, used by a client to indicate what content-codings it can handle, and which ones it prefers."
http://www8.org/w8-papers/5c-protocols/key/key.html
The best solution would be to implement bpo-1508475: support gzip in urllib.
[-]test_robotparser failure on "SPARC Solaris 10 (cc%2C 64b) [SB] 3.x" buildbot[/-][+]testPythonOrg() of test_robotparser fails with the new ww.python.org website[/+][-]testPythonOrg() of test_robotparser fails with the new ww.python.org website[/-][+]testPythonOrg() of test_robotparser fails with the new www.python.org website[/+]ned-deily commentedon Feb 22, 2014
Interesting. As of last night, I'm no longer seeing 'gzip' encoding and the test passes for me. But I see some of the buildbots intermittently failing. Looking at the headers for www.python.org/robots.txt, it appears that the file is being served from a varnish cache and from a CDN so there may be different responses depending on which server responds.
>>> r1.getheaders() [('Server', 'nginx'), ('Content-Type', 'text/plain'), ('X-Frame-Options', 'SAMEORIGIN'), ('Content-Length', '690'), ('Accept-Ranges', 'bytes'), ('Date', 'Fri, 21 Feb 2014 23:53:23 GMT'), ('Via', '1.1 varnish'), ('Age', '2858'), ('Connection', 'keep-alive'), ('X-Served-By', 'cache-sv62-SJC3'), ('X-Cache', 'HIT'), ('X-Cache-Hits', '1')]In any case, supporting gzip would be a good idea but tests will need to have a more repeatable URL.
python-dev commentedon Mar 2, 2014
New changeset 540ce9bb19e8 by Georg Brandl in branch '3.3':
bpo-20719: Disable the robotparser python.org test until the gzip encoding issue can be sorted.
http://hg.python.org/cpython/rev/540ce9bb19e8
vstinner commentedon Mar 18, 2015
This issue has been worked around.