Repository navigation
Support the Sitemap extension in robotparser #65674
Description
| BPO | 21475 |
|---|---|
| Nosy | @rhettinger, @ned-deily, @berkerpeksag, @matrixise, @mcscope |
| PRs | |
| Dependencies | |
| Files |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = <Date 2018-05-16.14:54:05.895>
created_at = <Date 2014-05-12.01:35:57.203>
labels = ['easy', '3.8', 'type-feature', 'library']
title = 'Support the Sitemap extension in robotparser'
updated_at = <Date 2018-05-16.14:54:05.893>
user = 'https://github.com/rhettinger'bugs.python.org fields:
activity = <Date 2018-05-16.14:54:05.893>
actor = 'ned.deily'
assignee = 'none'
closed = True
closed_date = <Date 2018-05-16.14:54:05.895>
closer = 'ned.deily'
components = ['Library (Lib)']
creation = <Date 2014-05-12.01:35:57.203>
creator = 'rhettinger'
dependencies = ['25497']
files = ['40791']
hgrepos = []
issue_num = 21475
keywords = ['patch', 'easy']
message_count = 14.0
messages = ['218308', '218318', '252528', '253027', '253035', '253063', '255225', '255228', '311416', '315362', '316740', '316743', '316811', '316813']
nosy_count = 7.0
nosy_names = ['rhettinger', 'ned.deily', 'berker.peksag', 'matrixise', 'pwirtz', 'stevensalbert', '[email protected]']
pr_nums = ['6883']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue21475'
versions = ['Python 3.8']
Activity
rhettinger commentedon May 12, 2014
Resources:
http://en.wikipedia.org/wiki/Robots_exclusion_standard#Nonstandard_extensions
https://support.google.com/webmasters/answer/183669?hl=en
https://github.com/seomoz/reppy
berkerpeksag commentedon May 12, 2014
There is a patch for Crawl-delay in bpo-16099.
berkerpeksag commentedon Oct 8, 2015
The Crawl-delay part(bpo-16099) is now committed.
[-]Support the Sitemap and Crawl-delay extensions in robotparser[/-][+]Support the Sitemap extension in robotparser[/+]pwirtz commentedon Oct 15, 2015
I would like to tackle this issue. Should I wait for bpo-25400 to be resolved first?
berkerpeksag commentedon Oct 15, 2015
bpo-25400 is not a blocker of this, so feel free to write a patch.
pwirtz commentedon Oct 15, 2015
Here is a patch that provides support for the Sitemap extension.
matrixise commentedon Nov 23, 2015
Add a test with your patch.
Thank you
berkerpeksag commentedon Nov 23, 2015
Peter didn't write a test because bpo-25497 (test_robotparser rewrite) needs to be committed first. See msg253016 in bpo-25400 for more information.
matrixise commentedon Feb 1, 2018
Hi @berker and @pwirtz.
could you write a test for this issue?
thanks
stevensalbert commentedon Apr 16, 2018
Kindly add a test for this issue
mcscope commentedon May 15, 2018
I wrote a test for this as it seems to have been abandoned, and opened a PR.
#6878
mcscope commentedon May 15, 2018
Sorry, wrong PR number. it is 6883, and attached to this ticket
ned-deily commentedon May 16, 2018
New changeset 5db5c06 by Ned Deily (Christopher Beacham) in branch 'master':
bpo-21475: Support the Sitemap extension in robotparser (GH-6883)
5db5c06
ned-deily commentedon May 16, 2018
Thanks for the patch, Peter, and thanks for the PR and test, Lady Red! Merged for release in 3.8.0.