Skip to content

robotparser doesn't return crawl delay for default entry #69586

New issue
New issue
Closed
Closed
robotparser doesn't return crawl delay for default entry#69586
Labels
3.7 (EOL)end of lifestdlibStandard Library Python modules in the Lib/ directorytype-bugAn unexpected behavior, bug, or error

Description

@pwirtz
pwirtz
mannequin
opened on Oct 14, 2015
BPO 25400
Nosy @berkerpeksag
PRs
  • [Do Not Merge] Convert Misc/NEWS so that it is managed by towncrier #552
    		•
    Files
  • robotparser_crawl_delay.patch: patch
  • robotparser_crawl_delay_v2.patch
  • issue25400_v2.diff
  • issue25400_v3.diff
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
closed_at = <Date 2016-09-18.17:18:17.950>
created_at = <Date 2015-10-14.01:21:42.690>
labels = ['3.7', 'type-bug', 'library']
title = "robotparser doesn't return crawl delay for default entry"
updated_at = <Date 2017-03-31.16:36:30.705>
user = 'https://bugs.python.org/pwirtz'

    bugs.python.org fields:

    activity = <Date 2017-03-31.16:36:30.705>
actor = 'dstufft'
assignee = 'none'
closed = True
closed_date = <Date 2016-09-18.17:18:17.950>
closer = 'berker.peksag'
components = ['Library (Lib)']
creation = <Date 2015-10-14.01:21:42.690>
creator = 'pwirtz'
dependencies = []
files = ['40777', '40784', '44739', '44740']
hgrepos = []
issue_num = 25400
keywords = ['patch']
message_count = 8.0
messages = ['252971', '252972', '253015', '253016', '253017', '275776', '276897', '276900']
nosy_count = 3.0
nosy_names = ['python-dev', 'berker.peksag', 'pwirtz']
pr_nums = ['552']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue25400'
versions = ['Python 3.6', 'Python 3.7']

    Activity

    1. pwirtz commented on Oct 14, 2015

      @pwirtz
      pwirtzmannequin
      on Oct 14, 2015
      MannequinAuthor

      After changeset http://hg.python.org/lookup/dbed7cacfb7e, calling the crawl_delay method for a robots.txt files that has a crawl-delay for * useragents always returns None.

      Ex:

      Python 3.6.0a0 (default:1aae9b6a6929+, Oct  9 2015, 22:08:05)
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.57)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import urllib.robotparser
>>> parser = urllib.robotparser.RobotFileParser()
>>> parser.set_url('https://www.carthage.edu/robots.txt')
>>> parser.read()
>>> parser.crawl_delay('test_robotparser')
>>> parser.crawl_delay('*')
>>> print(parser.default_entry.delay)
120
>>>

      Excerpt from https://www.carthage.edu/robots.txt:

      User-agent: *
      Crawl-Delay: 120
      Disallow: /cgi-bin

      I have written a patch that solves this. With patch, output is:

      Python 3.6.0a0 (default:1aae9b6a6929+, Oct  9 2015, 22:08:05)
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.57)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import urllib.robotparser
>>> parser = urllib.robotparser.RobotFileParser()
>>> parser.set_url('https://www.carthage.edu/robots.txt')
>>> parser.read()
>>> parser.crawl_delay('test_robotparser')
120
>>> parser.crawl_delay('*')
120
>>> print(parser.default_entry.delay)
120
>>>

      This also applies to the request_rate method.

    2. pwirtz
      added
      stdlibStandard Library Python modules in the Lib/ directory
      type-bugAn unexpected behavior, bug, or error
      on Oct 14, 2015

    3. pwirtz commented on Oct 14, 2015

      @pwirtz
      pwirtzmannequin
      on Oct 14, 2015
      MannequinAuthor

      This fix breaks the unit tests though. I am not sure how to go about checking those as this would be my first contribution to python and an open source project in general.

    4. pwirtz commented on Oct 14, 2015

      @pwirtz
      pwirtzmannequin
      on Oct 14, 2015
      MannequinAuthor

      On further inspection of the tests, it appears that the way in which the tests are written, a test case can only be tested for one useragent at a time. I will attempt to work on the tests so work correctly. Any advice would be much appreciated.

    5. berkerpeksag commented on Oct 14, 2015

      @berkerpeksag
      berkerpeksag
      on Oct 14, 2015
      Member

      Thanks for the patch Peter(and welcome to Python and open source development). I have a WIP patch to rewrite test_robotparser in a less magic way. So we can ignore test failures for now. I'll take a closer look to your patch.

    6. pwirtz commented on Oct 14, 2015

      @pwirtz
      pwirtzmannequin
      on Oct 14, 2015
      MannequinAuthor

      Ok, for the mean time, I reworked the test so it appears to test correctly and tests passes. There does seem to be some magic, so I do hope I did not overlook anything. Here is the new patch.

    7. berkerpeksag commented on Sep 11, 2016

      @berkerpeksag
      berkerpeksag
      on Sep 11, 2016
      Member

      I've now updated Lib/test/test_robotparser.py (bpo-25497) Peter, do you have time to update your patch? Thanks!

    8. berkerpeksag commented on Sep 18, 2016

      @berkerpeksag
      berkerpeksag
      on Sep 18, 2016
      Member

      Here's an updated patch.

    9. berkerpeksag
      added
      3.7 (EOL)end of life
      on Sep 18, 2016

    10. python-dev commented on Sep 18, 2016

      @python-dev
      python-devmannequin
      on Sep 18, 2016
      Mannequin

      New changeset d5d910cfd288 by Berker Peksag in branch '3.6':
      Issue bpo-25400: RobotFileParser now correctly returns default values for crawl_delay and request_rate
      https://hg.python.org/cpython/rev/d5d910cfd288

      New changeset 911070065e38 by Berker Peksag in branch 'default':
      Issue bpo-25400: Merge from 3.6
      https://hg.python.org/cpython/rev/911070065e38

    11. berkerpeksag
      closed this as completedon Sep 18, 2016
    12. ezio-melotti
      transferred this issue fromon Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

    Metadata

    Metadata

    Assignees

    No one assigned

      Labels

      3.7 (EOL)end of lifestdlibStandard Library Python modules in the Lib/ directorytype-bugAn unexpected behavior, bug, or error

      Projects

      No projects

        Milestone

        No milestone

        Relationships

        None yet

        Development

        No branches or pull requests

        Issue actions