Skip to content

additional fix for 504 errors #528

Description

@kartiksubbarao

Describe the bug
@justinpolygon I found another place in the code that needs to be updated in order for the new #525 retry logic to work properly. The following line (a remnant of the old code) needs to be removed:

https://github.com/polygon-io/client-python/blob/master/polygon/rest/base.py#L113

This line effectively overrides the new retry_strategy object specified in the urllib3.PoolManager initialization line, so it prevents that logic from taking effect. I discovered this when I backed out my tenacity workaround and got a 504 error last night.

To Reproduce
I found a way to reproduce this problem by monkey-patching the _make_request method in urllib3 to randomly return 504 instead of the actual error code, which forces an exercise of the retry logic. To implement this test, make the following change to polygon/rest/base.py. Right after the import urllib3 line, insert the following code:

original_make_request = urllib3.connectionpool.HTTPConnectionPool._make_request
def monkey_patched_make_request(self, conn, method, url, **kwargs):
    response = original_make_request(self, conn, method, url, **kwargs)
    import random
    if random.random() < 0.90: response.status = 504
    return response
urllib3.connectionpool.HTTPConnectionPool._make_request = monkey_patched_make_request

Here is a sample script to test the library with the above change:

import polygon
from pprint import pprint

polygon_api = polygon.RESTClient(POLYGON_KEY_REDACTED, retries=2, trace=True, verbose=True)

trades = [*polygon_api.list_trades('AAPL', timestamp_gt=1695944980000000000, timestamp_lte=1695945000000000000)]
pprint(trades)

With the current code, running the sample script results in the following kind of intermittent error, which indicates that retries aren't actually happening:

polygon.exceptions.BadResponse: {"results"': [...]}

When I delete line 113 from polygon/rest/base.api, I get a totally different intermittent error:

urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='api.polygon.io', port=443): Max retries exceeded with url: /v3/trades/AAPL?timestamp.lte=1695945000000000000&timestamp.gt=1695944980000000000 (Caused by ResponseError('too many 504 error responses'))

This indicates that the retries are actually happening.

No activity

Activity on this issue will appear here.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions