Skip to content

Pace requests to avoid rate limits #8

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
philipithomas merged 5 commits into from
Apr 2, 2016
Merged

Pace requests to avoid rate limits #8

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
360e533
Prevent rate limiting
philipithomas Apr 2, 2016
4b197d3
This is why we have testing
philipithomas Apr 2, 2016
a75fcbe
version bump
philipithomas Apr 2, 2016
db606c3
Fix for class methods
philipithomas Apr 2, 2016
87f0e81
oops - some are instantiated
philipithomas Apr 2, 2016
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,10 @@ Authentication keys are currently tied to an individual user's account. To issue

To get your organization ID, look at the URL path when you go to the Manager app while logged in.

## Rate Limits

This client sleeps for .5 seconds after every request. Thus, in a single thread, requests are limited to 120 per minute. This is done to avoid rate limiting. Staffjoy's API currently rate limits to 300 requests per second across keys and IPs. Thus, by using this library, you should never encounter a rate limit (assuming one executing thread per IP address).

## Updates

If you use this library, please subscribe to the [Staffjoy API Updates Google Group](https://groups.google.com/forum/#!forum/staffjoy-api-updates) for important notifications about changes and deprecations.
Expand Down Expand Up @@ -62,4 +66,3 @@ loc.delete()
```



2 changes: 1 addition & 1 deletion setup.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
from setuptools import setup, find_packages

version = "0.13"
version = "0.14"
setup(name="staffjoy",
packages=find_packages(),
version=version,
Expand Down
1 change: 1 addition & 0 deletions staffjoy/config.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ class DefaultConfig:
ENV = "prod"
LOG_LEVEL = logging.INFO
BASE = "https://www.staffjoy.com/api/v2/"
REQUEST_SLEEP = 0.5


class StageConfig(DefaultConfig):
Expand Down
12 changes: 12 additions & 0 deletions staffjoy/resource.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,11 +1,15 @@
import requests
import time
from copy import copy

from .config import config_from_env
from .exceptions import UnauthorizedException, NotFoundException, BadRequestException


class Resource:
# Seconds to sleep between requests (bc of rate limits)
REQUEST_SLEEP = 0.5
Copy link
Copy Markdown
Contributor

@andhess andhess Apr 3, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i feel like this can be brought down to 0.1 second?


PATH = "" # URL path added to base, including route variables
ID_NAME = None # What is this ID called in the route of children?
META_ENVELOPES = [] # Metadata keys for what to unpack from response
Expand Down Expand Up @@ -70,6 +74,7 @@ def get_all(cls, parent=None, **params):
r = requests.get(base_obj._url(),
auth=(base_obj.key, ""),
params=params)
time.sleep(cls.REQUEST_SLEEP)

if r.status_code not in cls.TRUTHY_CODES:
return base_obj._handle_request_exception(r)
Expand Down Expand Up @@ -117,6 +122,8 @@ def _handle_request_exception(request):
def fetch(self):
"""Perform a read request against the resource"""
r = requests.get(self._url(), auth=(self.key, ""))
time.sleep(self.REQUEST_SLEEP)

if r.status_code not in self.TRUTHY_CODES:
return self._handle_request_exception(r)

Expand All @@ -138,12 +145,16 @@ def delete(self):
"""Delete the object"""

r = requests.delete(self._url(), auth=(self.key, ""))
time.sleep(self.REQUEST_SLEEP)

if r.status_code not in self.TRUTHY_CODES:
return self._handle_request_exception(r)

def patch(self, **kwargs):
"""Change attributes of the item"""
r = requests.patch(self._url(), auth=(self.key, ""), data=kwargs)
time.sleep(self.REQUEST_SLEEP)

if r.status_code not in self.TRUTHY_CODES:
return self._handle_request_exception(r)

Expand All @@ -165,6 +176,7 @@ def create(cls, parent=None, **kwargs):
obj = cls(key=parent.key, route=route, config=parent.config)

response = requests.post(obj._url(), auth=(obj.key, ""), data=kwargs)
time.sleep(cls.REQUEST_SLEEP)

if response.status_code not in cls.TRUTHY_CODES:
return cls._handle_request_exception(response)
Expand Down