Python Content API

• A micro framework for content APIs with minimal codebase - less than 1000 lines of Python (see bin/loc)
• Postgresql access with psycopg2 (see db/pg.py)
• MongoDB access with pymongo (see db/mongodb.py)
• Generic CRUD model API that is easy to adapt to Flask or serverless etc. (see model_api.py and models/init.py and example models like models/urls.py and models/users.py)
• Flask CRUD API (a thin wrapper around the model API, see flask_app.py and model_routes.py). There is also support for Bottle in bottle_app.py and Tornado in tornado_app.py. With both Bottle and Tornado I had issues with internal URLs, i.e. where the server would make requests back to itself. Once I changed app_test.py to use external URLs this was resolved.
• Validation with jsonschema (see the validate_schema function in json_schema.py and its usages in request_validation.py, and app_test.py)
• API testing with pytest and the request package (see app_test.py)
• OpenAPI/Swagger documentation generated from model routes (see swagger.py)
• Deployment to Heroku
• Deployment with Zappa to AWS Lambda

TODO:

• Alternative filter param based on JSON data?
• OpenAPI supports parameter default values
• Unique constraint for mongodb
• Add a change log

Some alternatives for building an API like this in Python with popular frameworks:

• Flask + SQLAlchemy + (possibly) OpenAPI-SQLAlchemy
• Django REST Framework

python-heroku-rest-api.herokuapp.com

• If a model doesn't specify a routes attribute then it will get the five default CRUD routes (list, get, create, update, delete) based on the models json_schema and db_schema attributes (those need to be present). For examples see models/fetches.py. If you only want to expose a subset of the CRUD routes for a model you can set the route_names attribute, see models/users.py
• By specifying the routes property for a model you can customize the default CRUD routes, for example to add custom validation, see models/urls.py. You are also free to set any types of routes that you need for the model and the json_schema and db_schema properties are not required in this case. You may for example have a model that uses a different database or no database at all, see models/articles.py. The routes property needs to be a list of dictionaries with the keys method, path, handler, and the optional keys name (name of the route, defaults to the name of handler function), request_schema (JSON schema to validate in request body), response_schema (JSON schema of response body), and parameters (a list of OpenAPI parameters to validate in path/query/header - see models/articles.py). The default CRUD routes are defined in model_routes.py.

A route handler will receive a single argument request dict with these attributes:

• path_params - dict with parameters from the path, such as id for /v1/urls/<id>
• body - dict with body data for POST and PUT requests
• headers - dict with HTTP request headers
• query - dict with query parameters, such as {'page': 2} for /v1/articles?page=2

If you prefer you can use the @named_args decorator to unpack the request dict and have your handler receive the request attributes as named arguments, see models/articles.py.

A route handler returns a response dict with these attributes:

• body - data to be JSON serialized
• status (optional) - HTTP status code (defaults to 200)
• headers (optional) - a dict with HTTP response headers

What's usually referred to as middleware in web frameworks can be achieved by adding Python decorators to a route handler, see for example how this is done in model_api.py and in models/init.py or in this simple example model (notice that the order of decorators potentially matters):

from functools import wraps
import time

def with_headers(response, headers):
  return {**response, 'headers': {**response.get('headers', {}), **headers}}

def timer(handler):
  @wraps(handler)
  def with_timer(request):
    start_time = time.time()
    response = handler(request)
    elapsed = round((time.time() - start_time)*1000, 3)
    return with_headers(response, {'X-Response-Time': f'{elapsed}ms'})
  return with_timer

def cache_header(handler):
  @wraps(handler)
  def with_cache_header(request):
    response = handler(request)
    return with_headers(response, {'Cache-Control': 'max-age=120'})
  return with_cache_header

@timer
@cache_header
def decorators_example(request):
  return {'body': {}}

routes = [
  {
    'path': '/v1/decorators_example',
    'handler': decorators_example
  },
]

Note that the @wraps decorator in the code above is not strictly necessary but its main purpose is to preserve the name of the handler function, i.e. it makes sure that decorators_example.__name__ doesn't change.

Composing decorators is fairly straightforward, see models/composed_decorators_example.py.

Install packages in a virtual env:

python -m venv venv
. venv/bin/activate
pip install -r requirements.txt

Create database:

createdb -U postgres python-rest-api
python -c "import models; models.create_schema()"

Start a Flask server:

bin/start-dev

open http://localhost:5000

Use the FRAMEWORK env variable to start using a different web framework:

FRAMEWORK=bottle bin/start-dev
FRAMEWORK=tornado bin/start-dev

FRAMEWORK=flask bin/test
FRAMEWORK=bottle bin/test
FRAMEWORK=tornado bin/test

To run the API tests against mongodb:

DATABASE=mongodb bin/test

The API tests can be run against the Heroku demo app as well:

BASE_URL=https://python-heroku-rest-api.herokuapp.com pytest -s -vv app_test.py

Interactive HTML docs:

open http://localhost:5000/static/swagger/index.html

OpenAPI specification:

open http://localhost:5000/v1/swagger.json

Below is an example of testing the CRUD operations of the API from the command line using curl and jq (brew install jq):

export BASE_URL=http://localhost:5000

# create with invalid data yields 400
curl -i -H "Content-Type: application/json" -X POST -d '{"url":"http://www.google.com", "foo": 1}' $BASE_URL/v1/urls

# successful create
export URL=$(curl -H "Content-Type: application/json" -X POST -d '{"url":"http://www.google.com"}' $BASE_URL/v1/urls)
export ID=$(echo $URL | jq --raw-output '.id')

# list
curl -i $BASE_URL/v1/urls

# list - pagination
curl -i "$BASE_URL/v1/urls?offset=50&limit=50"

# list - sorting
curl -i "$BASE_URL/v1/urls?sort=created_at"

# list - filtering
curl -gi "$BASE_URL/v1/urls?filter.url=http://www.yahoo.com"
curl -gi "$BASE_URL/v1/urls?filter.url[contains]=652cd7805f3e4182960e7e8a0863e807"
curl -gi "$BASE_URL/v1/urls?filter.created_at[lt]=2020-08-06%2009:31:28.092946"


# get of non-existant id yields 404
curl -i $BASE_URL/v1/urls/12345

# get
curl -i $BASE_URL/v1/urls/$ID

# update of non-existant id yields 404
curl -i -H "Content-Type: application/json" -X PUT -d '{"url":"http://www.yahoo.com"}' $BASE_URL/v1/urls/12345

# update with invalid data yields 400
curl -i -H "Content-Type: application/json" -X PUT -d '{"url":"http://www.yahoo.com", "foo": 1}' $BASE_URL/v1/urls/$ID

# successful update
curl -i -H "Content-Type: application/json" -X PUT -d '{"url":"http://www.yahoo.com"}' $BASE_URL/v1/urls/$ID

# Check the update happened
curl -i $BASE_URL/v1/urls
curl -i $BASE_URL/v1/urls/$ID

# delete of non-existant id yields 404
curl -i -X DELETE $BASE_URL/v1/urls/12345

# successful delete
curl -i -X DELETE $BASE_URL/v1/urls/$ID

# Check the delete happened
curl -i $BASE_URL/v1/urls
curl -i $BASE_URL/v1/urls/$ID

From python:

python
import db.pg as db
from datetime import datetime

# create
db.execute('INSERT INTO urls (url, created_at) VALUES (%s, %s)', ("http://www.aftonbladet.se", datetime.now()))

# list
db.query("select * from urls")

# get
db.query_one("select * from urls where id = %s", [1])

# update
db.execute('UPDATE urls SET url = %s where id = %s', ("http://www.expressen.se", 1))

# delete
db.execute('DELETE from urls where id = %s', [1])

Connecting with psql:

psql -U postgres python-rest-api

delete from urls;

python
import db.mongodb as db
from datetime import datetime

# create
id = db.create('urls', {'url': 'http://www.aftonbladet.se', 'created_at': datetime.now()})

# list
db.find('urls')

# get
url = db.find_one('urls', id)

# update
db.update('urls', id, {**url, 'url': 'http://www.expressen.se'})

# delete
db.delete('urls', id)

Connecting with the Mongo shell:

mongo python-rest-api

db.urls.find()
db.urls.remove({})

Create and activate virtual python env:

python -m venv venv
echo 'venv' > .gitignore
. venv/bin/activate

Add packages and freeze them in requirements.txt:

pip install gunicorn Flask psycopg2 requests
pip freeze > requirements.txt

Create database:

createdb python-heroku-starter

Create script bin/start-dev and basic app.py.

Push files to git:

git add .
git commit -m 'hello world'
git push origin master

Specify Python version and Procfile for Heroku:

python --version # => Python 3.7.7
echo 'python-3.7.7' > runtime.txt
echo 'web gunicorn app:app' > Procfile

Create heroku app:

heroku apps:create --region eu python-rest-api

Deploy:

git push heroku master

Add the heroku-postgresql addon:

heroku addons:create heroku-postgresql:hobby-dev

For MongoDB you can use the mongolab addon:

heroku addons:create mongolab:sandbox

Test the app:

heroku open

Make sure you have an AWS account and set up a user with programmatic access in the AWS console. Add the keys to ~/.aws/credentials:

[default]
aws_access_key_id = ...
aws_secret_access_key = ...

Install Zappa:

pip install zappa
pip freeze > requirements.txt

Recreate the virtual env:

deactivate
rm -rf ./venv
python -m venv venv
. venv/bin/activate
pip install -r requirements.txt

zappa init

The zappa init command will create a zappa_settings.json file like the following (where I think I needed to add the aws_region manually):

{
    "production": {
        "app_function": "app.app",
        "profile_name": "private",
        "aws_region": "eu-north-1",
        "project_name": "python-rest-api",
        "runtime": "python3.8",
        "s3_bucket": "zappa-python-rest-api"
    }
}

Deploy:

zappa deploy production

Zappa error: Status check on the deployed lambda failed, see also Error loading psycopg2 module. NOTE: in order to get Zappa deployment to work I needed to replace the psycopg2 package with psycopg2-binary and "the binary package is a practical choice for development and testing but in production it is advised to use the package built from sources".

Zappa debug logs:

zappa tail

I used the AWS console for lambda to set the DATABASE_URL env variable for the Heroku app.

Issue: the AWS lambda app is deployed at a URL like https://779tuhzuhc.execute-api.eu-north-1.amazonaws.com/production i.e. it is not deployed at the root path but at /production. This breaks the swagger UI.

To re-deploy zappa:

zappa update production

The API tests can be run against the deployed app like so:

BASE_URL=https://779tuhzuhc.execute-api.eu-north-1.amazonaws.com/production python -m pytest -s app_test.py

• Installing Postgres via Brew on Mac
• Python Flask/Auth/Heroku Example app
• Getting Started on Heroku with Python
• Deploying Python Applications with Gunicorn (Heroku)
• Gunicorn homepage
• Specifying a Python Runtime
• Most Popular Python Packages
• Bottle Web Framework (alternative to Flask)
• Swagger UI
• OpenAPI 3 Meta Schema
• ReDoc - API Console feature request
• Redoc.ly Developer Portal (Commercial)
• How to build a REST API with Tornado
• MongoDB Tutorial
• MongoDB Collection Operations

Serverless (AWS Lambda) deployment:

• Chalice - Serverless Python
• AWS Lambda with Python - Simple Example
• AWS Lambda with Python - Getting Started Guide
• The Official Guide to Serverless Flask