<html lang="en"><head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8">

<meta charset="utf-8">

<meta http-equiv="X-UA-Compatible" content="IE=edge">

<meta name="viewport" content="width=device-width, initial-scale=1.0">

<meta name="author" content="Matt Makai">

<meta name="description" content="Full Stack Python explains each layer of the web application stack, from the server up through the rendering in a user's browser.">

<link rel="shortcut icon" href="theme/img/fsp-fav.png">

<title>Task Queues - Full Stack Python</title>

<!--[if lt IE 9]>

<style>

.toc-indent {padding-left: 25px;}

.toc-more-indent {padding-left: 37px;}

</style>

<script type="text/javascript">

var _gaq = _gaq || [];

_gaq.push(['_setAccount', 'UA-19910497-7']);

(function() {

var ga = document.createElement('script'); ga.type = 'text/javascript'; ga.async = true;

ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js';

var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s);

</script> </head>

<a href="https://github.com/makaimc/fullstackpython.github.com"><img style="position: absolute; top: 0; right: 0; border: 0;" src="https://camo.githubusercontent.com/a6677b08c955af8400f44c6298f40e7d19cc5b2d/68747470733a2f2f73332e616d617a6f6e6177732e636f6d2f6769746875622f726962626f6e732f666f726b6d655f72696768745f677261795f3664366436642e706e67" alt="Fork me on GitHub" data-canonical-src="https://s3.amazonaws.com/github/ribbons/forkme_right_gray_6d6d6d.png"></a>

<div class="container">

<div class="row">

<div class="col-md-12">

<div class="logo-header-section">

<a href="/" style="text-decoration: none; border: none;"><img src="theme/img/fsp-logo.png" height="52" width="52" class="logo-image" style="padding-top: 1px;"/></a>

<span class="logo-title"><a href="/">Full Stack Python</a></span>

</div>

</div>

</div> <div class="row">

<div class="col-md-8">

<h1>Task queues</h1>

<p>Task queues manage background work that must be executed outside the usual

HTTP request-response cycle.</p>

<h2>Why are task queues necessary?</h2>

<p>Tasks are handled asynchronously either because they are not initiated by

an HTTP request or because they are long-running jobs that would dramatically

reduce the performance of an HTTP response.</p>

<p>For example, a web application could poll the GitHub API every 10 minutes to

collect the names of the top 100 starred repositories. A task queue would

handle invoking code to call the GitHub API, process the results and store them

in a persistent database for later use.</p>

<p>Another example is when a database query would take too long during the HTTP

request-response cycle. The query could be performed in the background on a

fixed interval with the results stored in the database. When an

HTTP request comes in that needs those results a query would simply fetch the

precalculated result instead of re-executing the longer query.

This precalculation scenario is a form of <a href="/caching.html">caching</a> enabled

by task queues.</p>

<p>Other types of jobs for task queues include</p>

<p>spreading out large numbers of independent database inserts over time

instead of inserting everything at once</p>

<p>aggregating collected data values on a fixed interval, such as every

15 minutes</p>

<p>scheduling periodic jobs such as batch processes</p>

<h2>Task queue projects</h2>

<p>The defacto standard Python task queue is Celery. The other task queue

projects that arise tend to come from the perspective that Celery is overly

complicated for simple use cases. My recommendation is to put the effort into

Celery's reasonable learning curve as it is worth the time it takes to

understand how to use the project.</p>

<p>The <a href="http://www.celeryproject.org/">Celery</a> distributed task queue is the

most commonly used Python library for handling asynchronous tasks and

scheduling.</p>

<p>The <a href="http://python-rq.org/">RQ (Redis Queue)</a> is a simple Python

library for queueing jobs and processing them in the background with workers.

RQ is backed by Redis and is designed to have a low barrier to entry.

The <a href="http://nvie.com/posts/introducing-rq/">intro post</a> contains information

on design decisions and how to use RQ.</p>

<p><a href="https://github.com/dcramer/taskmaster">Taskmaster</a> is a lightweight simple

distributed queue for handling large volumes of one-off tasks. </p>

<p><a href="http://huey.readthedocs.org/en/latest/">Huey</a> is a simple task queue that

uses Redis on the backend but otherwise does not depend on other libraries.

The project was previously known as Invoker and the author changed the name.</p>

<h2>Hosted message and task queue services</h2>

<p>Task queue third party services aim to solve the complexity issues that arise

when scaling out a large deployment of distributed task queues.</p>

<p><a href="http://www.iron.io/">Iron.io</a> is a distributed messaging service platform

that works with many types of task queues such as Celery. It also is built

to work with other IaaS and PaaS environments such as Amazon Web Services

and Heroku.</p>

<p><a href="http://aws.amazon.com/sqs/">Amazon Simple Queue Service (SQS)</a> is a

set of five APIs for creating, sending, receiving, modifying and deleting

messages.</p>

<p><a href="http://www.cloudamqp.com/">CloudAMQP</a> is at its core managed servers with

RabbitMQ installed and configured. This service is an option if you are

using RabbitMQ and do not want to maintain RabbitMQ installations on your

own servers.</p>

<h2>Task queue resources</h2>

<p><a href="http://www.caktusgroup.com/blog/2014/06/23/scheduling-tasks-celery/">Getting Started Scheduling Tasks with Celery</a>

is a detailed walkthrough for setting up Celery with Django (although

Celery can also be used without a problem with other frameworks).</p>

<p><a href="http://justcramer.com/2012/05/04/distributing-work-without-celery/">Distributing work without Celery</a>

provides a scenario in which Celery and RabbitMQ are not the right tool

for scheduling asynchronous jobs.</p>

<p><a href="http://www.warski.org/blog/2014/07/evaluating-persistent-replicated-message-queues/">Evaluating persistent, replicated message queues</a>

is a detailed comparison of Amazon SQS, MongoDB, RabbitMQ, HornetQ and

Kafka's designs and performance.</p>

<p><a href="http://queues.io/">Queues.io</a> is a collection of task queue systems with

short summaries for each one. The task queues are not all compatible with

Python but ones that work with it are tagged with the "Python" keyword.</p>

<p><a href="http://www.slideshare.net/bryanhelmig/task-queues-comorichweb-12962619">Why Task Queues</a>

is a presentation for what task queues are and why they are needed. </p>

<p>Flask by Example <a href="https://realpython.com/blog/python/flask-by-example-implementing-a-redis-task-queue/">Implementing a Redis Task Queue</a>

provides a detailed walkthrough of setting up workers to use RQ with

Redis.</p>

<p><a href="https://www.digitalocean.com/community/articles/how-to-use-celery-with-rabbitmq-to-queue-tasks-on-an-ubuntu-vps">How to use Celery with RabbitMQ</a>

is a detailed walkthrough for using these tools on an Ubuntu VPS.</p>

<p>Heroku has a clear walkthrough for using

<a href="https://devcenter.heroku.com/articles/python-rq">RQ for background tasks</a>.</p>

<p><a href="http://www.linuxforu.com/2013/12/introducing-celery-pythondjango/">Introducing Celery for Python+Django</a>

provides an introduction to the Celery task queue.</p>

<p><a href="https://denibertovic.com/posts/celery-best-practices/">Celery - Best Practices</a>

explains things you should not do with Celery and shows some underused

features for making task queues easier to work with.</p>

<p>The "Django in Production" series by

<a href="https://twitter.com/robgolding63">Rob Golding</a> contains a post

specifically on <a href="http://www.robgolding.com/blog/2011/11/27/django-in-production-part-2---background-tasks/">Background Tasks</a>.</p>

<p><a href="http://blog.thecodepath.com/2012/11/15/asynchronous-processing-in-web-applications-part-1-a-database-is-not-a-queue/">Asynchronous Processing in Web Applications Part One</a>

and <a href="http://blog.thecodepath.com/2013/01/06/asynchronous-processing-in-web-applications-part-2-developers-need-to-understand-message-queues/">Part Two</a>

are great reads for understanding the difference between a task queue and

why you shouldn't use your database as one.</p>

<p><a href="http://www.caktusgroup.com/blog/2014/09/29/celery-production/">Celery in Production</a>

on the Caktus Group blog contains good practices from their experience

using Celery with RabbitMQ, monitoring tools and other aspects not often

discussed in existing documentation.</p>

<p><a href="https://www.youtube.com/watch?v=68QWZU_gCDA">A 4 Minute Intro to Celery</a> is

a short introductory task queue screencast.</p>

<p>Heroku wrote about how to

<a href="https://engineering.heroku.com/blogs/2014-09-15-securing-celery">secure Celery</a>

when tasks are otherwise sent over unencrypted networks.</p>

<p>Miguel Grinberg wrote a nice post on using the

<a href="http://blog.miguelgrinberg.com/post/using-celery-with-flask">task queue Celery with Flask</a>.

He gives an overview of Celery followed by specific code to set up the task

queue and integrate it with Flask.</p>

<h2>Task queue learning checklist</h2>

<p><i class="fa fa-check-square-o"></i>

Pick a slow function in your project that is called during an HTTP request.</p>

<p><i class="fa fa-check-square-o"></i>

Determine if you can precompute the results on a fixed interval instead of

during the HTTP request. If so, create a separate function you can call

from elsewhere then store the precomputed value in the database.</p>

<p><i class="fa fa-check-square-o"></i>

Read the Celery documentation and the links in the resources section below

to understand how the project works.</p>

<p><i class="fa fa-check-square-o"></i>

Install a message broker such as RabbitMQ or Redis and then add Celery to your

project. Configure Celery to work with the installed message broker.</p>

<p><i class="fa fa-check-square-o"></i>

Use Celery to invoke the function from step one on a regular basis.</p>

<p><i class="fa fa-check-square-o"></i>

Have the HTTP request function use the precomputed value instead of the

slow running code it originally relied upon.</p>

<h3>What's next after task queues?</h3>

<div class="row">

<div class="col-md-4">

<div class="well select-next">

<a href="/logging.html" class="btn btn-success btn-full"><i class="fa fa-align-left fa-inverse fa-2x"></i></a>

<p class="under-btn">

How do I monitor my app and its task queues with logging?

</p>

</div>

</div>

<div class="col-md-4">

<div class="well select-next">

<a href="/web-analytics.html" class="btn btn-success btn-full"><i class="fa fa-dashboard fa-2x"></i></a>

</a>

<p class="under-btn">

How can I learn more about the users of my application?

</p>

</div>

</div>

<div class="col-md-4">

<div class="well select-next">

<a href="/monitoring.html" class="btn btn-success btn-full"><i class="fa fa-bar-chart-o fa-inverse fa-2x"></i></a>

<p class="under-btn">

What tools exist for monitoring a live web application?

</p>

</div>

</div>

</div> <div id="mc_embed_signup">

<form action="//mattmakai.us2.list-manage.com/subscribe/post?u=b7e774f0c4f05dcebbfee183d&amp;id=b22335388d" method="post" id="mc-embedded-subscribe-form" name="mc-embedded-subscribe-form" class="validate" target="_blank" novalidate>

<div id="mc_embed_signup_scroll">

<h4>Interested in major updates to this site or an alert when a complete Full Stack Python book with detailed example code is released? Sign up here and you'll an occasional email only when there's big news to report.</h4>

<div class="row">

<div class="col-md-9">

<input type="email" value="" name="EMAIL" class="email form-control" id="mce-EMAIL" placeholder="email address" required>

<div style="position: absolute; left: -5000px;"><input type="text" name="b_b7e774f0c4f05dcebbfee183d_b22335388d" tabindex="-1" value=""></div>

</div>

<div class="col-md-3">

<div class="clear"><input type="submit" value="Subscribe" name="subscribe" id="mc-embedded-subscribe" class="btn btn-success" style="font-family: 'Helvetica Neue';"></div>

</div>

</div>

</div>

</div>

<div class="col-md-offset-1 col-md-3" id="sidebar">

<div class="panel panel-success">

<div class="panel-heading">

<h3 class="panel-head"><a href="/table-of-contents.html" style="color: #fff;">Table of Contents</a></h3>

</div>

<div class="list-group">

<a href="/introduction.html" class="list-group-item smaller-item ">Introduction</a>

<a href="/web-frameworks.html" class="list-group-item smaller-item ">Web Frameworks</a>

<a href="/django.html" class="list-group-item smaller-item ">Django</a>

<a href="/flask.html" class="list-group-item smaller-item ">Flask</a>

<a href="/bottle.html" class="list-group-item smaller-item ">Bottle</a>

<a href="/morepath.html" class="list-group-item smaller-item ">Morepath</a>

<a href="/other-web-frameworks.html" class="list-group-item smaller-item ">Other Web Frameworks</a>

<a href="/deployment.html" class="list-group-item smaller-item ">Deployment</a>

<a href="/servers.html" class="list-group-item smaller-item ">Servers</a>

<a href="/platform-as-a-service.html" class="list-group-item smaller-item ">Platform-as-a-service</a>

<a href="/operating-systems.html" class="list-group-item smaller-item ">Operating Systems</a>

<a href="/web-servers.html" class="list-group-item smaller-item ">Web Servers</a>

<a href="/wsgi-servers.html" class="list-group-item smaller-item ">WSGI Servers</a>

<a href="/source-control.html" class="list-group-item smaller-item ">Source Control</a>

<a href="/application-dependencies.html" class="list-group-item smaller-item ">Application Dependencies</a>

<a href="/databases.html" class="list-group-item smaller-item ">Databases</a>

<a href="/no-sql-datastore.html" class="list-group-item smaller-item ">NoSQL Data Stores</a>

<a href="/web-design.html" class="list-group-item smaller-item ">Web Design</a>

<a href="/cascading-style-sheets.html" class="list-group-item smaller-item ">Cascading Style Sheets</a>

<a href="/javascript.html" class="list-group-item smaller-item ">JavaScript</a>

<a href="/continuous-integration.html" class="list-group-item smaller-item ">Continuous Integration</a>

<a href="/code-metrics.html" class="list-group-item smaller-item ">Code Metrics</a>

<a href="/configuration-management.html" class="list-group-item smaller-item ">Configuration Management</a>

<a href="/static-content.html" class="list-group-item smaller-item ">Static Content</a>

<a href="/caching.html" class="list-group-item smaller-item ">Caching</a>

<a href="/task-queues.html" class="list-group-item smaller-item active">Task Queues</a>

<a href="/application-programming-interfaces.html" class="list-group-item smaller-item ">Application Programming Interfaces</a>

<a href="/api-integration.html" class="list-group-item smaller-item ">API Integration</a>

<a href="/api-creation.html" class="list-group-item smaller-item ">API Creation</a>

<a href="/logging.html" class="list-group-item smaller-item ">Logging</a>

<a href="/monitoring.html" class="list-group-item smaller-item ">Monitoring</a>

<a href="/web-analytics.html" class="list-group-item smaller-item ">Web Analytics</a>

<a href="/web-application-security.html" class="list-group-item smaller-item ">Web Application Security</a>

<a href="/best-python-resources.html" class="list-group-item smaller-item ">Best Python Resources</a>

<a href="/development-environments.html" class="list-group-item smaller-item ">Development Environments</a>

<a href="/about-author.html" class="list-group-item smaller-item ">About the Author</a>

<a href="/change-log.html" class="list-group-item smaller-item ">Change Log</a>

<a href="/future-directions.html" class="list-group-item smaller-item ">Future Directions</a>

<a href="/what-full-stack-means.html" class="list-group-item smaller-item ">What "Full Stack" Means</a>

<a href="/why-use-python.html" class="list-group-item smaller-item ">Why Use Python?</a>

<a href="/websockets.html" class="list-group-item smaller-item ">WebSockets</a>

<a href="/docker.html" class="list-group-item smaller-item ">Docker</a>

</div>

</div>

<div class="panel panel-success">

<div class="panel-heading"><h3 class="panel-head">Task Queues</h3></div>

<div class="panel-body">

<a href="/">Full Stack Python</a> is an open book that explains

each Python web application stack layer and provides the

best web resources for those topics.

<hr/>

There's a work-in-progress

<a href="/full-stack-python-map.pdf">subjects map (.pdf)</a>

that visually lays out each chapter in addition to the table of

contents found below.

<hr/>

Need more detailed tutorials and walkthroughs than you see here?

<a href="/email.html">Sign up for an email alert when that content is created.</a>

<hr/>

<a href="http://twitter.com/mattmakai">Matt Makai</a> built

this site with greatly appreciated assistance from community

<a href="https://github.com/makaimc/fullstackpython.github.com/pulls">pull requests</a>.

</div>

</div>

<hr/>

<div class="footer pull-right">

<a href="http://www.mattmakai.com/" class="underline">Matt Makai</a>

2015

</div>

</div>

<link href="theme/css/f.min.css" rel="stylesheet">

<link href="//netdna.bootstrapcdn.com/font-awesome/4.1.0/css/font-awesome.css" rel="stylesheet">