Commit 2bdbbd3

run('rm {}'.format(f))

chapter_dirs = [

'flow-shop',

'template-engine',

'pedometer',

]

image_paths = [

'./flow-shop/flow-shop-images',

'./pedometer/pedometer-images',

'./sampler/sampler-images',

title: Clustering by Consensus

author: Dustin J. Mitchell

In this chapter, we'll explore implementation of a network protocol designed to support reliable distributed computation.

Network protocols can be difficult to implement correctly, so we'll look at some techniques for minimizing bugs and for catching and fixing the remaining few.

title: A Web Crawler With asyncio Coroutines

author: A. Jesse Jiryu Davis and Guido van Rossum

Classical computer science emphasizes efficient algorithms that complete computations as quickly as possible. But many networked programs spend their time not computing, but holding open many connections that are slow, or have infrequent events. These programs present a very different challenge: to wait for a huge number of network events efficiently. A contemporary approach to this problem is asynchronous I/O, or "async".

This chapter presents a simple web crawler. The crawler is an archetypal async application because it waits for many responses, but does little computation. The more pages it can fetch at once, the sooner it completes. If it devotes a thread to each in-flight request, then as the number of concurrent requests rises it will run out of memory or other thread-related resource before it runs out of sockets. It avoids the need for threads by using asynchronous I/O.

Python 3.4's `DefaultSelector` uses the best `select`-like function available on your system. To register for notifications about network I/O, we create a non-blocking socket and register it with the default selector:

selector = DefaultSelector()

def connected(self, key, mask):

print('connected!')

selector.unregister(key.fd)

self.sock.send(request.encode('ascii'))

# Register the next callback.

The concept of a coroutine, dating to the elder days of computer science, is simple: it is a subroutine that can be paused and resumed. Whereas threads are preemptively multitasked by the operating system, coroutines multitask cooperatively: they choose when to pause, and which coroutine to run next.

To explain Python 3.4's generator-based coroutines, we will engage in an exposition of generators and how they are used as coroutines in asyncio, and trust you will enjoy reading it as much as we enjoyed writing it. Once we have explained generator-based coroutines, we shall use them in our async web crawler.

We collect the workers' shared state in a crawler class, and write the main logic in its `crawl` method. We start `crawl` on a coroutine and run asyncio's event loop until `crawl` finishes:

crawler = crawling.Crawler('http://xkcd.com',

max_redirect=10)

loop.run_until_complete(crawler.crawl())

# aiohttp's ClientSession does connection pooling and

# HTTP keep-alives for us.

# Put (URL, max_redirect) in the queue.

self.q.put((root_url, self.max_redirect))

But when we open our eyes and focus on the `yield from` statements, we see they mark points when the coroutine cedes control and allows others to run. Unlike threads, coroutines display where our code can be interrupted and where it cannot. In his illuminating essay "Unyielding"[^4], Glyph Lefkowitz writes, "Threads make local reasoning difficult, and local reasoning is perhaps the most important thing in software development." Explicitly yielding, however, makes it possible to "understand the behavior (and thereby, the correctness) of a routine by examining the routine itself rather than examining the entire system."

Despite these advances, the core ideas remain. Python's new native coroutines will be syntactically distinct from generators but work very similarly; indeed, they will share an implementation within the Python interpreter. Task, Future, and the event loop will continue to play their roles in asyncio.

[^16]: Guido introduced the standard asyncio library, called "Tulip" then, at PyCon 2013.

</latex>

It is a modest attempt to provide part of the functionality that the Datomic database provides (the main omitted functionality is the durability part).