|
85 | 85 | run out of threads before we run out of sockets. Per-thread overhead or |
86 | 86 | system limits on threads are the bottleneck. |
87 | 87 |
|
88 | | -In his influential article ``The C10K problem''\footnote{http://www.kegel.com/c10k.html}, |
| 88 | +In his influential article ``The C10K problem''\footnote{\url{http://www.kegel.com/c10k.html}}, |
89 | 89 | Dan Kegel outlines the limitations of multithreading for I/O |
90 | 90 | concurrency. He begins, |
91 | 91 |
|
|
496 | 496 | exception handler for a chain of callbacks, the way a ``try / except'' |
497 | 497 | block wraps a function call and its tree of descendents.\footnote{For a |
498 | 498 | complex solution to this problem, see |
499 | | - http://www.tornadoweb.org/en/stable/stack\_context.html} |
| 499 | + \url{http://www.tornadoweb.org/en/stable/stack_context.html}} |
500 | 500 |
|
501 | 501 | So, even apart from the long debate about the relative efficiencies of |
502 | 502 | multithreading and async, there is this other debate regarding which is |
|
1230 | 1230 | once its work is done. |
1231 | 1231 |
|
1232 | 1232 | Imagine if the workers were threads. How would we express the crawler's |
1233 | | -algorithm? We could use a synchronized queue\footnote{https://docs.python.org/3/library/queue.html} |
| 1233 | +algorithm? We could use a synchronized queue\footnote{\url{https://docs.python.org/3/library/queue.html}} |
1234 | 1234 | from the Python standard library. Each time an item is put in the queue, |
1235 | 1235 | the queue increments its count of ``tasks''. Worker threads call |
1236 | 1236 | \texttt{task\_done} after completing work on an item. The main thread |
1237 | 1237 | blocks on \texttt{Queue.join} until each item put in the queue is |
1238 | 1238 | matched by a \texttt{task\_done} call, then it exits. |
1239 | 1239 |
|
1240 | 1240 | Coroutines use the exact same pattern with a queue from asyncio! First |
1241 | | -we import asyncio's queue\footnote{https://docs.python.org/3/library/asyncio-sync.html}: |
| 1241 | +we import asyncio's queue\footnote{\url{https://docs.python.org/3/library/asyncio-sync.html}}: |
1242 | 1242 |
|
1243 | 1243 | \begin{Shaded} |
1244 | 1244 | \begin{Highlighting}[] |
|
1545 | 1545 | event unpauses \texttt{join} and the main coroutine completes. |
1546 | 1546 |
|
1547 | 1547 | The queue code that coordinates the workers and the main coroutine is |
1548 | | -like this\footnote{The actual asyncio.Queue implementation uses an |
1549 | | - asyncio.Event in place of the Future shown here. The difference is an |
1550 | | - Event can be reset, whereas a Future cannot transition from resolved |
1551 | | - back to pending.}: |
| 1548 | +like this\footnote{The actual \texttt{asyncio.Queue} implementation uses |
| 1549 | + an \texttt{asyncio.Event} in place of the Future shown here. The |
| 1550 | + difference is an Event can be reset, whereas a Future cannot |
| 1551 | + transition from resolved back to pending.}: |
1552 | 1552 |
|
1553 | 1553 | \begin{Shaded} |
1554 | 1554 | \begin{Highlighting}[] |
|
1696 | 1696 | statements, we see they mark points when the coroutine cedes control and |
1697 | 1697 | allows others to run. Unlike threads, coroutines display where our code |
1698 | 1698 | can be interrupted and where it cannot. In his illuminating essay |
1699 | | -``Unyielding''\footnote{https://glyph.twistedmatrix.com/2014/02/unyielding.html}, |
| 1699 | +``Unyielding''\footnote{\url{https://glyph.twistedmatrix.com/2014/02/unyielding.html}}, |
1700 | 1700 | Glyph Lefkowitz writes, ``Threads make local reasoning difficult, and |
1701 | 1701 | local reasoning is perhaps the most important thing in software |
1702 | 1702 | development.'' Explicitly yielding, however, makes it possible to |
|
0 commit comments