forked from UWPCE-PythonCert/ProgrammingInPython
-
Notifications
You must be signed in to change notification settings - Fork 0
Expand file tree
/
Copy pathConcurrency.html
More file actions
444 lines (420 loc) · 27.5 KB
/
Copy pathConcurrency.html
File metadata and controls
444 lines (420 loc) · 27.5 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
<!DOCTYPE html>
<html class="writer-html5" lang="en" data-content_root="../">
<head>
<meta charset="utf-8" /><meta name="viewport" content="width=device-width, initial-scale=1" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>Concurrent Programming — Programming in Python 8.0 documentation</title>
<link rel="stylesheet" type="text/css" href="../_static/pygments.css?v=b86133f3" />
<link rel="stylesheet" type="text/css" href="../_static/css/theme.css?v=e59714d7" />
<script src="../_static/jquery.js?v=5d32c60e"></script>
<script src="../_static/_sphinx_javascript_frameworks_compat.js?v=2cd50e6c"></script>
<script src="../_static/documentation_options.js?v=bb3927b2"></script>
<script src="../_static/doctools.js?v=9bcbadda"></script>
<script src="../_static/sphinx_highlight.js?v=dc90522c"></script>
<script src="../_static/js/theme.js"></script>
<link rel="index" title="Index" href="../genindex.html" />
<link rel="search" title="Search" href="../search.html" />
<link rel="next" title="Asychronous Programming" href="Async.html" />
<link rel="prev" title="Graph Databases" href="GraphDatabases.html" />
</head>
<body class="wy-body-for-nav">
<div class="wy-grid-for-nav">
<nav data-toggle="wy-nav-shift" class="wy-nav-side">
<div class="wy-side-scroll">
<div class="wy-side-nav-search" style="background: #4b2e83" >
<a href="../index.html">
</a>
<div role="search">
<form id="rtd-search-form" class="wy-form" action="../search.html" method="get">
<input type="text" name="q" placeholder="Search docs" aria-label="Search docs" />
<input type="hidden" name="check_keywords" value="yes" />
<input type="hidden" name="area" value="default" />
</form>
</div>
</div><div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="Navigation menu">
<p class="caption" role="heading"><span class="caption-text">Topics in the Program</span></p>
<ul class="current">
<li class="toctree-l1"><a class="reference internal" href="../topics/01-setting_up/index.html">1. Setting up your Environment</a></li>
<li class="toctree-l1"><a class="reference internal" href="../topics/02-basic_python/index.html">2. Basic Python</a></li>
<li class="toctree-l1"><a class="reference internal" href="../topics/03-recursion_booleans/index.html">3. Booleans and Recursion</a></li>
<li class="toctree-l1"><a class="reference internal" href="../topics/04-sequences_iteration/index.html">4. Sequences and Iteration</a></li>
<li class="toctree-l1"><a class="reference internal" href="../topics/05-text_handling/index.html">5. Basic Text Handling</a></li>
<li class="toctree-l1"><a class="reference internal" href="../topics/06-exceptions/index.html">6. Exception Handling</a></li>
<li class="toctree-l1"><a class="reference internal" href="../topics/07-unit_testing/index.html">7. Unit Testing</a></li>
<li class="toctree-l1"><a class="reference internal" href="../topics/08-dicts_sets/index.html">8. Dictionaries and Sets</a></li>
<li class="toctree-l1"><a class="reference internal" href="../topics/09-files/index.html">9. File Handling</a></li>
<li class="toctree-l1"><a class="reference internal" href="../topics/10-modules_packages/index.html">10. Modules and Packages</a></li>
<li class="toctree-l1"><a class="reference internal" href="../topics/11-argument_passing/index.html">11. Advanced Argument Passing</a></li>
<li class="toctree-l1"><a class="reference internal" href="../topics/12-comprehensions/index.html">12. Comprehensions</a></li>
<li class="toctree-l1"><a class="reference internal" href="../topics/13-intro_oo/index.html">13. Intro to Object Oriented Programing</a></li>
<li class="toctree-l1"><a class="reference internal" href="../topics/14-magic_methods/index.html">14. Properties and Magic Methods</a></li>
<li class="toctree-l1"><a class="reference internal" href="../topics/15-subclassing/index.html">15. Subclassing and Inheritance</a></li>
<li class="toctree-l1"><a class="reference internal" href="../topics/16-multiple_inheritance/index.html">16. Multiple Inheritance</a></li>
<li class="toctree-l1"><a class="reference internal" href="../topics/17-functional_programming/index.html">17. Introduction to Functional Programming</a></li>
<li class="toctree-l1"><a class="reference internal" href="../topics/18-advanced_testing/index.html">18. Advanced Testing</a></li>
<li class="toctree-l1 current"><a class="reference internal" href="../topics/99-extras/index.html">19. Extra Topics</a><ul class="current">
<li class="toctree-l2"><a class="reference internal" href="Pep8.html">Coding Style and Linting</a></li>
<li class="toctree-l2"><a class="reference internal" href="CodeReviews.html">Code Reviews</a></li>
<li class="toctree-l2"><a class="reference internal" href="PersistanceAndSerialization.html">Persistence and Serialization</a></li>
<li class="toctree-l2"><a class="reference internal" href="Unicode.html">Unicode in Python</a></li>
<li class="toctree-l2"><a class="reference internal" href="IteratorsAndGenerators.html">Iterators and Generators</a></li>
<li class="toctree-l2"><a class="reference internal" href="Decorators.html">Decorators</a></li>
<li class="toctree-l2"><a class="reference internal" href="../exercises/mailroom/mailroom-decorator.html">Mailroom – Decoratoring it</a></li>
<li class="toctree-l2"><a class="reference internal" href="ContextManagers.html">Context Managers</a></li>
<li class="toctree-l2"><a class="reference internal" href="../exercises/context-managers-exercise.html">A Couple Handy Context Managers</a></li>
<li class="toctree-l2"><a class="reference internal" href="MetaProgramming.html">Metaprogramming</a></li>
<li class="toctree-l2"><a class="reference internal" href="../exercises/mailroom/mailroom-meta.html">Mailroom – metaprogramming it!</a></li>
<li class="toctree-l2"><a class="reference internal" href="Logging.html">Logging and the logging module</a></li>
<li class="toctree-l2"><a class="reference internal" href="Debugging.html">Debugging</a></li>
<li class="toctree-l2"><a class="reference internal" href="NoSQL.html">No SQL Databases</a></li>
<li class="toctree-l2"><a class="reference internal" href="GraphDatabases.html">Graph Databases</a></li>
<li class="toctree-l2 current"><a class="current reference internal" href="#">Concurrent Programming</a></li>
<li class="toctree-l2"><a class="reference internal" href="Async.html">Asychronous Programming</a></li>
<li class="toctree-l2"><a class="reference internal" href="Coroutines.html">Notes on Coroutines</a></li>
<li class="toctree-l2"><a class="reference internal" href="ThreadingMultiprocessing.html">Threading and multiprocessing</a></li>
<li class="toctree-l2"><a class="reference internal" href="../exercises/threaded_downloader.html">Threaded Web Scraper</a></li>
<li class="toctree-l2"><a class="reference internal" href="Profiling.html">Performance and Profiling</a></li>
</ul>
</li>
</ul>
</div>
</div>
</nav>
<section data-toggle="wy-nav-shift" class="wy-nav-content-wrap"><nav class="wy-nav-top" aria-label="Mobile navigation menu" style="background: #4b2e83" >
<i data-toggle="wy-nav-top" class="fa fa-bars"></i>
<a href="../index.html">Programming in Python</a>
</nav>
<div class="wy-nav-content">
<div class="rst-content style-external-links">
<div role="navigation" aria-label="Page navigation">
<ul class="wy-breadcrumbs">
<li><a href="../index.html" class="icon icon-home" aria-label="Home"></a></li>
<li class="breadcrumb-item"><a href="../topics/99-extras/index.html"><span class="section-number">19. </span>Extra Topics</a></li>
<li class="breadcrumb-item active">Concurrent Programming</li>
<li class="wy-breadcrumbs-aside">
<a href="../_sources/modules/Concurrency.rst.txt" rel="nofollow"> View page source</a>
</li>
</ul><div class="rst-breadcrumbs-buttons" role="navigation" aria-label="Sequential page navigation">
<a href="GraphDatabases.html" class="btn btn-neutral float-left" title="Graph Databases" accesskey="p"><span class="fa fa-arrow-circle-left" aria-hidden="true"></span> Previous</a>
<a href="Async.html" class="btn btn-neutral float-right" title="Asychronous Programming" accesskey="n">Next <span class="fa fa-arrow-circle-right" aria-hidden="true"></span></a>
</div>
<hr/>
</div>
<div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
<div itemprop="articleBody">
<section id="concurrent-programming">
<span id="concurrency"></span><h1>Concurrent Programming<a class="headerlink" href="#concurrent-programming" title="Link to this heading"></a></h1>
<p>What does it mean to do something “Concurrently” ? It means multiple things are happening at the same time. But what are those “things”?</p>
<blockquote>
<div><ul class="simple">
<li><p>Parallelism is about processing multiple things at the same time – true parallelism requires multiple processors (or cores).</p></li>
<li><p>Concurrency is about handling multiple things at the same time – which may or may not be actually running in the processor at the same time (like network requests for instance).</p></li>
<li><p>Parallelism needs concurrency, but concurrency need not be in parallel.</p></li>
</ul>
</div></blockquote>
<section id="whirlwind-tour-of-concurrency">
<h2>Whirlwind Tour of Concurrency<a class="headerlink" href="#whirlwind-tour-of-concurrency" title="Link to this heading"></a></h2>
<p><strong>Concurrency:</strong></p>
<p>Having different code running at the same time, or kind of the same time.</p>
<p><strong>Asynchrony:</strong></p>
<p>The occurrence of events independent of the main program flow and ways to deal with such events.</p>
<p>Asynchrony and Concurrency are really two different things – you can do either one without the other – but they are closely related, and often used together. They solve different problems, but the problems and the solutions overlap.</p>
<p>“Concurrency is not parallelism” – Rob Pike: <a class="reference external" href="https://vimeo.com/49718712">https://vimeo.com/49718712</a></p>
<p>Despite Rob Pike using an example about burning books, I recommend listening to at least the first half of his talk.</p>
<p>In that talk Rob Pike makes a key point: Breaking down tasks into concurrent subtasks only allows parallelism, it’s the scheduling of these subtasks that creates it.</p>
<p>And, indeed, once you have a set of subtasks, they can be scheduled in a truly parallel fashion, or managed asynchronously in a single thread (concurrent, but not parallel)</p>
<section id="types-of-concurrency">
<h3>Types of Concurrency<a class="headerlink" href="#types-of-concurrency" title="Link to this heading"></a></h3>
<p><strong>Multithreading:</strong></p>
<blockquote>
<div><p>Multiple code paths sharing memory – one Python interpreter, one set of Python objects.</p>
</div></blockquote>
<p><strong>Multiprocessing:</strong></p>
<blockquote>
<div><p>Multiple code paths with separate memory space – completely separate Python interpreter.</p>
</div></blockquote>
<p><strong>Asyncronous programming:</strong></p>
<blockquote>
<div><p>Multiple “jobs” run at “arbitrary” times – but usually in one thread – i.e. only one code path, one interpreter.</p>
</div></blockquote>
<p>Lots of different packages for both in the standard library and 3rd party libraries.</p>
<p>How to know what to choose?</p>
<blockquote>
<div><ul class="simple">
<li><p>IO bound vs. CPU bound – CPU bound requires multiprocessing (at least with pure Python)</p></li>
<li><p>Event driven cooperative multitasking vs. preemptive multitasking</p></li>
<li><p>Callbacks vs coroutines + scheduler/event loop</p></li>
</ul>
</div></blockquote>
</section>
<section id="motivations-for-parallel-execution">
<h3>Motivations for parallel execution<a class="headerlink" href="#motivations-for-parallel-execution" title="Link to this heading"></a></h3>
<ul class="simple">
<li><p>Performance
- Limited by “Amdahl’s Law”: <a class="reference external" href="http://en.wikipedia.org/wiki/Amdahl%27s_law">http://en.wikipedia.org/wiki/Amdahl%27s_law</a></p>
<ul>
<li><p>CPUs aren’t getting much faster</p></li>
</ul>
</li>
<li><p>Event handling
- If a system handles asynchronous events, a separate thread of execution could handle those events and let other threads do other work</p>
<ul>
<li><dl class="simple">
<dt>Examples:</dt><dd><ul>
<li><p>Network applications</p></li>
<li><p>User interfaces</p></li>
</ul>
</dd>
</dl>
</li>
</ul>
</li>
</ul>
<p>Parallel programming can be hard!</p>
<p>If your problem can be solved sequentially, consider the costs and
benefits before going parallel.</p>
</section>
<section id="parallelization-strategy-for-performance">
<h3>Parallelization Strategy for Performance<a class="headerlink" href="#parallelization-strategy-for-performance" title="Link to this heading"></a></h3>
<div class="line-block">
<div class="line">1. Break problem down into chunks</div>
<div class="line">2. Execute chunks in parallel</div>
<div class="line">3. Reassemble output of chunks into result</div>
</div>
<a class="reference internal image-reference" href="../_images/OPP.0108.gif"><img alt="multitasking flow diagram" class="align-right" src="../_images/OPP.0108.gif" style="height: 450px;" />
</a>
<div class="line-block">
<div class="line"><br /></div>
<div class="line"><br /></div>
</div>
<ul class="simple">
<li><p>Not every problem is parallelizable</p></li>
<li><p>There is an optimal number of threads for each problem in each
environment, so make it tunable</p></li>
<li><p>Working concurrently opens up synchronization issues</p></li>
<li><p>Methods for synchronizing threads:</p>
<ul>
<li><p>locks</p></li>
<li><p>queues</p></li>
<li><p>signaling/messaging mechanisms</p></li>
</ul>
</li>
</ul>
</section>
<section id="other-options">
<h3>Other options<a class="headerlink" href="#other-options" title="Link to this heading"></a></h3>
<p>Traditionally, concurency has been achieved through multiple process
communication and in-process threads, as we’ve seen.</p>
<p>Another strategy is through micro-threads, implemented via coroutines
and a scheduler.</p>
<p>A coroutine is a generalization of a subroutine which allows multiple
entry points for suspending and resuming execution.</p>
<p>The threading and the multiprocessing modules follow a
<a class="reference external" href="http://en.wikipedia.org/wiki/Preemption_(computing)">preemptive multitasking model</a></p>
<p>Coroutine based solutions follow a
<a class="reference external" href="http://en.wikipedia.org/wiki/Computer_multitasking#Cooperative_multitasking.2Ftime-sharing">cooperative multitasking model:</a></p>
</section>
<section id="threads-versus-processes-in-python">
<h3>Threads versus processes in Python<a class="headerlink" href="#threads-versus-processes-in-python" title="Link to this heading"></a></h3>
<p>Threads are lightweight <a class="reference external" href="https://en.wikipedia.org/wiki/Light-weight_process">processes</a>, run in the address space of an OS
process, true OS level threads.</p>
<p>Therefore, a component of a process.</p>
<p>This allows multiple threads access to data in the same scope.</p>
<p>Threads can not gain the performance advantage of multiple processors
due to the Global Interpreter Lock (GIL)</p>
<p>But the GIL is released during IO, allowing IO bound processes to
benefit from threading</p>
</section>
<section id="id1">
<h3>Processes<a class="headerlink" href="#id1" title="Link to this heading"></a></h3>
<p>A process contains all the instructions and data required to execute
independently, so processes do not share data!</p>
<p>Mulitple processes best to speed up CPU bound operations.</p>
<p>The Python interpreter isn’t lightweight!</p>
<p>Communication between processes can be achieved via:</p>
<p><code class="docutils literal notranslate"><span class="pre">multiprocessing.Queue</span></code></p>
<p><code class="docutils literal notranslate"><span class="pre">multiprocessing.Pipe</span></code></p>
<p>and regular IPC (inter-process communication)</p>
<p>Data moved between processes must be pickleable</p>
</section>
<section id="advantages-disadvantages-of-threads">
<h3>Advantages / Disadvantages of Threads<a class="headerlink" href="#advantages-disadvantages-of-threads" title="Link to this heading"></a></h3>
<section id="advantages">
<h4>Advantages:<a class="headerlink" href="#advantages" title="Link to this heading"></a></h4>
<p>They share memory space:</p>
<blockquote>
<div><ul class="simple">
<li><p>Threads are relatively lightweight – shared memory means they can be created fairly quickly without much memory use.</p></li>
<li><p>Easy and cheap to pass data around (you are only passing a reference).</p></li>
</ul>
</div></blockquote>
</section>
<section id="disadvantages">
<h4>Disadvantages:<a class="headerlink" href="#disadvantages" title="Link to this heading"></a></h4>
<p>They share memory space:</p>
<blockquote>
<div><ul>
<li><p>Each thread is working with the <em>same</em> python objects.</p></li>
<li><p>Operations often take several steps and may be interrupted mid-stream</p></li>
<li><p>Thus, access to shared data is also non-deterministic</p>
<p>(race conditions)</p>
</li>
</ul>
</div></blockquote>
<p>Creating threads is easy, but programming with threads is difficult.</p>
<blockquote>
<div><p>Q: Why did the multithreaded chicken cross the road?</p>
<p>A: to To other side. get the</p>
<p class="attribution">—Jason Whittington</p>
</div></blockquote>
</section>
</section>
<section id="gil">
<h3>GIL<a class="headerlink" href="#gil" title="Link to this heading"></a></h3>
<p><strong>Global Interpreter Lock</strong></p>
<p>(<strong>GIL</strong>)</p>
<p>This is a lock which must be obtained by each thread before it can
execute, ensuring thread safety</p>
<a class="reference internal image-reference" href="../_images/gil.png"><img alt="../_images/gil.png" src="../_images/gil.png" style="width: 100.0%;" />
</a>
<p>The GIL is released during IO operations, so threads which spend time
waiting on network or disk access can enjoy performance gains</p>
<p>The GIL is not unlike multitasking in humans, some things can truly be
done in parallel, others have to be done by time slicing.</p>
<p>Note that potentially blocking or long-running operations, such as I/O, image processing, and NumPy number crunching, happen outside the GIL. Therefore it is only in multithreaded programs that spend a lot of time inside the GIL, interpreting CPython bytecode, that the GIL becomes a bottleneck. But: it can still cause performance degradation.</p>
<p>Not only will threads not help cpu-bound problems, but it can actually make things <em>worse</em>, especially on multi-core machines!</p>
<p>Python threads do not work well for computationally intensive work.</p>
<p>Python threads work well if the threads are spending time waiting for something:</p>
<blockquote>
<div><ul class="simple">
<li><p>Database Access</p></li>
<li><p>Network Access</p></li>
<li><p>File I/O</p></li>
</ul>
</div></blockquote>
<p>Some alternative Python implementations such as Jython and IronPython
have no GIL</p>
<p>cPython and PyPy have one</p>
<p>More about the gil</p>
<p>More on the GIL:</p>
<p><a class="reference external" href="https://emptysqua.re/blog/grok-the-gil-fast-thread-safe-python/">https://emptysqua.re/blog/grok-the-gil-fast-thread-safe-python/</a></p>
<p>If you really want to understand the GIL – and get blown away – watch this one:</p>
<p><a class="reference external" href="http://pyvideo.org/pycon-us-2010/pycon-2010--understanding-the-python-gil---82.html">http://pyvideo.org/pycon-us-2010/pycon-2010–understanding-the-python-gil—82.html</a></p>
<ul class="simple">
<li><p><a class="reference external" href="http://wiki.python.org/moin/GlobalInterpreterLock">http://wiki.python.org/moin/GlobalInterpreterLock</a></p></li>
<li><p><a class="reference external" href="https://docs.python.org/3/c-api/init.html#threads">https://docs.python.org/3/c-api/init.html#threads</a></p></li>
<li><p><a class="reference external" href="http://hg.python.org/cpython/file/05e8dde3229c/Python/pystate.c#l761">http://hg.python.org/cpython/file/05e8dde3229c/Python/pystate.c#l761</a></p></li>
</ul>
<p><strong>NOTE:</strong> The GIL <em>seems</em> like such an obvious limitation that you’ve got to wonder why it’s there. And there have been multiple efforts to remove it. But it turns out that Python’s design makes that very hard (impossible?) without severely reducing performance on single threaded programs.</p>
<p>The current “Best” effort is Larry Hastings’ <a class="reference external" href="https://speakerdeck.com/pycon2017/larry-hastings-the-gilectomy-hows-it-going">gilectomy</a></p>
<p>But that may be stalled out at this point, too. No one should count on it going away in cPython.</p>
<p>But: <strong>Personal Opinion:</strong> Python is not really (directly) suited to the kind of computationally intensive work that the GIL really hampers. And extension modules (i.e. numpy) can release the GIL!</p>
</section>
<section id="posted-without-comment">
<h3>Posted without comment<a class="headerlink" href="#posted-without-comment" title="Link to this heading"></a></h3>
<figure class="align-default">
<img alt="../_images/killGIL.jpg" class="fill" src="../_images/killGIL.jpg" />
</figure>
</section>
<section id="advantages-disadvantages-of-processes">
<h3>Advantages / Disadvantages of Processes<a class="headerlink" href="#advantages-disadvantages-of-processes" title="Link to this heading"></a></h3>
<p>Processes are heavier weight – each process makes a copy of the entire interpreter (Mostly…) – uses more resources.</p>
<p>You need to copy the data you need back and forth between processes.</p>
<p>Slower to start, slower to use, more memory.</p>
<p>But as the entire python process is copied, each subprocess is working with the different objects – they can’t step on each other. So there is:</p>
<blockquote>
<div><p><strong>no GIL</strong></p>
</div></blockquote>
<p>Multiprocessing is suitable for computationally intensive work.</p>
<p>Works best for “large” problems with not much data to pass back and forth, as that’s what’s expensive.</p>
<p>Note that there are ways to share memory between processes, if you have a lot of read-only data that needs to be used. (see <a class="reference external" href="https://docs.python.org/3/library/mmap.html">Memory Maps</a>)</p>
<p>Synchronization options:</p>
<blockquote>
<div><ul class="simple">
<li><p>Locks (Mutex: mutual exclusion, Rlock: reentrant lock)</p></li>
<li><p>Semaphore</p></li>
<li><p>BoundedSemaphore</p></li>
<li><p>Event</p></li>
<li><p>Condition</p></li>
<li><p>Queues</p></li>
</ul>
</div></blockquote>
</section>
<section id="mutex-locks-threading-lock">
<h3>Mutex locks (<code class="docutils literal notranslate"><span class="pre">threading.Lock</span></code>)<a class="headerlink" href="#mutex-locks-threading-lock" title="Link to this heading"></a></h3>
<blockquote>
<div><ul class="simple">
<li><p>Probably most common</p></li>
<li><p>Only one thread can modify shared data at any given time</p></li>
<li><p>Thread determines when unlocked</p></li>
<li><p>Must put lock/unlock around critical code in ALL threads</p></li>
<li><p>Difficult to manage</p></li>
</ul>
</div></blockquote>
<p>Easiest with context manager:</p>
<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="n">x</span> <span class="o">=</span> <span class="mi">0</span>
<span class="n">x_lock</span> <span class="o">=</span> <span class="n">threading</span><span class="o">.</span><span class="n">Lock</span><span class="p">()</span>
<span class="c1"># Example critical section</span>
<span class="k">with</span> <span class="n">x_lock</span><span class="p">:</span>
<span class="c1"># statements using x</span>
</pre></div>
</div>
<p>Only one lock per thread! (or risk mysterious deadlocks)</p>
<p>Or use RLock for code-based locking (locking function/method execution rather than data access)</p>
</section>
<section id="subprocesses-subprocess">
<h3>Subprocesses (<code class="docutils literal notranslate"><span class="pre">subprocess</span></code>)<a class="headerlink" href="#subprocesses-subprocess" title="Link to this heading"></a></h3>
<p>Subprocesses are completely separate processes invoked from a master process (your python program).</p>
<p>Usually used to call non-python programs (shell commands). But of course, a Python program can be a command line program as well, so you can call either your or other python programs this way.</p>
<p>Easy invocation:</p>
<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span><span class="w"> </span><span class="nn">subprocess</span>
<span class="n">subprocess</span><span class="o">.</span><span class="n">run</span><span class="p">(</span><span class="s1">'ls'</span><span class="p">)</span>
</pre></div>
</div>
<p>The program halts while waiting for the subprocess to finish. (unless you call it from a thread!)</p>
<p>You can control communication with the subprocess via:</p>
<p><code class="docutils literal notranslate"><span class="pre">stdout</span></code>, <code class="docutils literal notranslate"><span class="pre">stdin</span></code>, <code class="docutils literal notranslate"><span class="pre">stderr</span></code> with:</p>
<p><code class="docutils literal notranslate"><span class="pre">subprocess.Popen</span></code></p>
<p>Lots of options there!</p>
<section id="pipes-and-pickle-and-subprocess">
<h4>Pipes and <code class="docutils literal notranslate"><span class="pre">pickle</span></code> and <code class="docutils literal notranslate"><span class="pre">subprocess</span></code><a class="headerlink" href="#pipes-and-pickle-and-subprocess" title="Link to this heading"></a></h4>
<blockquote>
<div><ul class="simple">
<li><p>Very low level, for the brave of heart</p></li>
<li><p>Can send just about any Python object</p></li>
</ul>
</div></blockquote>
<p>For this to work, you need to send messages, as each process runs its own independent Python interpreter.</p>
</section>
</section>
</section>
<section id="when-to-use-what">
<h2>When to Use What<a class="headerlink" href="#when-to-use-what" title="Link to this heading"></a></h2>
<img alt="../_images/proc_thread_async.png" src="../_images/proc_thread_async.png" />
</section>
</section>
</div>
</div>
<footer><div class="rst-footer-buttons" role="navigation" aria-label="Footer">
<a href="GraphDatabases.html" class="btn btn-neutral float-left" title="Graph Databases" accesskey="p" rel="prev"><span class="fa fa-arrow-circle-left" aria-hidden="true"></span> Previous</a>
<a href="Async.html" class="btn btn-neutral float-right" title="Asychronous Programming" accesskey="n" rel="next">Next <span class="fa fa-arrow-circle-right" aria-hidden="true"></span></a>
</div>
<hr/>
<div role="contentinfo">
<p>© Copyright 2025, Natasha Aleksandrova, Christopher Barker, Brian Dorsey, Cris Ewing, Christy Heaton, Jon Jacky, Maria McKinley, Andy Miles, Rick Riehle, Joseph Schilz, Joseph Sheedy, Hosung Song. Creative Commons Attribution-ShareAlike 4.0 license.</p>
</div>
Built with <a href="https://www.sphinx-doc.org/">Sphinx</a> using a
<a href="https://github.com/readthedocs/sphinx_rtd_theme">theme</a>
provided by <a href="https://readthedocs.org">Read the Docs</a>.
</footer>
</div>
</div>
</section>
</div>
<script>
jQuery(function () {
SphinxRtdTheme.Navigation.enable(true);
});
</script>
</body>
</html>