-
Notifications
You must be signed in to change notification settings - Fork 1
Expand file tree
/
Copy pathdebugging.html
More file actions
322 lines (276 loc) · 15.2 KB
/
Copy pathdebugging.html
File metadata and controls
322 lines (276 loc) · 15.2 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
<!DOCTYPE HTML>
<html lang="en" class="light sidebar-visible" dir="ltr">
<head>
<!-- Book generated using mdBook -->
<meta charset="UTF-8">
<title>Debugging - The Shadow Simulator</title>
<!-- Custom HTML head -->
<meta name="description" content="">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="theme-color" content="#ffffff">
<link rel="icon" href="favicon.svg">
<link rel="shortcut icon" href="favicon.png">
<link rel="stylesheet" href="css/variables.css">
<link rel="stylesheet" href="css/general.css">
<link rel="stylesheet" href="css/chrome.css">
<link rel="stylesheet" href="css/print.css" media="print">
<!-- Fonts -->
<link rel="stylesheet" href="FontAwesome/css/font-awesome.css">
<link rel="stylesheet" href="fonts/fonts.css">
<!-- Highlight.js Stylesheets -->
<link rel="stylesheet" id="highlight-css" href="highlight.css">
<link rel="stylesheet" id="tomorrow-night-css" href="tomorrow-night.css">
<link rel="stylesheet" id="ayu-highlight-css" href="ayu-highlight.css">
<!-- Custom theme stylesheets -->
<link rel="stylesheet" href="custom.css">
<!-- Provide site root and default themes to javascript -->
<script>
const path_to_root = "";
const default_light_theme = "light";
const default_dark_theme = "navy";
window.path_to_searchindex_js = "searchindex.js";
</script>
<!-- Start loading toc.js asap -->
<script src="toc.js"></script>
</head>
<body>
<div id="mdbook-help-container">
<div id="mdbook-help-popup">
<h2 class="mdbook-help-title">Keyboard shortcuts</h2>
<div>
<p>Press <kbd>←</kbd> or <kbd>→</kbd> to navigate between chapters</p>
<p>Press <kbd>S</kbd> or <kbd>/</kbd> to search in the book</p>
<p>Press <kbd>?</kbd> to show this help</p>
<p>Press <kbd>Esc</kbd> to hide this help</p>
</div>
</div>
</div>
<div id="body-container">
<!-- Work around some values being stored in localStorage wrapped in quotes -->
<script>
try {
let theme = localStorage.getItem('mdbook-theme');
let sidebar = localStorage.getItem('mdbook-sidebar');
if (theme.startsWith('"') && theme.endsWith('"')) {
localStorage.setItem('mdbook-theme', theme.slice(1, theme.length - 1));
}
if (sidebar.startsWith('"') && sidebar.endsWith('"')) {
localStorage.setItem('mdbook-sidebar', sidebar.slice(1, sidebar.length - 1));
}
} catch (e) { }
</script>
<!-- Set the theme before any content is loaded, prevents flash -->
<script>
const default_theme = window.matchMedia("(prefers-color-scheme: dark)").matches ? default_dark_theme : default_light_theme;
let theme;
try { theme = localStorage.getItem('mdbook-theme'); } catch(e) { }
if (theme === null || theme === undefined) { theme = default_theme; }
const html = document.documentElement;
html.classList.remove('light')
html.classList.add(theme);
html.classList.add("js");
</script>
<input type="checkbox" id="sidebar-toggle-anchor" class="hidden">
<!-- Hide / unhide sidebar before it is displayed -->
<script>
let sidebar = null;
const sidebar_toggle = document.getElementById("sidebar-toggle-anchor");
if (document.body.clientWidth >= 1080) {
try { sidebar = localStorage.getItem('mdbook-sidebar'); } catch(e) { }
sidebar = sidebar || 'visible';
} else {
sidebar = 'hidden';
sidebar_toggle.checked = false;
}
if (sidebar === 'visible') {
sidebar_toggle.checked = true;
} else {
html.classList.remove('sidebar-visible');
}
</script>
<nav id="sidebar" class="sidebar" aria-label="Table of contents">
<!-- populated by js -->
<mdbook-sidebar-scrollbox class="sidebar-scrollbox"></mdbook-sidebar-scrollbox>
<noscript>
<iframe class="sidebar-iframe-outer" src="toc.html"></iframe>
</noscript>
<div id="sidebar-resize-handle" class="sidebar-resize-handle">
<div class="sidebar-resize-indicator"></div>
</div>
</nav>
<div id="page-wrapper" class="page-wrapper">
<div class="page">
<div id="menu-bar-hover-placeholder"></div>
<div id="menu-bar" class="menu-bar sticky">
<div class="left-buttons">
<label id="sidebar-toggle" class="icon-button" for="sidebar-toggle-anchor" title="Toggle Table of Contents" aria-label="Toggle Table of Contents" aria-controls="sidebar">
<i class="fa fa-bars"></i>
</label>
<button id="theme-toggle" class="icon-button" type="button" title="Change theme" aria-label="Change theme" aria-haspopup="true" aria-expanded="false" aria-controls="theme-list">
<i class="fa fa-paint-brush"></i>
</button>
<ul id="theme-list" class="theme-popup" aria-label="Themes" role="menu">
<li role="none"><button role="menuitem" class="theme" id="default_theme">Auto</button></li>
<li role="none"><button role="menuitem" class="theme" id="light">Light</button></li>
<li role="none"><button role="menuitem" class="theme" id="rust">Rust</button></li>
<li role="none"><button role="menuitem" class="theme" id="coal">Coal</button></li>
<li role="none"><button role="menuitem" class="theme" id="navy">Navy</button></li>
<li role="none"><button role="menuitem" class="theme" id="ayu">Ayu</button></li>
</ul>
<button id="search-toggle" class="icon-button" type="button" title="Search (`/`)" aria-label="Toggle Searchbar" aria-expanded="false" aria-keyshortcuts="/ s" aria-controls="searchbar">
<i class="fa fa-search"></i>
</button>
</div>
<h1 class="menu-title">The Shadow Simulator</h1>
<div class="right-buttons">
<a href="print.html" title="Print this book" aria-label="Print this book">
<i id="print-button" class="fa fa-print"></i>
</a>
<a href="https://github.com/shadow/shadow" title="Git repository" aria-label="Git repository">
<i id="git-repository-button" class="fa fa-github"></i>
</a>
</div>
</div>
<div id="search-wrapper" class="hidden">
<form id="searchbar-outer" class="searchbar-outer">
<div class="search-wrapper">
<input type="search" id="searchbar" name="searchbar" placeholder="Search this book ..." aria-controls="searchresults-outer" aria-describedby="searchresults-header">
<div class="spinner-wrapper">
<i class="fa fa-spinner fa-spin"></i>
</div>
</div>
</form>
<div id="searchresults-outer" class="searchresults-outer hidden">
<div id="searchresults-header" class="searchresults-header"></div>
<ul id="searchresults">
</ul>
</div>
</div>
<!-- Apply ARIA attributes after the sidebar and the sidebar toggle button are added to the DOM -->
<script>
document.getElementById('sidebar-toggle').setAttribute('aria-expanded', sidebar === 'visible');
document.getElementById('sidebar').setAttribute('aria-hidden', sidebar !== 'visible');
Array.from(document.querySelectorAll('#sidebar a')).forEach(function(link) {
link.setAttribute('tabIndex', sidebar === 'visible' ? 0 : -1);
});
</script>
<div id="content" class="content">
<main>
<h1 id="debugging"><a class="header" href="#debugging">Debugging</a></h1>
<h2 id="debugging-the-shadow-process"><a class="header" href="#debugging-the-shadow-process">Debugging the Shadow process</a></h2>
<p>Shadow is currently built with debugging symbols in both debug and release
builds, though it may be easier to debug a debug build (generated by passing
the <code>--debug</code> flag to <code>setup build</code>).</p>
<p>Shadow can be run under GDB by prepending <code>gdb --args</code> to its command-line.
e.g.:</p>
<pre><code>gdb --args shadow shadow.yaml
</code></pre>
<p>An alternative is to run Shadow with the <code>--gdb</code> flag, which will pause shadow
after startup and print its PID. You can then simply attach GDB to Shadow in a
new terminal and continue the experiment.</p>
<p>Example:</p>
<pre><code># terminal 1
# shadow will print its PID and pause
$ shadow --gdb shadow.yaml > shadow.log
** Starting Shadow
** Pausing with SIGTSTP to enable debugger attachment (pid 1234)
# terminal 2
$ gdb --pid=1234
> continue
</code></pre>
<h3 id="troubleshooting"><a class="header" href="#troubleshooting">Troubleshooting</a></h3>
<p>If when loading the shadow binary in gdb you see the error:</p>
<pre><code class="language-text">Reading symbols from /my/binary...
Dwarf Error: DW_FORM_strx1 found in non-DWO CU [in module /my/binary]
(No debugging symbols found in /my/binary)
</code></pre>
<p>It's likely that the version of Rust that you're building Shadow with is
incompatible with your version of GDB. You can read more at
<a href="https://github.com/rust-lang/rust/issues/98746#issuecomment-1224954019">Rust issue #98746</a>.</p>
<h2 id="debugging-managed-processes"><a class="header" href="#debugging-managed-processes">Debugging managed processes</a></h2>
<p>A simulation's managed processes are implemented as native OS processes, with
their syscalls interposed by Shadow. Since they are native processes, many
normal tools for inspecting native processes can be used on those as well. e.g.
<code>top</code> will show how much CPU and memory they are using.</p>
<h3 id="generating-a-core-file"><a class="header" href="#generating-a-core-file">Generating a core file</a></h3>
<p>If a managed process is crashing, it is sometimes easiest to let the native
process to generate a core file, and then use GDB to inspect it afterwards.</p>
<pre><code># Enable core dumps.
ulimit -c unlimited
# Ensure core dumps are written to a file.
# e.g. this is sometimes needed in Ubuntu to override the default behavior of
# piping the core file to the system crash handler.
echo core | sudo tee /proc/sys/kernel/core_pattern
# Run the simulation in which a process is crashing.
shadow shadow.yaml
# Tell gdb to inspect the core file. From within gdb you'll be able to
# inspect the state of the process when it was killed.
gdb <path-to-process-executable> <path-to-core-file>
</code></pre>
<h3 id="attaching-with-gdb"><a class="header" href="#attaching-with-gdb">Attaching with GDB</a></h3>
<p>You can attach GDB directly to the managed process. To make this easier you can
use the <code>--debug-hosts</code> option to pause Shadow after it launches each managed
process on the given hosts. Shadow will print the native process' PID before
stopping. For example, <code>--debug-hosts client,server</code> will pause Shadow after
launching any managed processees on hosts "client" and "server". This allows
you to attach GDB directly to those managed processes before resuming Shadow.</p>
<pre><code># terminal 1
$ shadow --debug-hosts client,server shadow.yaml > shadow.log
** Starting Shadow
** Pausing with SIGTSTP to enable debugger attachment to managed process 'server.nginx.1000' (pid 1234)
** If running Shadow under Bash, resume Shadow by pressing Ctrl-Z to background this task and then typing "fg".
** (If you wish to kill Shadow, type "kill %%" instead.)
** If running Shadow under GDB, resume Shadow by typing "signal SIGCONT".
# terminal 2
$ gdb --pid=1234
</code></pre>
<h3 id="debugging-with-gdb"><a class="header" href="#debugging-with-gdb">Debugging with GDB</a></h3>
<p>In managed processes, Shadow uses <code>SIGSYS</code> and <code>SIGSEGV</code> to intercept system
calls and some CPU instructions. By default, GDB stops every time these signals
are raised. In most cases you'll want to override this behavior to silently
continue executing instead:</p>
<pre><code>(gdb) handle SIGSYS noprint
(gdb) handle SIGSEGV noprint
</code></pre>
<p>Once you have reached a point of interest, it's often useful to look at the
backtrace for the current stack:</p>
<pre><code>(gdb) bt
</code></pre>
<p>In multi-threaded applications, you can get a backtrace for all stacks:</p>
<pre><code>(gdb) thread apply all bt
</code></pre>
</main>
<nav class="nav-wrapper" aria-label="Page navigation">
<!-- Mobile navigation buttons -->
<a rel="prev" href="coding.html" class="mobile-nav-chapters previous" title="Previous chapter" aria-label="Previous chapter" aria-keyshortcuts="Left">
<i class="fa fa-angle-left"></i>
</a>
<a rel="next prefetch" href="testing_determinism.html" class="mobile-nav-chapters next" title="Next chapter" aria-label="Next chapter" aria-keyshortcuts="Right">
<i class="fa fa-angle-right"></i>
</a>
<div style="clear: both"></div>
</nav>
</div>
</div>
<nav class="nav-wide-wrapper" aria-label="Page navigation">
<a rel="prev" href="coding.html" class="nav-chapters previous" title="Previous chapter" aria-label="Previous chapter" aria-keyshortcuts="Left">
<i class="fa fa-angle-left"></i>
</a>
<a rel="next prefetch" href="testing_determinism.html" class="nav-chapters next" title="Next chapter" aria-label="Next chapter" aria-keyshortcuts="Right">
<i class="fa fa-angle-right"></i>
</a>
</nav>
</div>
<script>
window.playground_copyable = true;
</script>
<script src="elasticlunr.min.js"></script>
<script src="mark.min.js"></script>
<script src="searcher.js"></script>
<script src="clipboard.min.js"></script>
<script src="highlight.js"></script>
<script src="book.js"></script>
<!-- Custom JS scripts -->
</div>
</body>
</html>