MFlowCode.github.io/documentation/md_running.html at master · BucketofJava/MFlowCode.github.io

History

297 lines (295 loc) · 19.1 KB

Raw

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

268

269

270

271

272

273

274

275

276

277

278

279

280

281

282

283

284

285

286

287

288

289

290

291

292

293

294

295

296

297

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "https://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">

<head>

window.dataLayer = window.dataLayer || [];

function gtag(){dataLayer.push(arguments);}

gtag('js', new Date());

gtag('config', 'G-SY496B9L99');

</script>

<title>MFC: Running</title>

MathJax.Hub.Config({

extensions: ["tex2jax.js", "TeX/AMSmath.js", "TeX/AMSsymbols.js"],

jax: ["input/TeX","output/HTML-CSS"],

});

// This file is set as MATHJAX_CODEFILE in the Doxyfile. It configures how

// MathJax renders expressions in Markdown so that it is consistent with GitHub.

MathJax.Hub.Config({

extensions: ["tex2jax.js"],

jax: ["input/TeX", "output/HTML-CSS"],

tex2jax: {

inlineMath: [ ['$', '$'], ["\$","\$"] ],

displayMath: [ ['$$','$$'], ["\\[","\\]"] ],

processEscapes: true,

ignoreClass: "line" // Ignore code blocks: https://web.archive.org/web/20120430100225/http://www.mathjax.org/docs/1.1/options/tex2jax.html

"HTML-CSS": {

fonts: ["TeX"]

}

});

</script>

</head>

<body>

<tbody>

<div id="projectname">MFC

</div>

<div id="projectbrief">High-fidelity multiphase flow simulation</div>

</td>

</tr>

</tbody>

</table>

</div>

/* @license magnet:?xt=urn:btih:d3d9a9a6595521f9666a5e94cc830dab83b65699&dn=expat.txt MIT */

var searchBox = new SearchBox("searchBox", "search/",'.html');

/* @license-end */

</script>

/* @license magnet:?xt=urn:btih:d3d9a9a6595521f9666a5e94cc830dab83b65699&dn=expat.txt MIT */

$(function() { codefold.init(0); });

/* @license-end */

</script>

/* @license magnet:?xt=urn:btih:d3d9a9a6595521f9666a5e94cc830dab83b65699&dn=expat.txt MIT */

$(function() {

initMenu('',true,false,'search.php','Search',true);

$(function() { init_search(); });

});

/* @license-end */

</script>

</div>

</div>

<div id="splitbar" style="-moz-user-select:none;"

class="ui-resizable-handle">

</div>

/* @license magnet:?xt=urn:btih:d3d9a9a6595521f9666a5e94cc830dab83b65699&dn=expat.txt MIT */

$(function(){initNavTree('md_running.html',''); initResizable(true); });

/* @license-end */

</script>

<div id="MSearchSelectWindow"

onmouseover="return searchBox.OnSearchSelectShow()"

onmouseout="return searchBox.OnSearchSelectHide()"

onkeydown="return searchBox.OnSearchSelectKey(event)">

</div>

<div class="SRStatus" id="Loading">Loading...</div>

<div class="SRStatus" id="Searching">Searching...</div>

<div class="SRStatus" id="NoMatches">No Matches</div>

</div>

<div class="headertitle"><div class="title">Running</div></div>

</div>

<div class="textblock"><p><a class="anchor" id="autotoc_md75"></a> MFC can be run using <code>mfc.sh</code>'s <code>run</code> command. It supports both interactive and batch execution, the latter being designed for multi-socket systems, namely supercomputers, equipped with a scheduler such as PBS, SLURM, and LSF. A full (and updated) list of available arguments can be acquired with <code>./mfc.sh run -h</code>.</p>

<p>MFC supports running simulations locally (Linux, MacOS, and Windows) as well as several supercomputer clusters, both interactively and through batch submission.</p>

<dl class="section important"><dt>Important</dt><dd>Running simulations locally should work out of the box. On supported clusters, you can append <code>-c <computer name></code> on the command line to instruct the MFC toolchain to make use of the template file <code>toolchain/templates/<computer name>.mako</code>. You can browse that directory and contribute your own files. Since systems and their schedulers do not have a standardized syntax to request certain resources, MFC can only provide support for a restricted subset of common or user-contributed configuration options.</dd></dl>

<p>Adding a new template file or modifying an existing one will most likely be required if:</p><ul>

<li>You are on a cluster that does not have a template yet.</li>

<li>Your cluster is configured with SLURM but interactive job launches fail when using <code>srun</code>. You might need to invoke <code>mpirun</code> instead.</li>

<li>Something in the existing default or computer template file is incompatible with your system or does not provide a feature you need.</li>

</ul>

<p>If <code>-c <computer name></code> is left unspecified, it defaults to <code>-c default</code>.</p>

<p>Additional flags can be appended to the MPI executable call using the <code>-f</code> (i.e <code>--flags</code>) option.</p>

<p>Please refer to <code>./mfc.sh run -h</code> for a complete list of arguments and options, along with their defaults.</p>

Interactive Execution</h1>

<p>To run all stages of MFC, that is <a href="https://github.com/MFlowCode/MFC/tree/master/src/pre_process/">pre_process</a>, <a href="https://github.com/MFlowCode/MFC/tree/master/src/simulation/">simulation</a>, and <a href="https://github.com/MFlowCode/MFC/tree/master/src/post_process/">post_process</a> on the sample case <a href="https://github.com/MFlowCode/MFC/tree/master/examples/2D_shockbubble/">2D_shockbubble</a>,</p>

<div class="fragment"><div class="line">./mfc.sh run examples/2D_shockbubble/case.py</div>

</div><p>If you want to run a subset of the available stages, you can use the <code>-t</code> argument. To use multiple threads, use the <code>-n</code> option along with the number of threads you wish to use. If a (re)build is required, it will be done automatically, with the number of threads specified with the <code>-j</code> option.</p>

<p>For example,</p>

<ul>

<li>Running <a href="https://github.com/MFlowCode/MFC/tree/master/src/pre_process/">pre_process</a> with 2 cores:</li>

</ul>

<div class="fragment"><div class="line">./mfc.sh run examples/2D_shockbubble/case.py -t pre_process -n 2</div>

</div><ul>

<li>Running <a href="https://github.com/MFlowCode/MFC/tree/master/src/simulation/">simulation</a> and <a href="https://github.com/MFlowCode/MFC/tree/master/src/post_process/">post_process</a> using 4 cores:</li>

</ul>

<div class="fragment"><div class="line">./mfc.sh run examples/2D_shockbubble/case.py -t simulation post_process -n 4</div>

</div><h1><a class="anchor" id="autotoc_md77"></a>

Batch Execution</h1>

<p>The MFC detects which scheduler your system is using and handles the creation and execution of batch scripts. The batch engine is requested via the <code>-e batch</code> option. The number of nodes can be specified with the <code>-N</code> (i.e., <code>--nodes</code>) option.</p>

<p>We provide a list of (baked-in) submission batch scripts in the <code>toolchain/templates</code> folder.</p>

<div class="fragment"><div class="line">./mfc.sh run examples/2D_shockbubble/case.py -e batch -N 2 -n 4 -t simulation -c <computer name></div>

</div><p>Other useful arguments include:</p>

<ul>

<li><code>-@ [email protected]</code> to receive emails from the scheduler. (i.e., <code>--email</code>)</li>

<li><code>-w hh:mm:ss</code> to specify the job's maximum allowed walltime. (i.e., <code>--walltime</code>)</li>

<li><code>-a <account name></code> to identify the account to be charged for the job. (i.e., <code>--account</code>)</li>

<li><code>-p <partition name></code> to select the job's partition. (i.e., <code>--partition</code>)</li>

</ul>

<p>As an example, one might request GPUs on a SLURM system using the following:</p>

<p><b>Disclaimer</b>: IBM's JSRUN on LSF-managed computers does not use the traditional node-based approach to allocate resources. Therefore, the MFC constructs equivalent resource sets in the task and GPU count.</p>

GPU Profiling</h2>

NVIDIA GPUs</h3>

<p>MFC provides two different arguments to facilitate profiling with NVIDIA Nsight. <b>Please ensure the used argument is placed at the end so their respective flags can be appended.</b></p><ul>

<li>Nsight Systems (Nsys): <code>./mfc.sh run ... -t simulation --nsys [nsys flags]</code> allows one to visualize MFC's system-wide performance with <a href="https://developer.nvidia.com/nsight-systems">NVIDIA Nsight Systems</a>. NSys is best for understanding the order and execution times of major subroutines (WENO, Riemann, etc.) in MFC. When used, <code>--nsys</code> will run the simulation and generate <code>.nsys-rep</code> files in the case directory for all targets. These files can then be imported into Nsight System's GUI, which can be downloaded <a href="https://developer.nvidia.com/nsight-systems/get-started#latest-Platforms">here</a>. It is best to run case files with a few timesteps to keep the report files small. Learn more about NVIDIA Nsight Systems <a href="https://docs.nvidia.com/nsight-systems/UserGuide/index.html">here</a>.</li>

<li>Nsight Compute (NCU): <code>./mfc.sh run ... -t simulation --ncu [ncu flags]</code> allows one to conduct kernel-level profiling with <a href="https://developer.nvidia.com/nsight-compute">NVIDIA Nsight Compute</a>. NCU provides profiling information for every subroutine called and is more detailed than NSys. When used, <code>--ncu</code> will output profiling information for all subroutines, including elapsed clock cycles, memory used, and more after the simulation is run. Adding this argument will significantly slow the simulation and should only be used on case files with a few timesteps. Learn more about NVIDIA Nsight Compute <a href="https://docs.nvidia.com/nsight-compute/NsightCompute/index.html">here</a>.</li>

</ul>

AMD GPUs</h3>

<ul>

<li>Rocprof (ROC): <code>./mfc.sh run ... -t simulation --roc --hip-trace [rocprof flags]</code> allows one to visualize MFC's system-wide performance with <a href="https://ui.perfetto.dev/">Perfetto UI</a>. When used, <code>--roc</code> will run the simulation and generate files in the case directory for all targets. <code>results.json</code> can then be imported in <a href="https://ui.perfetto.dev/">Perfetto's UI</a>. Learn more about AMD Rocprof <a href="https://rocm.docs.amd.com/projects/rocprofiler/en/docs-5.5.1/rocprof.html">here</a> It is best to run case files with a few timesteps to keep the report files small.</li>

<li>Omniperf (OMNI): <code>./mfc.sh run ... -t simulation --omni [omniperf flags]</code>allows one to conduct kernel-level profiling with <a href="https://rocm.github.io/omniperf/introduction.html#what-is-omniperf">AMD Omniperf</a>. When used, <code>--omni</code> will output profiling information for all subroutines, including rooflines, cache usage, register usage, and more after the simulation is run. Adding this argument will moderately slow down the simulation and run the MFC executable several times. For this reason it should only be used with case files that have a few timesteps.</li>

</ul>

Restarting Cases</h2>

<p>When running a simulation, MFC generates a <code>./restart_data</code> folder in the case directory that contains <code>lustre_*.dat</code> files that can be used to restart a simulation from saved timesteps. This allows a user to simulate some timestep $X$, then continue it to run to another timestep $Y$, where $Y > X$. The user can also choose to add new patches at the intermediate timestep.</p>

<p>If you want to restart a simulation,</p>

<ul>

<li>Set up the initial simulation, with:<ul>

<li><code>t_step_start</code> : $t_i$</li>

<li><code>t_step_save</code> : $SF$ in which $t_i$ is the starting time, $t_f$ is the final time, and $SF$ is the saving frequency time.</li>

</ul>

</li>

<li>Run <code>pre_process</code> and <code>simulation</code> on the case.<ul>

<li><code>./mfc.sh run case.py -t pre_process simulation</code></li>

</ul>

</li>

<li>As the simulation runs, it will create Lustre files for each saved timestep in <code>./restart_data</code>.</li>

<li>When the simulation stops, choose any Lustre file as the restarting point (lustre_ $t_s$.dat)</li>

<li>Create a new duplicate input file (e.g., <code>restart_case.py</code>), which should have:</li>

</ul>

<li>For the Computational Domain Parameters<ul>

<li>Have the following removed <b>except</b> <code>m</code>, <code>n</code>, and <code>p</code>:<ul>

<li>All domain/mesh information<ul>

<li><code>(xyz)_domainbeg</code></li>

<li><code>(xyz)_domainend</code></li>

<li><code>stretch_(xyz)</code></li>

</ul>

</li>

</ul>

</li>

<li>Alter the following:<ul>

<li><code>t_step_start</code> : $t_s$ (the point at which the simulation will restart)</li>

<li><code>t_step_stop</code> : $t_{f2}$ (new final simulation time, which can be the same as $t_f$)</li>

<li><code>t_step_save</code> : ${SF}_2$ (if interested in changing the saving frequency)</li>

</ul>

</li>

<li>Add the following:<ul>

<li><code>old_ic</code> : 'T' (to specify that we have initial conditions from previous simulations)</li>

<li><code>old_grid</code> : 'T' (to specify that we have a grid from previous simulations)</li>

<li><code>t_step_old</code> : $t_i$ (the time step used as the <code>t_step_start</code> of the original <code>case.py</code> file)</li>

</ul>

</li>

</ul>

</li>

<li>For the Simulation Algorithm Parameters<ul>

<li>Substitute <code>num_patches</code> to reflect the number of ADDED patches in the <code>restart_case.py</code> file. If no patches are added, set <code>num_patches: 0</code></li>

</ul>

</li>

<li>For Patches<ul>

<li>Have all information about old patches (used in the <code>case.py</code> file) removed.<ul>

<li><code>patch_icpp(1)all variables</code></li>

<li><code>patch_icpp(2)all variables</code></li>

<li><code>patch_icpp(num_patches)all variables</code></li>

</ul>

</li>

<li>Add information about new patches that will be introduced, if any. The parameter num_patches should reflect this addition.<ul>

<li>e.g. <code>patch_icpp(1)some variables of interest</code> <br />

</li>

</ul>

</li>

</ul>

</li>

<li>For Fluid properties<ul>

<li>Keep information about the fluid properties</li>

</ul>

</li>

</ol>

<ul>

<li>Run pre-process and simulation on <code>restart_case.py</code><ul>

<li><code>./mfc.sh run restart_case.py -t pre_process simulation</code></li>

</ul>

</li>

<li>Run the post_process<ul>

<li>There are several ways to do this. Keep in mind that, regardless of the .py file used, the post_process command will generate output files in the [<code>t_step_start</code>, <code>t_step_stop</code>] range, with <code>t_step_save</code> as the spacing between files.</li>

<li>One way is to set <code>t_step_stop</code> to the restarting point $t_s$ in <code>case.py</code>. Then, run the commands below. The first command will run on timesteps $[t_i, t_s]$. The second command will run on $[t_s, t_{f2}]$. Therefore, the whole range $[t_i, t_{f2}]$ will be post processed.</li>

</ul>

</li>

</ul>

<div class="fragment"><div class="line">./mfc.sh run case.py -t post_process</div>

<div class="line">./mfc.sh run restart_case.py -t post_process</div>

</div><p>We have provided an example, <code>case.py</code> and <code>restart_case.py</code> in <code>/examples/1D_vacuum_restart/</code>. This simulation is a duplicate of the <code>1D_vacuum</code> case. It demonstrates stopping at timestep 7000, adding a new patch, and restarting the simulation. To test this code, run:</p>

<div class="fragment"><div class="line">./mfc.sh run examples/1D_vacuum_restart/case.py -t pre_process simulation</div>

<div class="line">./mfc.sh run examples/1D_vacuum_restart/restart_case.py -t pre_process simulation</div>

<div class="line">./mfc.sh run examples/1D_vacuum_restart/case.py -t post_process</div>

<div class="line">./mfc.sh run examples/1D_vacuum_restart/restart_case.py -t post_process</div>

</div><h2><a class="anchor" id="autotoc_md82"></a>

Example Runs</h2>

<ul>

<li>Oak Ridge National Laboratory's <a href="https://www.olcf.ornl.gov/summit/">Summit</a>:</li>

</ul>

<div class="fragment"><div class="line">./mfc.sh run examples/2D_shockbubble/case.py -e batch \</div>

<div class="line"> -N 2 -n 4 -t simulation -a <redacted> -c summit</div>

</div> </div></div>

</div>

</div>

<ul>

<li class="footer">Generated by <a href="https://www.doxygen.org/index.html"><img class="footer" src="doxygen.svg" width="104" height="31" alt="doxygen"/></a> 1.11.0 </li>

</ul>

</div>

</body>

</html>

FilesExpand file tree

md_running.html

Latest commit

History

md_running.html

File metadata and controls