nextcode-python-sdk/docs/api_workflow.html at master · Haffi/nextcode-python-sdk

604 lines (546 loc) · 52 KB
<!DOCTYPE html>
    <meta charset="utf-8" />
    <meta name="viewport" content="width=device-width, initial-scale=1.0" />
    <title>Workflow Service &#8212; nextcode-python-sdk 1.6.0 documentation</title>
    <link rel="stylesheet" href="_static/pygments.css" type="text/css" />
    <link rel="stylesheet" href="_static/bootstrap-sphinx.css" type="text/css" />
    <link rel="stylesheet" type="text/css" href="_static/css/custom.css" />
    <script id="documentation_options" data-url_root="./" src="_static/documentation_options.js"></script>
    <script src="_static/jquery.js"></script>
    <script src="_static/underscore.js"></script>
    <script src="_static/doctools.js"></script>
    <script crossorigin="anonymous" integrity="sha256-Ae2Vz/4ePdIu6ZyI/5ZGsYnb+m0JlOmKPjt6XZ9JJkA=" src="https://cdnjs.cloudflare.com/ajax/libs/require.js/2.3.4/require.min.js"></script>
    <link rel="index" title="Index" href="genindex.html" />
    <link rel="search" title="Search" href="search.html" />
    <link rel="next" title="Pipelines Service" href="api_pipelines.html" />
    <link rel="prev" title="Query Service" href="api_query.html" />
<meta charset='utf-8'>
<meta http-equiv='X-UA-Compatible' content='IE=edge,chrome=1'>
<meta name='viewport' content='width=device-width, initial-scale=1.0, maximum-scale=1'>
<meta name="apple-mobile-web-app-capable" content="yes">
<script type="text/javascript" src="_static/js/jquery-1.11.0.min.js "></script>
<script type="text/javascript" src="_static/js/jquery-fix.js "></script>
<script type="text/javascript" src="_static/bootstrap-3.3.7/js/bootstrap.min.js "></script>
<script type="text/javascript" src="_static/bootstrap-sphinx.js "></script>
  </head><body>
  <div id="navbar" class="navbar navbar-default navbar-fixed-top">
    <div class="container">
      <div class="navbar-header">
        <!-- .btn-navbar is used as the toggle for collapsed navbar content -->
        <button type="button" class="navbar-toggle" data-toggle="collapse" data-target=".nav-collapse">
          <span class="icon-bar"></span>
          <span class="icon-bar"></span>
          <span class="icon-bar"></span>
        </button>
        <a class="navbar-brand" href="index.html">
          nextcode-python-sdk</a>
        <span class="navbar-text navbar-version pull-left"><b>1.6.0</b></span>
      </div>
        <div class="collapse navbar-collapse nav-collapse">
          <ul class="nav navbar-nav">
              <li class="dropdown globaltoc-container">
  <a role="button"
     id="dLabelGlobalToc"
     data-toggle="dropdown"
     data-target="#"
     href="index.html">Contents <b class="caret"></b></a>
  <ul class="dropdown-menu globaltoc"
      role="menu"
      aria-labelledby="dLabelGlobalToc"><ul class="current">
<li class="toctree-l1"><a class="reference internal" href="quickstart.html">Quick Start</a></li>
<li class="toctree-l1"><a class="reference internal" href="jupyter.html">Jupyter Notebooks Integration</a></li>
<li class="toctree-l1 current"><a class="reference internal" href="api.html">API Reference</a></li>
<li class="toctree-l1"><a class="reference internal" href="notebook.html">Building queries with the Python SDK</a></li>
<li class="toctree-l1"><a class="reference internal" href="apidoc.html">Server API Documention</a></li>
<li class="toctree-l1"><a class="reference internal" href="development.html">Developer information</a></li>
<li class="toctree-l1"><a class="reference internal" href="local_workflows.html">Running a local workflow</a></li>
                <li class="dropdown">
  <a role="button"
     id="dLabelLocalToc"
     data-toggle="dropdown"
     data-target="#"
     href="#">Page <b class="caret"></b></a>
  <ul class="dropdown-menu localtoc"
      role="menu"
      aria-labelledby="dLabelLocalToc"><ul>
<li><a class="reference internal" href="#">Workflow Service</a><ul>
<li><a class="reference internal" href="#service-class">Service class</a></li>
<li><a class="reference internal" href="#workflow-job">Workflow Job</a></li>
<li><a class="reference internal" href="#pytest-plugin">Pytest plugin</a></li>
<li><a class="reference internal" href="#exceptions">Exceptions</a></li>
    <a href="api_query.html" title="Previous Chapter: Query Service"><span class="glyphicon glyphicon-chevron-left visible-sm"></span><span class="hidden-sm hidden-tablet">&laquo; Query Service</span>
    <a href="api_pipelines.html" title="Next Chapter: Pipelines Service"><span class="glyphicon glyphicon-chevron-right visible-sm"></span><span class="hidden-sm hidden-tablet">Pipelines Service &raquo;</span>
          </ul>
<form class="navbar-form navbar-right" action="search.html" method="get">
 <div class="form-group">
  <input type="text" name="q" class="form-control" placeholder="Search" />
  <input type="hidden" name="check_keywords" value="yes" />
  <input type="hidden" name="area" value="default" />
        </div>
<div class="container">
  <div class="row">
    <div class="body col-md-12 content" role="main">
  <div class="section" id="module-nextcode.services.workflow.service">
<span id="workflow-service"></span><h1>Workflow Service<a class="headerlink" href="#module-nextcode.services.workflow.service" title="Permalink to this headline">¶</a></h1>
<div class="section" id="service-class">
<h2>Service class<a class="headerlink" href="#service-class" title="Permalink to this headline">¶</a></h2>
<p>Service object for interfacing with the Workflow Service API.</p>
<p>This class instance is used to communicate with a RESTFul service. The <cite>post_job</cite> method
creates a new job on the server and <cite>find_job</cite> and <cite>get_jobs</cite> allow you to inspect running
and past workflow jobs.</p>
<p>Note: This service has not been fully integrated into the SDK and access is still quite raw.</p>
<dl class="py class">
<dt id="nextcode.services.workflow.service.Service">
<em class="property"><span class="pre">class</span> </em><code class="sig-prename descclassname"><span class="pre">nextcode.services.workflow.service.</span></code><code class="sig-name descname"><span class="pre">Service</span></code><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">client</span></span></em>, <em class="sig-param"><span class="o"><span class="pre">*</span></span><span class="n"><span class="pre">args</span></span></em>, <em class="sig-param"><span class="o"><span class="pre">**</span></span><span class="n"><span class="pre">kwargs</span></span></em><span class="sig-paren">)</span><a class="reference internal" href="_modules/nextcode/services/workflow/service.html#Service"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#nextcode.services.workflow.service.Service" title="Permalink to this definition">¶</a></dt>
<dd><p>A connection to the workflow service API server</p>
<dl class="py method">
<dt id="nextcode.services.workflow.service.Service.current_user">
<em class="property"><span class="pre">property</span> </em><code class="sig-name descname"><span class="pre">current_user</span></code><a class="headerlink" href="#nextcode.services.workflow.service.Service.current_user" title="Permalink to this definition">¶</a></dt>
<dd><p>User JWT decoded by the service</p>
<dl class="field-list simple">
<dt class="field-odd">Return type</dt>
<dd class="field-odd"><p><code class="xref py py-class docutils literal notranslate"><span class="pre">Dict</span></code></p>
<dl class="py method">
<dt id="nextcode.services.workflow.service.Service.endpoints">
<em class="property"><span class="pre">property</span> </em><code class="sig-name descname"><span class="pre">endpoints</span></code><a class="headerlink" href="#nextcode.services.workflow.service.Service.endpoints" title="Permalink to this definition">¶</a></dt>
<dd><p>Dictionary of name: endpoint url for exposed endpoints in the service</p>
<dl class="field-list simple">
<dt class="field-odd">Return type</dt>
<dd class="field-odd"><p><code class="xref py py-class docutils literal notranslate"><span class="pre">Dict</span></code></p>
<dl class="py method">
<dt id="nextcode.services.workflow.service.Service.find_job">
<code class="sig-name descname"><span class="pre">find_job</span></code><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">job_id</span></span></em><span class="sig-paren">)</span><a class="reference internal" href="_modules/nextcode/services/workflow/service.html#Service.find_job"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#nextcode.services.workflow.service.Service.find_job" title="Permalink to this definition">¶</a></dt>
<dd><p>Return a job proxy object</p>
<dl class="field-list simple">
<dt class="field-odd">Return type</dt>
<dd class="field-odd"><p><a class="reference internal" href="#nextcode.services.workflow.job.WorkflowJob" title="nextcode.services.workflow.job.WorkflowJob"><code class="xref py py-class docutils literal notranslate"><span class="pre">WorkflowJob</span></code></a></p>
<dl class="py method">
<dt id="nextcode.services.workflow.service.Service.get_jobs">
<code class="sig-name descname"><span class="pre">get_jobs</span></code><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">user_name</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">None</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">status</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">None</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">project</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">None</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">pipeline</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">None</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">state</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">None</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">context</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">None</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">limit</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">50</span></span></em><span class="sig-paren">)</span><a class="reference internal" href="_modules/nextcode/services/workflow/service.html#Service.get_jobs"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#nextcode.services.workflow.service.Service.get_jobs" title="Permalink to this definition">¶</a></dt>
<dd><p>Get a list of jobs satisfying the supplied criteria</p>
<dl class="field-list simple">
<dt class="field-odd">Parameters</dt>
<dd class="field-odd"><ul class="simple">
<li><p><strong>user_name</strong> (<code class="xref py py-data docutils literal notranslate"><span class="pre">Optional</span></code>[<code class="xref py py-class docutils literal notranslate"><span class="pre">str</span></code>]) – The user who created the job</p></li>
<li><p><strong>status</strong> (<code class="xref py py-data docutils literal notranslate"><span class="pre">Optional</span></code>[<code class="xref py py-class docutils literal notranslate"><span class="pre">str</span></code>]) – Current status of jobs</p></li>
<li><p><strong>project</strong> (<code class="xref py py-data docutils literal notranslate"><span class="pre">Optional</span></code>[<code class="xref py py-class docutils literal notranslate"><span class="pre">str</span></code>]) – Filter by project</p></li>
<li><p><strong>pipeline</strong> (<code class="xref py py-data docutils literal notranslate"><span class="pre">Optional</span></code>[<code class="xref py py-class docutils literal notranslate"><span class="pre">str</span></code>]) – Filter by pipeline name</p></li>
<li><p><strong>state</strong> (<code class="xref py py-data docutils literal notranslate"><span class="pre">Optional</span></code>[<code class="xref py py-class docutils literal notranslate"><span class="pre">str</span></code>]) – Filter by state, each state encapsulates several statuses (running, finished)</p></li>
<li><p><strong>context</strong> (<code class="xref py py-data docutils literal notranslate"><span class="pre">Optional</span></code>[<code class="xref py py-class docutils literal notranslate"><span class="pre">str</span></code>]) – Filter by context string</p></li>
<li><p><strong>limit</strong> (<code class="xref py py-data docutils literal notranslate"><span class="pre">Optional</span></code>[<code class="xref py py-class docutils literal notranslate"><span class="pre">int</span></code>]) – Maximum number of jobs to return</p></li>
<dt class="field-even">Return type</dt>
<dd class="field-even"><p><code class="xref py py-class docutils literal notranslate"><span class="pre">List</span></code>[<a class="reference internal" href="#nextcode.services.workflow.job.WorkflowJob" title="nextcode.services.workflow.job.WorkflowJob"><code class="xref py py-class docutils literal notranslate"><span class="pre">WorkflowJob</span></code></a>]</p>
<dl class="py method">
<dt id="nextcode.services.workflow.service.Service.get_pipelines">
<code class="sig-name descname"><span class="pre">get_pipelines</span></code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="_modules/nextcode/services/workflow/service.html#Service.get_pipelines"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#nextcode.services.workflow.service.Service.get_pipelines" title="Permalink to this definition">¶</a></dt>
<dd><p>Returns the pipelines available on the current server</p>
<p>Refer to the API documentation for the Workflow service to see formatting of data.</p>
<dl class="field-list simple">
<dt class="field-odd">Return type</dt>
<dd class="field-odd"><p><code class="xref py py-class docutils literal notranslate"><span class="pre">List</span></code></p>
<dt class="field-even">Returns</dt>
<dd class="field-even"><p>List of pipelines</p>
<dl class="py method">
<dt id="nextcode.services.workflow.service.Service.get_projects">
<code class="sig-name descname"><span class="pre">get_projects</span></code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="_modules/nextcode/services/workflow/service.html#Service.get_projects"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#nextcode.services.workflow.service.Service.get_projects" title="Permalink to this definition">¶</a></dt>
<dd><p>Returns the projects that have been created on the current server</p>
<p>Refer to the API documentation for the Workflow service to see formatting of data.</p>
<dl class="field-list simple">
<dt class="field-odd">Return type</dt>
<dd class="field-odd"><p><code class="xref py py-class docutils literal notranslate"><span class="pre">List</span></code></p>
<dt class="field-even">Returns</dt>
<dd class="field-even"><p>List of projects</p>
<dl class="py method">
<dt id="nextcode.services.workflow.service.Service.healthy">
<code class="sig-name descname"><span class="pre">healthy</span></code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#nextcode.services.workflow.service.Service.healthy" title="Permalink to this definition">¶</a></dt>
<dd><p>Is the service healthy?</p>
<dl class="field-list simple">
<dt class="field-odd">Return type</dt>
<dd class="field-odd"><p><code class="xref py py-class docutils literal notranslate"><span class="pre">bool</span></code></p>
<dl class="py method">
<dt id="nextcode.services.workflow.service.Service.openapi_spec">
<code class="sig-name descname"><span class="pre">openapi_spec</span></code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#nextcode.services.workflow.service.Service.openapi_spec" title="Permalink to this definition">¶</a></dt>
<dd><p>Raw openapi spec for the service</p>
<dl class="field-list simple">
<dt class="field-odd">Return type</dt>
<dd class="field-odd"><p><code class="xref py py-class docutils literal notranslate"><span class="pre">Dict</span></code></p>
<dl class="py method">
<dt id="nextcode.services.workflow.service.Service.post_job">
<code class="sig-name descname"><span class="pre">post_job</span></code><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">pipeline_name</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">None</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">project_name</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">None</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">params</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">None</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">script</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">None</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">revision</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">None</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">build_source</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">None</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">build_context</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">None</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">profile</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">None</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">trace</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">False</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">details</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">None</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">description</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">None</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">executor_memory_mb</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">None</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">context</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">None</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">storage_type</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">None</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">dedicated_storage_size</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">None</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">keep_resources</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">None</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">mount_jobs</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">None</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">credentials</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">None</span></span></em><span class="sig-paren">)</span><a class="reference internal" href="_modules/nextcode/services/workflow/service.html#Service.post_job"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#nextcode.services.workflow.service.Service.post_job" title="Permalink to this definition">¶</a></dt>
<dd><p>Run a workflow job</p>
<p>This is a low-level implementation on top of the workflow service’s POST /jobs endpoint.</p>
<dl class="field-list simple">
<dt class="field-odd">Parameters</dt>
<dd class="field-odd"><ul class="simple">
<li><p><strong>pipeline_name</strong> (<code class="xref py py-data docutils literal notranslate"><span class="pre">Optional</span></code>[<code class="xref py py-class docutils literal notranslate"><span class="pre">str</span></code>]) – Name of the pipeline to run</p></li>
<li><p><strong>project_name</strong> (<code class="xref py py-data docutils literal notranslate"><span class="pre">Optional</span></code>[<code class="xref py py-class docutils literal notranslate"><span class="pre">str</span></code>]) – Name of the project to run this job on, defaults to GOR_API_PROJECT in environment</p></li>
<li><p><strong>params</strong> (<code class="xref py py-data docutils literal notranslate"><span class="pre">Optional</span></code>[<code class="xref py py-class docutils literal notranslate"><span class="pre">Dict</span></code>]) – Dictionary of parameters to forward to the job</p></li>
<li><p><strong>script</strong> (<code class="xref py py-data docutils literal notranslate"><span class="pre">Optional</span></code>[<code class="xref py py-class docutils literal notranslate"><span class="pre">str</span></code>]) – Git repository url to run (only when server is in development mode)</p></li>
<li><p><strong>revision</strong> (<code class="xref py py-data docutils literal notranslate"><span class="pre">Optional</span></code>[<code class="xref py py-class docutils literal notranslate"><span class="pre">str</span></code>]) – Git revision or tag (only when server is in development mode)</p></li>
<li><p><strong>build_source</strong> (<code class="xref py py-data docutils literal notranslate"><span class="pre">Optional</span></code>[<code class="xref py py-class docutils literal notranslate"><span class="pre">str</span></code>]) – Source type of the nextflow build (builtin, git, url)</p></li>
<li><p><strong>build_context</strong> (<code class="xref py py-data docutils literal notranslate"><span class="pre">Optional</span></code>[<code class="xref py py-class docutils literal notranslate"><span class="pre">str</span></code>]) – Context for build_source, depends on the type.</p></li>
<li><p><strong>profile</strong> (<code class="xref py py-data docutils literal notranslate"><span class="pre">Optional</span></code>[<code class="xref py py-class docutils literal notranslate"><span class="pre">str</span></code>]) – Nextflow profile name to use.</p></li>
<li><p><strong>trace</strong> (<code class="xref py py-class docutils literal notranslate"><span class="pre">bool</span></code>) – Instruct the job to run nextflow trace flags for debugging.</p></li>
<li><p><strong>details</strong> (<code class="xref py py-data docutils literal notranslate"><span class="pre">Optional</span></code>[<code class="xref py py-class docutils literal notranslate"><span class="pre">Dict</span></code>]) – Dictionary containing the initial values for ‘details’ in the job</p></li>
<li><p><strong>description</strong> (<code class="xref py py-data docutils literal notranslate"><span class="pre">Optional</span></code>[<code class="xref py py-class docutils literal notranslate"><span class="pre">str</span></code>]) – Human readable description of the job</p></li>
<li><p><strong>executor_memory_mb</strong> (<code class="xref py py-data docutils literal notranslate"><span class="pre">Optional</span></code>[<code class="xref py py-class docutils literal notranslate"><span class="pre">int</span></code>]) – Override the memory limit of the nextflow executor</p></li>
<li><p><strong>context</strong> (<code class="xref py py-data docutils literal notranslate"><span class="pre">Optional</span></code>[<code class="xref py py-class docutils literal notranslate"><span class="pre">str</span></code>]) – Optional string to allow querying for custom information</p></li>
<li><p><strong>storage_type</strong> (<code class="xref py py-data docutils literal notranslate"><span class="pre">Optional</span></code>[<code class="xref py py-class docutils literal notranslate"><span class="pre">str</span></code>]) – Optional string specifying the storage option to use for the pipeline</p></li>
<dl class="simple">
<dt>:param dedicated_storage_size Optional string specifying the size of the dedicated volume if using</dt><dd><p>storage_type=dedicated</p>
<dl class="field-list simple">
<dt class="field-odd">Parameters</dt>
<dd class="field-odd"><ul class="simple">
<li><p><strong>keep_resources</strong> (<code class="xref py py-data docutils literal notranslate"><span class="pre">Optional</span></code>[<code class="xref py py-class docutils literal notranslate"><span class="pre">bool</span></code>]) – Optional boolean specifying if resources created by workflow service for the Job should
be exempt from being cleanup up by the workflow resource cleanup process.</p></li>
<li><p><strong>mount_jobs</strong> (<code class="xref py py-data docutils literal notranslate"><span class="pre">Optional</span></code>[<code class="xref py py-class docutils literal notranslate"><span class="pre">List</span></code>[<code class="xref py py-class docutils literal notranslate"><span class="pre">int</span></code>]]) – Optional list of job-ids whose work folder will be mounted into this job under
the folder /job/&lt;job-id&gt;.</p></li>
<li><p><strong>credentials</strong> (<code class="xref py py-data docutils literal notranslate"><span class="pre">Optional</span></code>[<code class="xref py py-class docutils literal notranslate"><span class="pre">Dict</span></code>]) – Optional dict containing credentials to forward to workflow-service</p></li>
<dt>example credentials:  {</dt><dd><dl class="simple">
<dt>‘download’: {</dt><dd><p>‘aws_access_key_id’: ‘ASIA…’,
‘aws_secret_access_key’: ‘k6q5…’</p>
‘upload’: {</p>
<blockquote>
<div><p>‘aws_access_key_id’: ‘ASIA…’,
‘aws_secret_access_key’: ‘k6q5…’</p>
</div></blockquote>
<dl class="py method">
<dt id="nextcode.services.workflow.service.Service.run_local">
<code class="sig-name descname"><span class="pre">run_local</span></code><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">path</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">project_name</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">None</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">params</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">None</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">profile</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">None</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">trace</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">False</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">details</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">None</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">description</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">None</span></span></em><span class="sig-paren">)</span><a class="reference internal" href="_modules/nextcode/services/workflow/service.html#Service.run_local"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#nextcode.services.workflow.service.Service.run_local" title="Permalink to this definition">¶</a></dt>
<dd><p>Run a workflow job from a local nextflow folder</p>
<p>This will package the passed-in folder and upload to a specific bucket that is exported from the workflow
service on the cluster (see ‘scratch-bucket’ from the svc.status() method)
AWS access keys must be set up correctly to upload to this bucket using typical aws/boto environment and
credential management.</p>
<dl class="field-list simple">
<dt class="field-odd">Parameters</dt>
<dd class="field-odd"><ul class="simple">
<li><p><strong>path</strong> – Path (relative or absolute) to a folder containing a nextflow script</p></li>
<li><p><strong>project_name</strong> (<code class="xref py py-data docutils literal notranslate"><span class="pre">Optional</span></code>[<code class="xref py py-class docutils literal notranslate"><span class="pre">str</span></code>]) – Name of the project to run this job on, defaults to GOR_API_PROJECT in environment</p></li>
<li><p><strong>params</strong> (<code class="xref py py-data docutils literal notranslate"><span class="pre">Optional</span></code>[<code class="xref py py-class docutils literal notranslate"><span class="pre">Dict</span></code>]) – Dictionary of parameters to forward to the job</p></li>
<li><p><strong>profile</strong> (<code class="xref py py-data docutils literal notranslate"><span class="pre">Optional</span></code>[<code class="xref py py-class docutils literal notranslate"><span class="pre">str</span></code>]) – Nextflow profile name to use.</p></li>
<li><p><strong>trace</strong> (<code class="xref py py-class docutils literal notranslate"><span class="pre">bool</span></code>) – Instruct the job to run nextflow trace flags for debugging.</p></li>
<li><p><strong>details</strong> (<code class="xref py py-data docutils literal notranslate"><span class="pre">Optional</span></code>[<code class="xref py py-class docutils literal notranslate"><span class="pre">Dict</span></code>]) – Dictionary containing the initial values for ‘details’ in the job</p></li>
<li><p><strong>description</strong> (<code class="xref py py-data docutils literal notranslate"><span class="pre">Optional</span></code>[<code class="xref py py-class docutils literal notranslate"><span class="pre">str</span></code>]) – Human readable description of the job</p></li>
<dl class="py method">
<dt id="nextcode.services.workflow.service.Service.status">
<code class="sig-name descname"><span class="pre">status</span></code><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">force</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">False</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#nextcode.services.workflow.service.Service.status" title="Permalink to this definition">¶</a></dt>
<dd><p>Service information from the root endpoint</p>
<dl class="field-list simple">
<dt class="field-odd">Return type</dt>
<dd class="field-odd"><p><code class="xref py py-class docutils literal notranslate"><span class="pre">Dict</span></code></p>
<span class="target" id="module-nextcode.services.workflow.job"></span><div class="section" id="workflow-job">
<h2>Workflow Job<a class="headerlink" href="#workflow-job" title="Permalink to this headline">¶</a></h2>
<p>The Query class represents a workflow job model from the RESTFul Workflow Service API</p>
<dl class="py class">
<dt id="nextcode.services.workflow.job.WorkflowJob">
<em class="property"><span class="pre">class</span> </em><code class="sig-prename descclassname"><span class="pre">nextcode.services.workflow.job.</span></code><code class="sig-name descname"><span class="pre">WorkflowJob</span></code><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">session</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">job_id</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">job</span></span></em><span class="sig-paren">)</span><a class="reference internal" href="_modules/nextcode/services/workflow/job.html#WorkflowJob"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#nextcode.services.workflow.job.WorkflowJob" title="Permalink to this definition">¶</a></dt>
<dd><p>Proxy object for a serverside workflow job.</p>
<p>This object can be queried for the current status of a workflow job and will
automatically refresh from server until the job is finished (or failed)</p>
<dl class="py method">
<dt id="nextcode.services.workflow.job.WorkflowJob.cancel">
<code class="sig-name descname"><span class="pre">cancel</span></code><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">status</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">None</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">status_message</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">None</span></span></em><span class="sig-paren">)</span><a class="reference internal" href="_modules/nextcode/services/workflow/job.html#WorkflowJob.cancel"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#nextcode.services.workflow.job.WorkflowJob.cancel" title="Permalink to this definition">¶</a></dt>
<dd><p>Cancel a running job</p>
<dl class="field-list simple">
<dt class="field-odd">Parameters</dt>
<dd class="field-odd"><p><strong>status</strong> (<code class="xref py py-data docutils literal notranslate"><span class="pre">Optional</span></code>[<code class="xref py py-class docutils literal notranslate"><span class="pre">str</span></code>]) – status of the cancelled job. Defaults to CANCELLED</p>
<dt class="field-even">Return type</dt>
<dd class="field-even"><p><code class="xref py py-data docutils literal notranslate"><span class="pre">Optional</span></code>[<code class="xref py py-class docutils literal notranslate"><span class="pre">str</span></code>]</p>
<dt class="field-odd">Returns</dt>
<dd class="field-odd"><p>Status message from the server</p>
<dl class="py method">
<dt id="nextcode.services.workflow.job.WorkflowJob.done">
<em class="property"><span class="pre">property</span> </em><code class="sig-name descname"><span class="pre">done</span></code><a class="headerlink" href="#nextcode.services.workflow.job.WorkflowJob.done" title="Permalink to this definition">¶</a></dt>
<dd><p>Has the job completed successfully</p>
<dl class="field-list simple">
<dt class="field-odd">Return type</dt>
<dd class="field-odd"><p><code class="xref py py-class docutils literal notranslate"><span class="pre">bool</span></code></p>
<dl class="py method">
<dt id="nextcode.services.workflow.job.WorkflowJob.duration">
<em class="property"><span class="pre">property</span> </em><code class="sig-name descname"><span class="pre">duration</span></code><a class="headerlink" href="#nextcode.services.workflow.job.WorkflowJob.duration" title="Permalink to this definition">¶</a></dt>
<dd><p>Stringified duration of job for human consumption</p>
<dl class="field-list simple">
<dt class="field-odd">Return type</dt>
<dd class="field-odd"><p><code class="xref py py-class docutils literal notranslate"><span class="pre">str</span></code></p>
<dl class="py method">
<dt id="nextcode.services.workflow.job.WorkflowJob.events">
<code class="sig-name descname"><span class="pre">events</span></code><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">limit</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">50</span></span></em><span class="sig-paren">)</span><a class="reference internal" href="_modules/nextcode/services/workflow/job.html#WorkflowJob.events"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#nextcode.services.workflow.job.WorkflowJob.events" title="Permalink to this definition">¶</a></dt>
<dd><p>Get a list of events reported by Nextflow for this job</p>
<dl class="field-list simple">
<dt class="field-odd">Parameters</dt>
<dd class="field-odd"><p><strong>limit</strong> (<code class="xref py py-class docutils literal notranslate"><span class="pre">int</span></code>) – Maximum number of events to return</p>
<dt class="field-even">Return type</dt>
<dd class="field-even"><p><code class="xref py py-class docutils literal notranslate"><span class="pre">List</span></code>[<code class="xref py py-class docutils literal notranslate"><span class="pre">Dict</span></code>]</p>
<dl class="py method">
<dt id="nextcode.services.workflow.job.WorkflowJob.failed">
<em class="property"><span class="pre">property</span> </em><code class="sig-name descname"><span class="pre">failed</span></code><a class="headerlink" href="#nextcode.services.workflow.job.WorkflowJob.failed" title="Permalink to this definition">¶</a></dt>
<dd><p>Is the job in a failed state</p>
<dl class="field-list simple">
<dt class="field-odd">Return type</dt>
<dd class="field-odd"><p><code class="xref py py-class docutils literal notranslate"><span class="pre">bool</span></code></p>
<dl class="py method">
<dt id="nextcode.services.workflow.job.WorkflowJob.finished">
<em class="property"><span class="pre">property</span> </em><code class="sig-name descname"><span class="pre">finished</span></code><a class="headerlink" href="#nextcode.services.workflow.job.WorkflowJob.finished" title="Permalink to this definition">¶</a></dt>
<dd><p>Has the job finished (might be failed)</p>
<dl class="field-list simple">
<dt class="field-odd">Return type</dt>
<dd class="field-odd"><p><code class="xref py py-class docutils literal notranslate"><span class="pre">bool</span></code></p>
<dl class="py method">
<dt id="nextcode.services.workflow.job.WorkflowJob.inspect">
<code class="sig-name descname"><span class="pre">inspect</span></code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="_modules/nextcode/services/workflow/job.html#WorkflowJob.inspect"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#nextcode.services.workflow.job.WorkflowJob.inspect" title="Permalink to this definition">¶</a></dt>
<dd><p>Inspect a failed job for debugging.</p>
<p>Returns unfiltered pod and node information from the kubernetes system</p>
<dl class="field-list simple">
<dt class="field-odd">Return type</dt>
<dd class="field-odd"><p><code class="xref py py-class docutils literal notranslate"><span class="pre">Dict</span></code></p>
<dt class="field-even">Returns</dt>
<dd class="field-even"><p>Dictionary containing low-level debugging information</p>
<dt class="field-odd">Raises</dt>
<dd class="field-odd"><p><code class="xref py py-exc docutils literal notranslate"><span class="pre">ServerError</span></code> if the server is not configured for inspection capabilities</p>
<dl class="py method">
<dt id="nextcode.services.workflow.job.WorkflowJob.log_groups">
<code class="sig-name descname"><span class="pre">log_groups</span></code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="_modules/nextcode/services/workflow/job.html#WorkflowJob.log_groups"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#nextcode.services.workflow.job.WorkflowJob.log_groups" title="Permalink to this definition">¶</a></dt>
<dd><p>Get available log groups</p>
<dl class="field-list simple">
<dt class="field-odd">Return type</dt>
<dd class="field-odd"><p><code class="xref py py-class docutils literal notranslate"><span class="pre">Dict</span></code></p>
<dt class="field-even">Returns</dt>
<dd class="field-even"><p>Dictionary with log group name and server url to the log group</p>
<dl class="py method">
<dt id="nextcode.services.workflow.job.WorkflowJob.logs">
<code class="sig-name descname"><span class="pre">logs</span></code><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">log_group</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">'pod'</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">log_filter</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">None</span></span></em><span class="sig-paren">)</span><a class="reference internal" href="_modules/nextcode/services/workflow/job.html#WorkflowJob.logs"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#nextcode.services.workflow.job.WorkflowJob.logs" title="Permalink to this definition">¶</a></dt>
<dd><p>Get text logs for the specified log group</p>
<dl class="field-list simple">
<dt class="field-odd">Parameters</dt>
<dd class="field-odd"><ul class="simple">
<li><p><strong>log_group</strong> (<code class="xref py py-class docutils literal notranslate"><span class="pre">str</span></code>) – Name of the log group to view</p></li>
<li><p><strong>log_filter</strong> (<code class="xref py py-data docutils literal notranslate"><span class="pre">Optional</span></code>[<code class="xref py py-class docutils literal notranslate"><span class="pre">str</span></code>]) – Optional filter to apply to the logs</p></li>
<dt class="field-even">Raises</dt>
<dd class="field-even"><p><code class="xref py py-exc docutils literal notranslate"><span class="pre">ServerError</span></code> If the log group is not available</p>
<dt class="field-odd">Return type</dt>
<dd class="field-odd"><p><code class="xref py py-class docutils literal notranslate"><span class="pre">str</span></code></p>
<dl class="py method">
<dt id="nextcode.services.workflow.job.WorkflowJob.processes">
<code class="sig-name descname"><span class="pre">processes</span></code><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">process_id</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">None</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">is_all</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">False</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">limit</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">50</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">status</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">None</span></span></em><span class="sig-paren">)</span><a class="reference internal" href="_modules/nextcode/services/workflow/job.html#WorkflowJob.processes"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#nextcode.services.workflow.job.WorkflowJob.processes" title="Permalink to this definition">¶</a></dt>
<dd><p>Get a list of all nextflow processes in this job</p>
<dl class="field-list simple">
<dt class="field-odd">Parameters</dt>
<dd class="field-odd"><ul class="simple">
<li><p><strong>process_id</strong> (<code class="xref py py-data docutils literal notranslate"><span class="pre">Optional</span></code>[<code class="xref py py-class docutils literal notranslate"><span class="pre">int</span></code>]) – process_id to show</p></li>
<li><p><strong>is_all</strong> (<code class="xref py py-class docutils literal notranslate"><span class="pre">bool</span></code>) – Show all processes, otherwise show only running processes</p></li>
<li><p><strong>limit</strong> (<code class="xref py py-class docutils literal notranslate"><span class="pre">int</span></code>) – Maximum number of processes to return</p></li>
<li><p><strong>status</strong> (<code class="xref py py-data docutils literal notranslate"><span class="pre">Optional</span></code>[<code class="xref py py-class docutils literal notranslate"><span class="pre">str</span></code>]) – Filter processes by status</p></li>
<dt class="field-even">Return type</dt>
<dd class="field-even"><p><code class="xref py py-class docutils literal notranslate"><span class="pre">List</span></code>[<code class="xref py py-class docutils literal notranslate"><span class="pre">Dict</span></code>]</p>
<dl class="py method">
<dt id="nextcode.services.workflow.job.WorkflowJob.refresh">
<code class="sig-name descname"><span class="pre">refresh</span></code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="_modules/nextcode/services/workflow/job.html#WorkflowJob.refresh"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#nextcode.services.workflow.job.WorkflowJob.refresh" title="Permalink to this definition">¶</a></dt>
<dd><p>Refresh the local cache of the serverside job object</p>
<dl class="field-list simple">
<dt class="field-odd">Return type</dt>
<dd class="field-odd"><p><code class="xref py py-obj docutils literal notranslate"><span class="pre">None</span></code></p>
<dl class="py method">
<dt id="nextcode.services.workflow.job.WorkflowJob.resume">
<code class="sig-name descname"><span class="pre">resume</span></code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="_modules/nextcode/services/workflow/job.html#WorkflowJob.resume"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#nextcode.services.workflow.job.WorkflowJob.resume" title="Permalink to this definition">¶</a></dt>
<dd><p>Rerun a job that has previously failed</p>
<dl class="field-list simple">
<dt class="field-odd">Return type</dt>
<dd class="field-odd"><p><code class="xref py py-obj docutils literal notranslate"><span class="pre">None</span></code></p>
<dl class="py method">
<dt id="nextcode.services.workflow.job.WorkflowJob.running">
<em class="property"><span class="pre">property</span> </em><code class="sig-name descname"><span class="pre">running</span></code><a class="headerlink" href="#nextcode.services.workflow.job.WorkflowJob.running" title="Permalink to this definition">¶</a></dt>
<dd><p>Is the job currently running</p>
<dl class="field-list simple">
<dt class="field-odd">Return type</dt>
<dd class="field-odd"><p><code class="xref py py-class docutils literal notranslate"><span class="pre">bool</span></code></p>
<dl class="py method">
<dt id="nextcode.services.workflow.job.WorkflowJob.wait">
<code class="sig-name descname"><span class="pre">wait</span></code><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">max_seconds</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">None</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">poll_period</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">0.5</span></span></em><span class="sig-paren">)</span><a class="reference internal" href="_modules/nextcode/services/workflow/job.html#WorkflowJob.wait"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#nextcode.services.workflow.job.WorkflowJob.wait" title="Permalink to this definition">¶</a></dt>
<dd><p>Wait for a running job to complete.</p>
<p>This is similar to the wait method in the Query Service and will wait by default
indefinitely for a workflow job to complete and poll the server regularly to update
the local status. When the job is completed (or failed) the method will return the
workflow job object.</p>
<dl class="field-list simple">
<dt class="field-odd">Parameters</dt>
<dd class="field-odd"><ul class="simple">
<li><p><strong>max_seconds</strong> (<code class="xref py py-data docutils literal notranslate"><span class="pre">Optional</span></code>[<code class="xref py py-class docutils literal notranslate"><span class="pre">int</span></code>]) – raise an exception if the job runs longer than this</p></li>
<li><p><strong>poll_period</strong> (<code class="xref py py-class docutils literal notranslate"><span class="pre">float</span></code>) – Number of seconds to wait between polling (max 10 seconds)</p></li>
<dt class="field-even">Returns</dt>
<dd class="field-even"><p>WorkflowJob</p>
<dt class="field-odd">Raises</dt>
<dd class="field-odd"><p><code class="xref py py-exc docutils literal notranslate"><span class="pre">JobError</span></code></p>
<span class="target" id="module-nextcode.services.workflow.pytest_plugin"></span><div class="section" id="pytest-plugin">
<h2>Pytest plugin<a class="headerlink" href="#pytest-plugin" title="Permalink to this headline">¶</a></h2>
<p>This is a pytest plugin that is used by workflow CI jobs for automatically testing against a live server.</p>
<span class="target" id="module-nextcode.services.workflow.exceptions"></span><div class="section" id="exceptions">
<h2>Exceptions<a class="headerlink" href="#exceptions" title="Permalink to this headline">¶</a></h2>
<p>Custom exceptions raised by the workflow service.</p>
<dl class="py exception">
<dt id="nextcode.services.workflow.exceptions.JobError">
<em class="property"><span class="pre">exception</span> </em><code class="sig-prename descclassname"><span class="pre">nextcode.services.workflow.exceptions.</span></code><code class="sig-name descname"><span class="pre">JobError</span></code><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">message</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">job_id</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">None</span></span></em><span class="sig-paren">)</span><a class="reference internal" href="_modules/nextcode/services/workflow/exceptions.html#JobError"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#nextcode.services.workflow.exceptions.JobError" title="Permalink to this definition">¶</a></dt>
<dd><dl class="py method">
<dt id="nextcode.services.workflow.exceptions.JobError.with_traceback">
<code class="sig-name descname"><span class="pre">with_traceback</span></code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#nextcode.services.workflow.exceptions.JobError.with_traceback" title="Permalink to this definition">¶</a></dt>
<dd><p>Exception.with_traceback(tb) –
set self.__traceback__ to tb and return self.</p>
<footer class="footer">
  <div class="container">
    <p class="pull-right">
      <a href="#">Back to top</a>
        <br/>
<div id="sourcelink">
  <a href="_sources/api_workflow.rst.txt"
     rel="nofollow">Source</a>
        &copy; Copyright 2021,Genuity Science.<br/>
      Last updated on 2021-04-20.<br/>
      Created using <a href="http://sphinx-doc.org/">Sphinx</a> 3.5.4.<br/>
FilesExpand file tree

api_workflow.html

Latest commit

History

api_workflow.html

File metadata and controls
