dstack is an open-source tool that automates ML workflows, enabling effective management on any cloud platform.

It empowers your team to prepare data, train, and fine-tune models using their preferred frameworks and dev environments without spending time on engineering and infrastructure.

Use pip to install dstack:

pip install dstack

By default, workflows run locally. To run workflows remotely (e.g. in a configured cloud account), configure a remote using the dstack config command.

dstack config

? Choose backend. Use arrows to move, type to filter
> [aws]
  [gcp]
  [hub]

Choose hub if you prefer managing cloud credentials and settings through a user interface while working in a team.

For running remote workflows with local cloud credentials, select aws or gcp.

Define ML workflows, their output artifacts, hardware requirements, and dependencies via YAML.

workflows:
  - name: train-mnist
    provider: bash
    commands:
      - pip install torchvision pytorch-lightning tensorboard
      - python examples/mnist/train_mnist.py
    artifacts:
      - path: ./lightning_logs

By default, workflows run locally on your machine.

dstack run train-mnist

RUN        WORKFLOW     SUBMITTED  STATUS     TAG  BACKENDS
penguin-1  train-mnist  now        Submitted       local

Provisioning... It may take up to a minute. ✓

To interrupt, press Ctrl+C.

GPU available: True, used: True

Epoch 1: [00:03<00:00, 280.17it/s, loss=1.35, v_num=0]

To run a workflow remotely (e.g. in a configured cloud account), add the --remote flag to the dstack run command:

The necessary hardware resources can be configured either via YAML or through arguments in the dstack run command, such as --gpu and --gpu-name.

dstack run train-mnist --remote --gpu 1

RUN       WORKFLOW     SUBMITTED  STATUS     TAG  BACKENDS
turtle-1  train-mnist  now        Submitted       aws

Provisioning... It may take up to a minute. ✓

To interrupt, press Ctrl+C.

GPU available: True, used: True

Epoch 1: [00:03<00:00, 280.17it/s, loss=1.35, v_num=0]

Upon running a workflow remotely, dstack automatically creates resources in the configured cloud account and destroys them once the workflow is complete.

dstack supports multiple providers to set up environments, run scripts, and launch interactive development environments and applications.

dstack allows you to save output artifacts and conveniently reuse them across workflows.

For additional information and examples, see the following links:

• Docs
• Quick start
• Playground
• Setup
• Usage
• Examples

Mozilla Public License 2.0