Repository navigation
Implement Fleets - #1415
Implement Fleets#1415
Conversation
This migration did work when applied individually, but if all migrations were applied to a new PostgreSQL database at once, it failed with `sqlalchemy.exc.NoSuchTableError: repos` when trying to autodetect the `repos` table description.
| The default pool is created automatically. | ||
|
|
||
| !!! warning "Deprecation" | ||
| Pools are deprecated in favor of fleets and will be removed in 0.19.0. |
There was a problem hiding this comment.
(nit) Maybe mention it concepts/pools.md too?
Co-authored-by: jvstme <[email protected]>
|
@jvstme, Thanks for the review. I'm going to merge so that everyone can try fleets already. If any non-trivial changes are required, they'll will probably land next week. |
|
|
||
| async def _terminate_fleet_instances(fleet_model: FleetModel, instance_nums: Optional[List[int]]): | ||
| if is_fleet_in_use(fleet_model, ignore_instance_nums=instance_nums): | ||
| raise ServerClientError(f"Failed to delete fleet {fleet_model.name}. Fleet is in use.") |
There was a problem hiding this comment.
Is it expected that users cannot delete an instance if another instance in the same fleet is busy? This may waste resources, as users will have to keep the instances they no longer need.
> dstack fleet
FLEET INSTANCE BACKEND RESOURCES PRICE STATUS CREATED
massive-eel 0 aws (us-west-2) 2xCPU, 8GB, 100.0GB (disk), SPOT $0.031 busy 11 mins ago
2 aws (us-west-2) 2xCPU, 8GB, 100.0GB (disk), SPOT $0.031 idle 11 mins ago
> dstack fleet delete massive-eel -i 2
Delete the fleet massive-eel instances [2]? [y/n]: y
Failed to delete fleet massive-eel. Fleet is in use.The error message is also a bit misleading - the user was deleting an instance, not the whole fleet.
There was a problem hiding this comment.
That's a bug. I'll fix it.
| ) | ||
| if not confirmed and not args.yes: | ||
| if not confirm_ask( | ||
| f"Fleet [code]{conf.name}[/] does not exist yet. Create the fleet?" |
There was a problem hiding this comment.
> dstack apply -f fleet.dstack.yml
Fleet None does not exist yet. Create the fleet? [y/n]:I think unnamed fleet configurations can have a better confirmation prompt, e.g. Are you sure you want to create a new fleet?
| return value | ||
|
|
||
|
|
||
| class InstanceGroupParams(CoreModel): |
There was a problem hiding this comment.
Consider validating that SSH-specific and cloud-specific parameters are not specified at the same time. Currently, cloud-specific parameters would be ignored, which can be counterintuitive.
| ssh_key: Optional[SSHKey] = None | ||
|
|
||
|
|
||
| class SSHParams(CoreModel): |
There was a problem hiding this comment.
Consider allowing to specify region, as it is possible to specify it with dstack pool add-ssh.
There was a problem hiding this comment.
@jvstme region for dstack pool add-ssh can be used to group interconnected/non-interconnected instances. This doesn't seem to be necessary for fleets since instances are grouped by fleets. Why would one need to add ssh fleets and specify different regions?
There was a problem hiding this comment.
If each instance has a region, it will allow to group instances across fleets or differentiate instances within a fleet based on their geographic location.
Problem 1: I have 2 fleets in Europe and 2 fleets in the US. I need to run a configuration on any of the fleets, as long as it is in the US.
Solution 1: Specify region in the run configuration and allow dstack to choose one of the US fleets automatically.
Problem 2: I need to run a service with 3 replicas: 1 in Europe, 1 in the US, 1 in Asia. I also need to have one stand-by instance in each region in case the other instance in this region fails.
Solution 2: Create the my-fleet fleet with 6 instances: 2 in Europe, 2 in the US, 2 in Asia. In the service configuration, specify fleet: my-fleet, replicas: 3, placement: geo-distributed.
Closes #1327
This PR:
fleetconfiguration type to provision/delete cloud instances and add on-prem clusters usingdstack apply/dstack delete.dstack fleet list,dstack fleet delete.Tested:
dstack runworks as before and creates or reuses fleets.dstack poolcommands work as before.dstack applycreates cloud fleets.dstack applycreates ssh fleets.dstack runare auto-cleaned when no longer in use.