Repository navigation
[Bug]: Error creating AWS security group concurrently #4398
Description
Steps to reproduce
- Create a fresh AWS VPC.
- Configure it in
dstackbackend settings (vpc_name). - Provision an AWS fleet with two instances in the same region as the VPC.
Actual behaviour
One of the instances fails to provision the first offer and proceeds with the more expensive one.
Fleet test does not exist yet.
Create the fleet? [y/n]: y
⠏ Provisioning test...
NAME NODES GPU SPOT BACKEND PRICE STATUS CREATED
test 2 auto aws active 44 sec ago
instance=0 - on-demand aws (eu-west-1) $0.0057 provisioning 44 sec ago
instance=1 - spot aws (eu-west-1) $0.0014 provisioning 44 sec ago
Expected behaviour
Both instances successfully provision the first offer.
dstack version
0.22.4
Server logs
DEBUG dstack._internal.core.backends.aws.compute:341 Trying provisioning t3.nano in eu-west-1c
DEBUG dstack._internal.core.backends.aws.compute:341 Trying provisioning t3.nano in eu-west-1c
[14:30:33] ERROR dstack._internal.server.background.pipeline_tasks.instances.cloud_provisioning:202 Got exception when launching t3.nano in aws/eu-west-1
Traceback (most recent call last):
File "/dstack/src/dstack/_internal/server/background/pipeline_tasks/instances/cloud_provisioning.py", line 174, in create_cloud_instance
job_provisioning_data = await run_async(
File "/dstack/src/dstack/_internal/utils/common.py", line 56, in run_async
return await asyncio.get_running_loop().run_in_executor(None, func_with_args)
File "/usr/lib64/python3.10/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
File "/dstack/src/dstack/_internal/core/backends/aws/compute.py", line 353, in create_instance
security_group_id = self._create_security_group(
File "/dstack/.venv/lib64/python3.10/site-packages/cachetools/__init__.py", line 717, in wrapper
v = method(self, *args, **kwargs)
File "/dstack/src/dstack/_internal/core/backends/aws/compute.py", line 1194, in _create_security_group
return aws_resources.create_security_group(
File "/dstack/src/dstack/_internal/core/backends/aws/resources.py", line 85, in create_security_group
security_group = ec2_client.create_security_group(
File "/dstack/.venv/lib64/python3.10/site-packages/botocore/client.py", line 606, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/dstack/.venv/lib64/python3.10/site-packages/botocore/context.py", line 123, in wrapper
return func(*args, **kwargs)
File "/dstack/.venv/lib64/python3.10/site-packages/botocore/client.py", line 1094, in _make_api_call
raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (InvalidGroup.Duplicate) when calling the CreateSecurityGroup operation: The security group 'dstack_security_group_test' already exists for VPC 'vpc-0ca9dd689a5145b15'
DEBUG dstack._internal.server.background.pipeline_tasks.instances.cloud_provisioning:165 Trying t3.micro in aws/eu-west-1 for $0.0030 per hour
DEBUG dstack._internal.core.backends.aws.compute:341 Trying provisioning t3.micro in eu-west-1c
ERROR dstack._internal.server.background.pipeline_tasks.instances.cloud_provisioning:202 Got exception when launching t3.micro in aws/eu-west-1
Traceback (most recent call last):
File "/dstack/src/dstack/_internal/server/background/pipeline_tasks/instances/cloud_provisioning.py", line 174, in create_cloud_instance
job_provisioning_data = await run_async(
File "/dstack/src/dstack/_internal/utils/common.py", line 56, in run_async
return await asyncio.get_running_loop().run_in_executor(None, func_with_args)
File "/usr/lib64/python3.10/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
File "/dstack/src/dstack/_internal/core/backends/aws/compute.py", line 353, in create_instance
security_group_id = self._create_security_group(
File "/dstack/.venv/lib64/python3.10/site-packages/cachetools/__init__.py", line 717, in wrapper
v = method(self, *args, **kwargs)
File "/dstack/src/dstack/_internal/core/backends/aws/compute.py", line 1194, in _create_security_group
return aws_resources.create_security_group(
File "/dstack/src/dstack/_internal/core/backends/aws/resources.py", line 127, in create_security_group
_add_egress_security_group_rule_if_missing(
File "/dstack/src/dstack/_internal/core/backends/aws/resources.py", line 504, in _add_egress_security_group_rule_if_missing
ec2_client.authorize_security_group_egress(
File "/dstack/.venv/lib64/python3.10/site-packages/botocore/client.py", line 606, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/dstack/.venv/lib64/python3.10/site-packages/botocore/context.py", line 123, in wrapper
return func(*args, **kwargs)
File "/dstack/.venv/lib64/python3.10/site-packages/botocore/client.py", line 1094, in _make_api_call
raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (InvalidPermission.Duplicate) when calling the AuthorizeSecurityGroupEgress operation: the specified rule "peer: sg-0895665b449c7a8f2, ALL, ALLOW" already existsAdditional information
No response
Activity