Skip to content

[Bug]: Blocks not released when jobs stop concurrently #3956

Open
Open
[Bug]: Blocks not released when jobs stop concurrently#3956
Labels
bugSomething isn't workingstale

Description

@jvstme
jvstme
opened on Jun 11, 2026

Steps to reproduce

  1. Create this fleet: 
    type: fleet
name: test-fleet
nodes: 1
blocks: 2
resources:
  cpu: 2
backends: [aws]
  2. Run this service: 
    type: service
name: test-service
image: nginx
port: 80
replicas: 2
gateway: false
fleets: [test-fleet]
resources:
  cpu: 1..
  memory: 1GB..
  3. Wait until both replicas are running.
  4. Stop the service.

Actual behaviour

One of the blocks is still identified as busy, although no jobs are running on the instance.

$ dstack event --within-fleet test-fleet
[2026-06-11 13:25:22] [👤admin] [fleet test-fleet] Fleet created. Status: ACTIVE
[2026-06-11 13:25:22] [instance test-fleet-0] Instance created on fleet submission. Status: PENDING
[2026-06-11 13:25:41] [instance test-fleet-0] Instance status changed PENDING -> PROVISIONING
[2026-06-11 13:26:12] [instance test-fleet-0] Instance status changed PROVISIONING -> IDLE
[2026-06-11 13:28:30] [instance test-fleet-0] Instance status changed IDLE -> BUSY
[2026-06-11 13:28:30] [job test-service-0-0, instance test-fleet-0] Job assigned to instance. Instance blocks: 1/2 busy
[2026-06-11 13:28:44] [job test-service-0-1, instance test-fleet-0] Job assigned to instance. Instance blocks: 2/2 busy
[2026-06-11 13:31:26] [job test-service-0-0, instance test-fleet-0] Job unassigned from instance. Instance blocks: 1/2 busy
[2026-06-11 13:31:26] [job test-service-0-1, instance test-fleet-0] Job unassigned from instance. Instance blocks: 1/2 busy

Expected behaviour

All blocks are idle.

dstack version

0.20.23

Server logs

[13:31:12] DEBUG    dstack._internal.server.background.pipeline_tasks.base:357 Processing jobs item a53688cc-6807-45d7-bb11-af8757805f15
           DEBUG    dstack._internal.server.background.pipeline_tasks.base:357 Processing jobs item 53edc4c0-3f15-48af-846f-65666db52a4f
           DEBUG    dstack._internal.server.services.jobs:355 job(a53688)test-service-0-0: stopping runner
           DEBUG    dstack._internal.server.services.jobs:355 job(53edc4)test-service-0-1: stopping runner
[13:31:13] ERROR    dstack._internal.server.background.pipeline_tasks.jobs_terminating:525 Failed to update related instance 5a674418-03f3-41a3-a400-962d8a3bd845 for
                    terminating job 53edc4c0-3f15-48af-846f-65666db52a4f.
           INFO     dstack._internal.server.services.events:205 Emitting event: Graceful job stop requested. Event targets: job(53edc4)test-service-0-1. Actor: system
           DEBUG    dstack._internal.server.background.pipeline_tasks.base:364 Processed jobs item 53edc4c0-3f15-48af-846f-65666db52a4f in 0.982
           INFO     dstack._internal.server.services.events:205 Emitting event: Graceful job stop requested. Event targets: job(a53688)test-service-0-0. Actor: system
           DEBUG    dstack._internal.server.background.pipeline_tasks.base:364 Processed jobs item a53688cc-6807-45d7-bb11-af8757805f15 in 0.999
[13:31:16] DEBUG    dstack._internal.server.background.pipeline_tasks.base:357 Processing runs item 9e74d3bc-076a-4a63-aeb6-c52aee9b51e4
           DEBUG    dstack._internal.server.background.pipeline_tasks.base:364 Processed runs item 9e74d3bc-076a-4a63-aeb6-c52aee9b51e4 in 0.028
[13:31:24] DEBUG    dstack._internal.server.background.pipeline_tasks.base:357 Processing runs item 9e74d3bc-076a-4a63-aeb6-c52aee9b51e4
           DEBUG    dstack._internal.server.background.pipeline_tasks.base:364 Processed runs item 9e74d3bc-076a-4a63-aeb6-c52aee9b51e4 in 0.027
           DEBUG    dstack._internal.server.background.pipeline_tasks.base:357 Processing jobs item 53edc4c0-3f15-48af-846f-65666db52a4f
           DEBUG    dstack._internal.server.background.pipeline_tasks.base:357 Processing jobs item a53688cc-6807-45d7-bb11-af8757805f15
           DEBUG    dstack._internal.server.background.pipeline_tasks.jobs_terminating:663 job(53edc4)test-service-0-1: stopping container
           DEBUG    dstack._internal.server.background.pipeline_tasks.jobs_terminating:663 job(a53688)test-service-0-0: stopping container
[13:31:26] INFO     dstack._internal.server.services.events:205 Emitting event: Job status changed TERMINATING -> TERMINATED. Event targets: job(a53688)test-service-0-0.
                    Actor: system
           INFO     dstack._internal.server.services.events:205 Emitting event: Job unassigned from instance. Instance blocks: 1/2 busy. Event targets:
                    job(a53688)test-service-0-0, instance(5a6744)test-fleet-0. Actor: system
           INFO     dstack._internal.server.services.events:205 Emitting event: Service replica unregistered from receiving requests. Event targets:
                    job(a53688)test-service-0-0. Actor: system
           DEBUG    dstack._internal.server.background.pipeline_tasks.base:364 Processed jobs item a53688cc-6807-45d7-bb11-af8757805f15 in 1.654
           ERROR    dstack._internal.server.background.pipeline_tasks.jobs_terminating:525 Failed to update related instance 5a674418-03f3-41a3-a400-962d8a3bd845 for
                    terminating job 53edc4c0-3f15-48af-846f-65666db52a4f.
           INFO     dstack._internal.server.services.events:205 Emitting event: Job status changed TERMINATING -> TERMINATED. Event targets: job(53edc4)test-service-0-1.
                    Actor: system
           INFO     dstack._internal.server.services.events:205 Emitting event: Job unassigned from instance. Instance blocks: 1/2 busy. Event targets:
                    job(53edc4)test-service-0-1, instance(5a6744)test-fleet-0. Actor: system
           INFO     dstack._internal.server.services.events:205 Emitting event: Service replica unregistered from receiving requests. Event targets:
                    job(53edc4)test-service-0-1. Actor: system
           DEBUG    dstack._internal.server.background.pipeline_tasks.base:364 Processed jobs item 53edc4c0-3f15-48af-846f-65666db52a4f in 1.687
           DEBUG    dstack._internal.server.background.pipeline_tasks.base:357 Processing fleets item d90f5d4a-5594-4373-8012-7340605bdc4a
           DEBUG    dstack._internal.server.background.pipeline_tasks.base:364 Processed fleets item d90f5d4a-5594-4373-8012-7340605bdc4a in 0.051
           DEBUG    dstack._internal.server.background.pipeline_tasks.base:357 Processing runs item 9e74d3bc-076a-4a63-aeb6-c52aee9b51e4
           INFO     dstack._internal.server.services.events:205 Emitting event: Run status changed TERMINATING -> TERMINATED. Event targets: run(9e74d3)test-service. Actor:
                    system

Additional information

No response

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingstale

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions