Repository navigation
[Bug]: Unreleased global_exports lock on Postgres #3881
Description
Steps to reproduce
Do any of these actions several times:
- Create a project.
- Create a global export.
- Promote an existing export to global.
Actual behaviour
The first action succeeds. A few more may succeed. But at some point, all successive attempts start to hang indefinitely.
$ dstack export --project test-1 create test-1 --global
NAME FLEETS GATEWAYS IMPORTERS
test-1 - - *
$ dstack export --project test-1 create test-2 --global
NAME FLEETS GATEWAYS IMPORTERS
test-2 - - *
$ dstack export --project test-1 create test-3 --global
NAME FLEETS GATEWAYS IMPORTERS
test-3 - - *
$ dstack export --project test-1 create test-4 --global
⠧ Creating export...Expected behaviour
No hanging.
dstack version
master
Server logs
Additional information
Postgres logs show that at some point the global_exports advisory lock fails to be released, because it is attempted to be released by a connection different than the one that acquired it.
2026-05-14 12:08:35.800 UTC [404] LOG: execute __asyncpg_stmt_ab__: SELECT pg_advisory_lock($1::BIGINT) AS pg_advisory_lock_1
2026-05-14 12:08:35.800 UTC [404] DETAIL: Parameters: $1 = '6333502497592769007'
2026-05-14 12:08:35.840 UTC [404] LOG: execute __asyncpg_stmt_2a6__: SELECT projects.id, projects.name, projects.owner_id, users_1.id AS id_1, users_1.name AS name_1, users_1.created_at, users_1.token, users_1.token_hash, users_1.global_role, users_1.active, users_1.deleted, users_1.original_name, users_1.ssh_private_key, users_1.ssh_public_key, users_1.email, users_1.projects_quota
FROM projects LEFT OUTER JOIN users AS users_1 ON users_1.id = projects.owner_id
WHERE projects.deleted = false
2026-05-14 12:08:35.897 UTC [404] LOG: execute __asyncpg_stmt_2a7__: UPDATE exports SET is_global=$1::BOOLEAN WHERE exports.id = $2::UUID
2026-05-14 12:08:35.897 UTC [404] DETAIL: Parameters: $1 = 't', $2 = 'f3c448fe-19c2-4d8e-a945-276001dac9a7'
2026-05-14 12:08:35.898 UTC [400] LOG: statement: COMMIT;
2026-05-14 12:08:35.917 UTC [404] LOG: execute __asyncpg_stmt_2a8__: INSERT INTO exported_gateways (id, export_id, gateway_id) VALUES ($1::UUID, $2::UUID, $3::UUID)
2026-05-14 12:08:35.917 UTC [404] DETAIL: Parameters: $1 = 'e429087b-c12c-4b07-93ff-d9f6d1d8dfe3', $2 = 'f3c448fe-19c2-4d8e-a945-276001dac9a7', $3 = '2dc8a0c4-55df-4063-805c-db58ac557fec'
2026-05-14 12:08:35.951 UTC [404] LOG: execute __asyncpg_stmt_2a9__: INSERT INTO imports (id, project_id, export_id, created_at) VALUES ($1::UUID, $2::UUID, $3::UUID, $4::TIMESTAMP WITHOUT TIME ZONE), ($5::UUID, $6::UUID, $7::UUID, $8::TIMESTAMP WITHOUT TIME ZONE), ($9::UUID, $10::UUID, $11::UUID, $12::TIMESTAMP WITHOUT TIME ZONE)
2026-05-14 12:08:35.951 UTC [404] DETAIL: Parameters: $1 = '3d5c5944-150d-4307-8a4d-96125a2a1911', $2 = '9be9ffb9-2e03-4e09-a363-5aee2cb0d890', $3 = 'f3c448fe-19c2-4d8e-a945-276001dac9a7', $4 = '2026-05-14 12:08:35.943889', $5 = '6d8558be-6579-4609-a429-b432a13b231d', $6 = '4a556b2a-0d47-4cfb-ba7e-d595a14351c4', $7 = 'f3c448fe-19c2-4d8e-a945-276001dac9a7', $8 = '2026-05-14 12:08:35.943904', $9 = 'a11df706-92bf-46d6-be55-d4d132f523d3', $10 = '6adacd1a-fb78-4685-b460-1f8ea9d60adc', $11 = 'f3c448fe-19c2-4d8e-a945-276001dac9a7', $12 = '2026-05-14 12:08:35.943915'
2026-05-14 12:08:35.961 UTC [404] LOG: statement: COMMIT;
2026-05-14 12:08:35.976 UTC [395] LOG: statement: BEGIN;
2026-05-14 12:08:35.983 UTC [395] LOG: execute __asyncpg_stmt_113__: SELECT pg_advisory_unlock($1::BIGINT) AS pg_advisory_unlock_1
2026-05-14 12:08:35.983 UTC [395] DETAIL: Parameters: $1 = '6333502497592769007'
2026-05-14 12:08:35.983 UTC [395] WARNING: you don't own a lock of type ExclusiveLock
Activity