[Bug]: Runs may fail due to local backend updating files non-atomically #328

dstack version

master

Python version

3.11

Host OS

macOS 12.5

Host Arch

ARM

What happened?

We have an integration test of the run command that usually passes but failed once: https://github.com/dstackai/dstack/actions/runs/4696536596/jobs/8326767393

This non-deterministic behaviour is most probably due to local backend updating job file non-atomically – it just opens the file for writing even if it already exists. Thus, the reader may read the inconsistent state.

We fixed a similar issue with job head file renaming already (#296). The same principal with rename should be used to update file's contents.

Relevant log output

No response

Additional Information

No response