Repository navigation
[Bug]: Occasionally dstack run throws an exception #294
Description
dstack version
0.2.2
Python version
Any
Host OS
Any
Host Arch
Any
What happened?
Occasionally, dstack run throws the following error:
Traceback (most recent call last):
File "/workspace/.pyenv_mirror/user/3.8.16/bin/dstack", line 8, in <module>
sys.exit(main())
File "/workspace/.pyenv_mirror/user/current/lib/python3.8/site-packages/dstack/cli/main.py", line 50, in main
args.func(args)
File "/workspace/.pyenv_mirror/user/current/lib/python3.8/site-packages/dstack/cli/commands/__init__.py", line 41, in __command
self._command(args)
File "/workspace/.pyenv_mirror/user/current/lib/python3.8/site-packages/dstack/core/error.py", line 31, in decorator
func(*args, **kwargs)
File "/workspace/.pyenv_mirror/user/current/lib/python3.8/site-packages/dstack/core/error.py", line 41, in decorator
func(*args, **kwargs)
File "/workspace/.pyenv_mirror/user/current/lib/python3.8/site-packages/dstack/core/error.py", line 51, in decorator
func(*args, **kwargs)
File "/workspace/.pyenv_mirror/user/current/lib/python3.8/site-packages/dstack/cli/commands/run/__init__.py", line 347, in _command
poll_run(
File "/workspace/.pyenv_mirror/user/current/lib/python3.8/site-packages/dstack/cli/commands/run/__init__.py", line 219, in poll_run
poll_logs_ws(backend, repo_address, jobs[0], ports)
File "/workspace/.pyenv_mirror/user/current/lib/python3.8/site-packages/dstack/cli/commands/run/__init__.py", line 134, in poll_logs_ws
_job_head = backend.get_job(repo_address, job.job_id)
File "/workspace/.pyenv_mirror/user/current/lib/python3.8/site-packages/dstack/backend/local/__init__.py", line 52, in get_job
return base_jobs.get_job(self._storage, repo_address, job_id)
File "/workspace/.pyenv_mirror/user/current/lib/python3.8/site-packages/dstack/backend/base/jobs.py", line 31, in get_job
job = Job.unserialize(yaml.load(obj, yaml.FullLoader))
File "/workspace/.pyenv_mirror/user/current/lib/python3.8/site-packages/dstack/core/job.py", line 327, in unserialize
_requirements = job_data.get("requirements")
AttributeError: 'NoneType' object has no attribute 'get'
This is likely to happen because the runner updates the local job YAML file not atomically.
Possible solution: use https://github.com/google/renameio
Relevant log output
No response
Additional Information
No response
Activity