Repository navigation
[Bug]: Job log tail lost on dstack stop #4285
Description
Steps to reproduce
type: task
name: task
commands:
- nohup sleep inf &
- |
for e in $(seq 100000); do
printf '%07d\n' $e
sleep 1
donedstack apply, wait until some logs produced, then dstack stop.
Actual behaviour
~/.dstack/runners/task-0-0-a8fbccc0/runner/job.log:
[...]
0000053
0000054
0000055
0000056
^C
dstack logs task (7 records + ^C lost):
[...]
0000045
0000046
0000047
0000048
0000049
~/.dstack/runners/task-0-0-a8fbccc0/runner/runner.log
[...]
time=2026-09-11T04:10:11.541553-04:00 level=info msg=Job state changed new=running
time=2026-09-11T04:10:11.541585-04:00 level=trace msg=Using credentials user=uid=0(root) gid=0 groups=[0] home=/root
time=2026-09-11T04:10:11.544841-04:00 level=info msg=Required capability is missing, cannot set resource limits missing=SYS_RESOURCE
time=2026-09-11T04:10:11.54491-04:00 level=trace msg=Starting exec cmd=[...]
time=2026-09-11T04:11:15.361874-04:00 level=warning msg=Processes still running after the interrupt, sent SIGHUP pids=[78]
time=2026-09-11T04:11:20.38817-04:00 level=error msg=Processes still running after SIGHUP, sent SIGKILL pids=[78]
time=2026-09-11T04:11:20.388462-04:00 level=error msg=Job canceled
time=2026-09-11T04:11:20.388758-04:00 level=info msg=Job state changed new=terminated
time=2026-09-11T04:11:20.388776-04:00 level=error msg=Executor failed err=job canceled: wait for command: exit status 1
dstack logs -d task (termination messages lost):
[...]
time=2026-09-11T04:10:11.541553-04:00 level=info msg=Job state changed new=running
time=2026-09-11T04:10:11.541585-04:00 level=trace msg=Using credentials user=uid=0(root) gid=0 groups=[0] home=/root
time=2026-09-11T04:10:11.544841-04:00 level=info msg=Required capability is missing, cannot set resource limits missing=SYS_RESOURCE
time=2026-09-11T04:10:11.54491-04:00 level=trace msg=Starting exec cmd=[...]
Expected behaviour
No response
dstack version
Server logs
Additional information
No response
Activity
Collect the last job logs before killing the container (dstackai#4289)