Skip to content

[Bug]: Python API Run.logs() returns nothing for a finished run #4175

Closed

Description

@r4victor

Steps to reproduce

from dstack.api import Client
c = Client.from_config()
r = c.runs.get("<a run that has finished>")
print(len(b"".join(r.logs())))                 # 0
print(len(b"".join(r.logs(replica_num=0))))    # the actual logs

Actual behaviour

logs() yields nothing for any terminal run - no exception, no warning. dstack logs <same run> prints the logs in full.

_find_job matches on replica_num alone when it is given, but when it is None it matches only a submission whose status is RUNNING:

def _find_job(self, replica_num: Optional[int], job_num: int) -> Optional[Job]:
for j in self._run.jobs:
if (
replica_num is not None
and j.job_spec.replica_num == replica_num
or replica_num is None
and j.job_submissions[-1].status == JobStatus.RUNNING
) and j.job_spec.job_num == job_num:
return j
return None

Once a run is terminal nothing is RUNNING, so _find_job returns None and logs() returns an empty generator. The CLI is unaffected because its --replica argparse default is 0, so it never passes None.

Expected behaviour

logs() with default arguments should return the logs of a finished run, as the CLI does - fall back to replica 0 when no submission is running, or raise instead of yielding nothing.

dstack version

master at 2517069, server run from source.

Activity

  1. r4victor commented on Aug 20, 2026

    @r4victor
    CollaboratorAuthor

    Some history, so the fix doesn't undo the rolling-deploy behaviour.

    Both logs() and attach() defaulted to replica_num=0 with no status check until #2821 (rolling deployment for services) changed them to None and added the RUNNING condition, documented as "any running replica". That's right for attach: under a rolling deploy replica 0 may be draining or gone, and you can't tunnel into a dead container anyway.

    It's wrong only for logs(), which shares _find_job but should outlive the replica. So the fallback belongs in the logs path; attach should keep matching any running replica.

  2. self-assigned this
    on Aug 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions