[Bug]: Dynamo example's etcd/NATS wait loop never completes when a worker group uses a custom image

Steps to reproduce

Take the Dynamo example configuration from https://dstack.ai/docs/examples/inference/dynamo/#apply-a-configuration.
In a worker group, replace python: "3.12" / nvcc: true with a custom image (e.g. image: vllm/vllm-openai:latest or any Debian/Ubuntu-based image) and switch the engine commands accordingly. Keep the etcd/NATS wait loop from the example as is.
dstack apply -f the configuration.

Actual behaviour

The router replica comes up fine (etcd, NATS and the frontend start), but the worker replica never gets past the wait loop and prints this every 3 seconds until stopped:

waiting for etcd/NATS on 192.168.0.44...

This looks like a network problem between the router and the workers, but it isn't: from the worker host both ports are reachable, and curl http://$DSTACK_ROUTER_INTERNAL_IP:2379/health inside the worker container returns {"health":"true"}.

The loop fails because the example probes the ports with bash's /dev/tcp redirection:

dstack/mkdocs/docs/examples/inference/dynamo.md

Lines 62 to 68 in ae1c7cc

    
                 - export DYN_SYSTEM_PORT="8000" 
        
                 # Wait until the router's etcd and NATS ports are actually accepting connections. 
        
                 - | 
        
                   until (echo > /dev/tcp/$DSTACK_ROUTER_INTERNAL_IP/2379) 2>/dev/null \ 
        
                      && (echo > /dev/tcp/$DSTACK_ROUTER_INTERNAL_IP/4222) 2>/dev/null; do 
        
                     echo "waiting for etcd/NATS on $DSTACK_ROUTER_INTERNAL_IP..."; sleep 3 
        
                   done

When a group sets its own image, dstack runs the group's commands with /bin/sh -i -c:

dstack/src/dstack/_internal/server/services/jobs/configurators/service.py

Lines 65 to 83 in ae1c7cc

    
           def _shell(self) -> str: 
        
               # Shell resolution order: 
        
               #   1. If `shell:` is set explicitly, the base honors it. 
        
               #   2. If this group sets `docker: true`, use /bin/bash — the 
        
               #      DIND image ships bash, matching the service-level path. 
        
               #   3. If this group sets its own `image`, force /bin/sh. The 
        
               #      base returns /bin/bash when service-level `image` is None, 
        
               #      but a group-level custom image (e.g. alpine) may not ship 
        
               #      bash. 
        
               #   4. Otherwise defer to the base (bash for dstackai/base, sh 
        
               #      for a service-level custom image). 
        
               if self.run_spec.configuration.shell is None: 
        
                   group = self._current_replica_group() 
        
                   if group is not None: 
        
                       if group.docker is True: 
        
                           return "/bin/bash" 
        
                       if group.image is not None: 
        
                           return "/bin/sh" 
        
               return super()._shell()

In Debian/Ubuntu-based images /bin/sh is dash, which has no /dev/tcp, so the probe fails regardless of connectivity. The wait loop swallows the error with 2>/dev/null, so nothing hints at the cause. Running the same probe inside the container shows it:

$ sh -c 'echo > /dev/tcp/192.168.0.44/2379'
sh: 1: cannot create /dev/tcp/192.168.0.44/2379: Directory nonexistent

The published example works only because its worker groups use the dstack base image, where the shell is bash.

Expected behaviour

The example should work when a worker group uses a custom image, since that is the natural way to plug in a different engine. Options:

Make the wait loop POSIX, e.g. probe etcd with curl -sf http://$DSTACK_ROUTER_INTERNAL_IP:2379/health and NATS with nc -z $DSTACK_ROUTER_INTERNAL_IP 4222 (or a small python3 socket check, which is available in every engine image).
Or set shell: bash at the top level of the example and mention why. Note that shell is a top-level field and can't be set per group.
Drop the 2>/dev/null from the probe so a failure like the one above is visible in the logs.

dstack version

master (after 0.21.3)

Server logs

No response

Additional information

Related: the shell field docs say it defaults to /bin/sh when image is specified, but the per-group image case isn't mentioned in the groups docs, and the Dynamo example doesn't warn that its commands are bash-only.

Navigation Menu

Repository navigation

[Bug]: Dynamo example's etcd/NATS wait loop never completes when a worker group uses a custom image #4249

Steps to reproduce

Actual behaviour

Expected behaviour

dstack version

Server logs

Additional information

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

	- export DYN_SYSTEM_PORT="8000"
	# Wait until the router's etcd and NATS ports are actually accepting connections.
	- \|
	until (echo > /dev/tcp/$DSTACK_ROUTER_INTERNAL_IP/2379) 2>/dev/null \
	&& (echo > /dev/tcp/$DSTACK_ROUTER_INTERNAL_IP/4222) 2>/dev/null; do
	echo "waiting for etcd/NATS on $DSTACK_ROUTER_INTERNAL_IP..."; sleep 3
	done

	def _shell(self) -> str:
	# Shell resolution order:
	# 1. If `shell:` is set explicitly, the base honors it.
	# 2. If this group sets `docker: true`, use /bin/bash — the
	# DIND image ships bash, matching the service-level path.
	# 3. If this group sets its own `image`, force /bin/sh. The
	# base returns /bin/bash when service-level `image` is None,
	# but a group-level custom image (e.g. alpine) may not ship
	# bash.
	# 4. Otherwise defer to the base (bash for dstackai/base, sh
	# for a service-level custom image).
	if self.run_spec.configuration.shell is None:
	group = self._current_replica_group()
	if group is not None:
	if group.docker is True:
	return "/bin/bash"
	if group.image is not None:
	return "/bin/sh"
	return super()._shell()

[Bug]: Dynamo example's etcd/NATS wait loop never completes when a worker group uses a custom image #4249

Description

Steps to reproduce

Actual behaviour

Expected behaviour

dstack version

Server logs

Additional information

Activity

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions