Repository navigation
[Bug]: Custom probes may break worker registration with type: sglang router #4310
Description
Steps to reproduce
type: service
name: smg
env:
- MODEL_ID=Qwen/Qwen2.5-0.5B-Instruct
groups:
- replicas: 1
python: "3.12"
commands:
- pip install smg
- |
smg launch \
--host 0.0.0.0 \
--port 8000 \
--enable-igw \
--policy cache_aware \
--model-path $MODEL_ID
router:
type: sglang
resources:
gpu: 0
- replicas: 1
image: lmsysorg/sglang:v0.5.10.post1
commands:
- |
python3 -m sglang.launch_server \
--model-path $MODEL_ID \
--host 0.0.0.0 \
--port 8000 \
--context-length 4096
resources:
gpu:
memory: 16GB..
# SGLang's CuTe DSL kernels don't compile for Turing (sm_75, e.g. T4)
compute_capability: 8.0
port: 8000
model:
type: chat
name: Qwen/Qwen2.5-0.5B-Instruct
format: openai
prefix: /v1
probes:
- type: http
url: /health_generate
interval: 5s/health_generate is a correct probe for both Shepherd Model Gateway and SGLang Server – “Check the health [...] by generating one token” – roughly the same as dstack's _openai_model_probe_spec; but adding this probe reintroduces the same kind of deadlock that was fixed in #4270. Replacing the custom probe with probes: [] solves the issue.
Actual behaviour
The worker is ready, the router is never ready (see the PROBES column):
NAME BACKEND RESOURCES INSTANCE TYPE PRICE STATUS PROBES SUBMITTED ERROR
smg deployment=0 - - running 7 mins ago
group=0 replica=0 deployment=0 gcp (us-west1) cpu=x86:2 mem=2GB disk=20GB e2-highcpu-2 $0.0495 running × 7 mins ago
group=1 replica=1 deployment=0 gcp (us-west1) cpu=x86:4 mem=16GB disk=100GB gpu=L4:24GB:1 g2-standard-4 $0.7068 running ✓ 7 mins ago
Server logs:
# Both are not ready
dstack._internal.server.background.scheduled_tasks.probes:122 probe(fa83a6)smg-0-0-0: probe status code: 503
dstack._internal.server.background.scheduled_tasks.probes:125 probe(ea13f1)smg-0-1-0: probe failed: ReadError('')
# Both are not ready
dstack._internal.server.background.scheduled_tasks.probes:122 probe(fa83a6)smg-0-0-0: probe status code: 503
dstack._internal.server.background.scheduled_tasks.probes:122 probe(ea13f1)smg-0-1-0: probe status code: 503
# The worker is ready; the router returns 503 infinitely as there is no registered worker
dstack._internal.server.background.scheduled_tasks.probes:122 probe(fa83a6)smg-0-0-0: probe status code: 503
dstack._internal.server.background.scheduled_tasks.probes:122 probe(ea13f1)smg-0-1-0: probe status code: 200
/health_generate response from the router:
GET /health_generate HTTP/1.1
Accept: */*
Accept-Encoding: gzip, deflate
Connection: keep-alive
Host: localhost:8000
User-Agent: HTTPie/3.2.4
HTTP/1.1 503 Service Unavailable
access-control-allow-origin: *
access-control-expose-headers: *
content-length: 110
content-type: application/json
date: Thu, 24 Sep 2026 09:19:06 GMT
x-request-id: req-N5bKymgO02OuEHl01IoT6pE5
x-smg-error-code: no_workers
{
"error": {
"code": "no_workers",
"message": "No workers are available",
"param": null,
"type": "Service Unavailable"
}
}
/health_generate response from the worker:
GET /health_generate HTTP/1.1
Accept: */*
Accept-Encoding: gzip, deflate
Connection: keep-alive
Host: localhost:8000
User-Agent: HTTPie/3.2.4
HTTP/1.1 200 OK
content-length: 0
date: Thu, 24 Sep 2026 09:19:59 GMT
server: uvicorn
Expected behaviour
No response
dstack version
Server logs
DEBUG dstack._internal.server.services.runs.router_worker_sync:568 run(f7ba4f)smg: no ready router job in group 0, skipping worker syncAdditional information
No response
Activity
Don't require probes to pass in router worker sync (dstackai#4313)