Repository navigation
[Bug]: Gateway proxy_read_timeout is hardcoded to 300s, truncating slow long-context streaming responses #4239
Description
Steps to reproduce
- Deploy a long-context model as a service on a single GPU, e.g.
Qwen/Qwen3.8-27B-FP8on one H100 with--max-model-len 262144 --max-num-seqs 16, behind adstackgateway. - Send ~30 concurrent streaming chat-completion requests, each with 60k-100k input tokens.
- Wait 5 minutes.
Actual behaviour
At 300s the gateway closes connections mid-body. The client sees a truncated chunked response rather than an HTTP status code:
ClientPayloadError("Response payload is not completed:
<TransferEncodingError: 400, message='Not enough data to satisfy transfer length header.'>")
Timing matched exactly: requests dispatched at 15:41:52, connection cut at 15:46:59.
Nothing was failing upstream. For the whole preceding 300s the load generator reported errors=0, and 22 of the 30 requests had already returned successfully. The remaining requests were simply queued behind ~1.9M tokens of prefill.
The mechanism is that proxy_read_timeout bounds the gap between successive reads from upstream, not total request duration. For a streaming LLM response the engine sends response headers immediately and the first data: chunk only after prefill completes, so this timeout effectively caps time-to-first-token. Under load with 100k-token prompts, TTFT can exceed 300s legitimately. For example, I hit it during benchmark warmup phase that fired ~30 concurrent 100k-token requests at once. Steady-state serving rarely leaves a request queued for five minutes, so this is not a common availability problem, but it may still happen.
Expected behaviour
proxy_read_timeout should be configurable per service, the way client_max_body_size already is via DEFAULT_SERVICE_CLIENT_MAX_BODY_SIZE and a per-service field.
A higher default for services that declare model: would also be reasonable, since long-context inference is exactly the workload where a 300s inter-read gap is reachable without anything being wrong.
dstack version
0.21.3 (master, e828c9e14)
Activity
mikemikimike commentedon Sep 1, 2026
r4victor commentedon Sep 30, 2026
The problem is more critical with the model entrypoint (
gateway.<gateway-domain>), where requests fail after 60s rather than 300s.The gateway app forwards entrypoint requests to the service's nginx site with an httpx client whose timeout is
DSTACK_SERVICE_CLIENT_TIMEOUT, 60s by default:dstack/src/dstack/_internal/proxy/lib/services/service_connection.py
Lines 146 to 152 in e7eb5f0
httpx applies the read timeout per socket read, same as
proxy_read_timeout. A streaming request that waits more than 60s for its first token fails with502 Error requesting model: ReadTimeout(''), and so does a non-streaming request whose generation takes more than 60s. The variable is read from the gateway app's environment, which users can't configure, so on gateways the limit is effectively fixed at 60s.dstack/src/dstack/_internal/proxy/lib/services/service_connection.py
Lines 28 to 29 in e7eb5f0