Repository navigation
[Bug]: CPU instances never become reachable - shim fails on /api/instance/health #2997
Description
Steps to reproduce
Apply the configuration:
type: fleet
nodes: 1
backends: [aws]Actual behaviour
The fleet is provisioned quickly.
Expected behaviour
The fleet is stuck in provisioning until it is terminated by timeout.
dstack version
0.19.24
Server logs
{"timestamp": "2025-08-18 19:27:29,833", "logger": "dstack._internal.server.services.runner.client", "level": "DEBUG", "message": "shim version: 0.19.24 (0, 19, 24) (API v2)"}
{"timestamp": "2025-08-18 19:27:29,904", "logger": "dstack._internal.server.background.tasks.process_instances", "level": "DEBUG", "message": "shim.get_instance_health(): request error: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))"}
{"timestamp": "2025-08-18 19:27:29,913", "logger": "dstack._internal.server.background.tasks.process_instances", "level": "DEBUG", "message": "Instance rude-jellyfish-0 check: reachable=False health_status=HEALTHY message=\"shim.get_instance_health(): request error: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))\"", "instance_name": "rude-jellyfish-0", "health_status": "healthy"}Additional information
Cloud-init logs:
2025/08/18 19:29:21 http: panic serving 127.0.0.1:3786: reflect: call of reflect.Value.IsNil on zero Value
goroutine 115 [running]:
net/http.(*conn).serve.func1()
/opt/hostedtoolcache/go/1.23.8/x64/src/net/http/server.go:1947 +0xbe
panic({0x9834e0?, 0xc00041c030?})
/opt/hostedtoolcache/go/1.23.8/x64/src/runtime/panic.go:791 +0x132
reflect.Value.IsNil(...)
/opt/hostedtoolcache/go/1.23.8/x64/src/reflect/value.go:1561
github.com/dstackai/dstack/runner/internal/shim/api.(*ShimServer).InstanceHealthHandler(0xc000376000, {0x8?, 0x9858e0?}, 0xc000366001?)
/home/runner/work/dstack/dstack/runner/internal/shim/api/handlers.go:28 +0x272
github.com/dstackai/dstack/runner/internal/api.(*Router).AddHandler.JSONResponseHandler.func1({0xb198b8, 0xc0002e22a0}, 0xc0001c8780)
/home/runner/work/dstack/dstack/runner/internal/api/common.go:111 +0x77
net/http.HandlerFunc.ServeHTTP(0xc00034c000?, {0xb198b8?, 0xc0002e22a0?}, 0xc000105b38?)
/opt/hostedtoolcache/go/1.23.8/x64/src/net/http/server.go:2220 +0x29
net/http.(*ServeMux).ServeHTTP(0xc000105b88?, {0xb198b8, 0xc0002e22a0}, 0xc0001c8780)
/opt/hostedtoolcache/go/1.23.8/x64/src/net/http/server.go:2747 +0x1ca
net/http.serverHandler.ServeHTTP({0xc000520db0?}, {0xb198b8?, 0xc0002e22a0?}, 0x6?)
/opt/hostedtoolcache/go/1.23.8/x64/src/net/http/server.go:3210 +0x8e
net/http.(*conn).serve(0xc0000b6870, {0xb1a888, 0xc0003786c0})
/opt/hostedtoolcache/go/1.23.8/x64/src/net/http/server.go:2092 +0x5d0
created by net/http.(*Server).Serve in goroutine 1
/opt/hostedtoolcache/go/1.23.8/x64/src/net/http/server.go:3360 +0x485
Activity
[shim] Fix `DCGMWrapperInterface` nil check (bis) (#3001)