Repository navigation
[Bug]: 0.21.4 CLI is not compatible with servers < 0.21.3 (service groups) #4255
Description
Steps to reproduce
- Run a
dstackserver 0.21.2 (or any version < 0.21.3). - Use a 0.21.4 CLI against it.
- Apply any service configuration, e.g.:
type: service
name: httpbin
image: ghcr.io/psf/httpbin:0.10.2
replicas: 1
port: 8080
https: auto
auth: false
gateway: truedstack apply -f .dstack/httpbin.yamlCompatibility matrix observed:
| client | server | result |
|---|---|---|
| 0.21.3 | 0.21.2 | ok |
| 0.21.4 | 0.21.3 | ok |
| 0.21.4 | 0.21.2 | fails |
Actual behaviour
Server validation error:
{'detail': [{'input': None,
'loc': ['body', 'run_spec', 'configuration', 'service', 'groups'],
'msg': 'Extra inputs are not permitted',
'type': 'extra_forbidden'}]}
The 0.21.4 client always serializes configuration.groups for services, including as
null for a homogeneous service. Servers before 0.21.3 have no such field and reject it
because request models are extra="forbid".
The regression can be reproduced without a running server by validating the client's
request payload against the 0.21.2 request schema:
run_spec.configuration.service.groups
Extra inputs are not permitted [type=extra_forbidden, input_value=None, input_type=NoneType]
Expected behaviour
Any service configuration that does not use the 0.21.3+ groups feature should apply
successfully against older servers, as it did with the 0.21.3 client. Per the client
compatibility convention documented in dstack/_internal/core/compatibility/runs.py,
new fields must be excluded from requests when they are not set.
dstack version
0.21.4 (client); server 0.21.2
Server logs
Not relevant — the request is rejected by request-body validation before it reaches any
run logic. The 422 response body is quoted under "Actual behaviour".Additional information
ServiceConfigurationParams.groups was introduced in 0.21.3 by #4177, which renamed
service replica groups from the legacy replicas: [...] list to groups. That PR also
added a @model_serializer(mode="wrap") (_serialize_legacy_replica_groups) on
ServiceConfigurationParams that popped groups out of every dump and re-emitted it as
the legacy replicas: [{count: ...}] shape. Because it lived on the model, it applied in
both directions, so a 0.21.3 client never put groups on the wire — which is why
0.21.3 → 0.21.2 works.
#4248 removed that serializer and moved the rewrite to server/compatibility/runs.py,
gated on client_version < 0.21.3. That covers the server → old client direction, but
nothing replaced the client → old server direction, so 0.21.4 clients now always send
groups.
ServiceConfiguration.groups is the only field added since 0.21.2 across the service,
task and dev-environment configurations, RunSpec and ProfileParams, so this appears
to be the only source of 0.21.4 → 0.21.2 incompatibility.
Note that #4248 also dropped the ability to run a heterogeneous service (groups
actually set) against a pre-0.21.3 server, which the 0.21.3 client supported by
downgrading the value to the legacy replicas: [{count: ...}] shape. Restoring that is
not expressible with the client's exclude-mapping mechanism, which can only drop fields,
not rename them.
Activity
[CLI] Exclude unset service `groups` from requests (dstackai#4256)