Repository navigation
place null last in anyOf for optional unions in JSON schema - #1028
Conversation
|
Hi @SarthakB11, this currently fails |
The 'len(structs) >= 2' branch flipped 'if options:' to 'if options or
has_none:', which forced the wrapped shape (struct_schema nested inside the
outer anyOf) even when no other members were present. The expected schema for
Union[Tag1, Tag2, None] is flat: top-level discriminator + top-level anyOf of
$refs with {type: null} appended.
Restore the original 'if options:' predicate, and in the else branch append
{type: null} to the resulting top-level anyOf when has_none. Discriminator
stays at the top level.
test_struct_tagged_union_with_none and full tests/unit/test_schema.py pass
locally.
Signed-off-by: SarthakB11 <[email protected]>
|
Thanks for the catch, @Siyet. The bug was in the Fix: keep the flat |
Merging this PR will not alter performance
Comparing Footnotes
|
Siyet
left a comment
There was a problem hiding this comment.
Thanks for the quick turnaround on the tagged-union fix! I did a deep pass on b5b078a, including an empirical schema diff against 0.21.1 across ~25 type shapes. The ordering change itself looks right and minimal for most shapes, but two behavior changes go beyond reordering and need fixing before this can merge:
1. Meta metadata on a None union member is silently dropped
from typing import Annotated, Union
from msgspec import Meta, json
typ = Union[Annotated[None, Meta(description="explicit null means reset")], int]
json.schema(typ)- 0.21.1:
{"anyOf": [{"description": "explicit null means reset", "type": "null"}, {"type": "integer"}]} - this PR:
{"anyOf": [{"type": "integer"}, {"type": "null"}]}- thedescription(same fortitle/examples/extra_json_schema) is lost.
The unwrap loop detects NoneType through the Metadata wrapper but then discards the subtype and appends a hardcoded {"type": "null"}. Fix: remember the subtype (e.g. none_member = subtype) and append self.to_schema(none_member) instead of the literal.
2. Optional[tagged union] changes schema shape, not just order
For Union[A, B, None] where A/B are tagged structs and there are no other members:
- 0.21.1:
{"anyOf": [{"type": "null"}, {"anyOf": [ref A, ref B], "discriminator": ...}]}- discriminator scoped to the struct members only. - this PR:
{"discriminator": ..., "anyOf": [ref A, ref B, {"type": "null"}]}- discriminator hoisted onto a composite that now contains a null branch.
Problems:
- OpenAPI 3.x expects every schema under a discriminator-bearing composite to carry the discriminator property; an inline
{"type": "null"}can't, and it has nomappingentry. Strict tooling (openapi-generator, spectral lints) chokes on this. - It's inconsistent within this same PR:
Union[A, B, int, None]keeps the nested{anyOf + discriminator}member form, so the same tagged union renders in two different shapes depending on its neighbors. - pydantic emits the nested form here, so this diverges exactly where #1027 asks for parity.
Fix: in the len(structs) >= 2 branch with empty options, keep the nested form and emit {"anyOf": [struct_schema, {"type": "null"}]} instead of schema.update(struct_schema) plus a top-level append. test_struct_tagged_union_with_none currently pins the flat shape and would need updating along with this.
Smaller notes:
- Test gaps worth closing while you're here:
Union[A, B, int, None](the mixed branch isn't covered by any test) and theAnnotated[None, Meta(...)]case from point 1. - The PR description still mentions the
if options or has_none:guard that's no longer in the diff - worth refreshing once the above lands.
Everything else checked out clean: no lost union members, no duplicated nulls, $defs/recursive refs intact, plain unions byte-identical to main, and all outputs pass JSON Schema Draft 2020-12 meta-validation.
…gged union
Address review on the optional-union null-ordering change:
- The union unwrap loop remembered only a has_none flag and appended a
hardcoded {"type": "null"}, dropping any Meta metadata carried on an
Annotated[None, Meta(...)] member. Track the actual none_member subtype
and emit self.to_schema(none_member) so its metadata is preserved.
- Optional[tagged union] flattened the schema shape when no other members
were present. Keep the nested form and emit
{"anyOf": [struct_schema, <null>]} so only ordering, not shape, changes.
Update test_struct_tagged_union_with_none to the nested anyOf shape and add
coverage for Union[Point, Point3D, int, None] and Annotated[None, Meta(...)].
|
Thanks for the review. Both points are addressed.
Tests: updated test_struct_tagged_union_with_none to the nested anyOf shape, and added test_struct_tagged_union_with_none_and_other (Union[Point, Point3D, int, None]) and test_optional_union_null_member_metadata_preserved (Annotated[None, Meta(...)]). Full tests/unit/test_schema.py passes (115 passed, 8 skipped). One note: the PR description still references an |
Siyet
left a comment
There was a problem hiding this comment.
Both change requests are resolved. Re-checked empirically against current main across the shapes from the last pass:
- None-member
Metais preserved now:Union[Annotated[None, Meta(...)], int]keepsdescription(andtitle/examples/extra_json_schema) on the null branch, just moved last. Optional[tagged union]keeps the nested form:Union[A, B, None]emits{"anyOf": [{"anyOf": [refs], "discriminator": ...}, {"type": "null"}]}with no top-level discriminator, andUnion[A, B, int, None]renders the same nested member, so the two are consistent again.
No regressions: plain unions and tagged unions without None are byte-identical to main, null is only relocated (never duplicated or dropped), $defs/refs intact, and every output passes JSON Schema Draft 2020-12 meta-validation. test_schema.py is green (115 passed) and the branch merges cleanly on current main. Thanks for the thorough follow-up!
Follow-up to msgspec#1028. msgspec#1028 introduced `none_member: mi.Type | None` in the union branch of `_json_schema.py` and guarded the `self.to_schema(none_member)` calls with a separate `has_none` boolean. mypy can't narrow `none_member` through the flag, so it errors: ``` _json_schema.py:363: error: Argument 1 to "to_schema" of "_SchemaGenerator" has incompatible type "Type | None"; expected "Type" [arg-type] ``` This is not caught by current CI because the msgspec source is not type-checked there (only `tests/typing` is); msgspec#1116 (adding stubtest) is what surfaces it. Fixing it here so msgspec#1116 can go green. The fix drops the redundant `has_none` flag and checks `none_member is not None` at the call sites, which gives mypy the narrowing. No runtime behavior change (`has_none` and `none_member` were always set together): `tests/unit/test_schema.py` still passes (115), and stubtest with mypy 2.2.0 is clean. Co-authored-by: Siyet <[email protected]>
The 0.21.1 to 0.22.0 changelog batch (msgspec#1112) landed on 2026-07-03. Four user-visible changes merged between then and the 0.22.0 release on 2026-08-11 never got an entry: - msgspec#700, overloads on the `Meta` stub, so type checkers reject mixing `gt` with `ge` and `lt` with `le` - msgspec#1028, `null` placed last in the `anyOf` generated for optional unions - msgspec#1114, `__struct_encode_fields__` added to `src/msgspec/__init__.pyi` (its companion msgspec#813 only touched `tests/typing`, so it needs no entry of its own) - msgspec#1118, `json.encode` raising `TypeError` for a `dict` keyed by a plain `str`-valued `enum.Enum` I went over every PR merged in that window. The rest need no entry: msgspec#813 and msgspec#1117 are tests only, msgspec#1121 fixes an unreleased regression from msgspec#1028, msgspec#1145 is an unused variable under free-threaded builds, and msgspec#1146 and msgspec#1148 are CI and tooling. msgspec#1127, msgspec#1135 and msgspec#1080 already have entries. Docs build clean with `--fail-on-warning`. Note for whoever retries the release: the section is still headed `Version 0.22.0 (2026-08-11)`, and that date will need updating, since the tag was deleted after the upload failed. Refs msgspec#1134. Co-authored-by: Siyet <[email protected]>
Fixes #1027.
For
T | None,_json_schema.pywas emitting{"type": "null"}beforethe struct/
$refentry inanyOf. Tools that pick the firstanyOfelement for example generation (Swagger UI in the issue) ended up showing
null. Pydantic putsnulllast for the same reason.Move
NoneTypeout of the union iteration into ahas_noneflag andappend
{"type": "null"}last in all three union branches (no-struct,single-struct, tagged-struct). The tagged-struct branch's
if options:guard becomes
if options or has_none:soTagged | Nonekeepsnullin its
anyOfinstead of dropping it.Observable changes:
Struct | None: was[null, $ref], now[$ref, null].Union[X, None, Y]:nullmoves from the middle to the end.str | None: already[string, null]on CPython, unchanged.Tests added in
tests/unit/test_schema.pyforOptional[Struct],Union[int, None, str], andUnion[Tagged, None](covers the thirdbranch that previously dropped
nullentirely).