Skip to content

place null last in anyOf for optional unions in JSON schema - #1028

Merged
Siyet merged 4 commits into
msgspec:mainfrom
SarthakB11:fix/issue-1027
Jul 9, 2026
Merged

Siyet merged 4 commits into
msgspec:mainfrom
SarthakB11:fix/issue-1027

Conversation

@SarthakB11

Copy link
Copy Markdown
Contributor

Fixes #1027.

For T | None, _json_schema.py was emitting {"type": "null"} before
the struct/$ref entry in anyOf. Tools that pick the first anyOf
element for example generation (Swagger UI in the issue) ended up showing
null. Pydantic puts null last for the same reason.

Move NoneType out of the union iteration into a has_none flag and
append {"type": "null"} last in all three union branches (no-struct,
single-struct, tagged-struct). The tagged-struct branch's if options:
guard becomes if options or has_none: so Tagged | None keeps null
in its anyOf instead of dropping it.

Observable changes:

  • Struct | None: was [null, $ref], now [$ref, null].
  • Union[X, None, Y]: null moves from the middle to the end.
  • str | None: already [string, null] on CPython, unchanged.

Tests added in tests/unit/test_schema.py for Optional[Struct],
Union[int, None, str], and Union[Tagged, None] (covers the third
branch that previously dropped null entirely).

@Siyet

Siyet commented Jun 15, 2026

Copy link
Copy Markdown
Contributor

Hi @SarthakB11, this currently fails tests/unit/test_schema.py::test_struct_tagged_union_with_none across all Python versions, the null-last reordering interacts with tagged-union schemas. Could you take a look when you get a chance? Happy to review once CI is green.

The 'len(structs) >= 2' branch flipped 'if options:' to 'if options or
has_none:', which forced the wrapped shape (struct_schema nested inside the
outer anyOf) even when no other members were present. The expected schema for
Union[Tag1, Tag2, None] is flat: top-level discriminator + top-level anyOf of
$refs with {type: null} appended.

Restore the original 'if options:' predicate, and in the else branch append
{type: null} to the resulting top-level anyOf when has_none. Discriminator
stays at the top level.

test_struct_tagged_union_with_none and full tests/unit/test_schema.py pass
locally.

Signed-off-by: SarthakB11 <[email protected]>
@SarthakB11

Copy link
Copy Markdown
Contributor Author

Thanks for the catch, @Siyet. The bug was in the len(structs) >= 2 branch: I flipped if options: to if options or has_none:, which forced the wrapped shape even when there were no other members, hiding the discriminator one level down.

Fix: keep the flat schema.update(struct_schema) path when only None is added, and append {"type": "null"} to the resulting anyOf after the update. All 113 tests in tests/unit/test_schema.py pass locally. Pushed 139e407.

@codspeed

codspeed Bot commented Jul 2, 2026 •

Copy link
Copy Markdown

Merging this PR will not alter performance

✅ 139 untouched benchmarks
⏩ 135 skipped benchmarks1


Comparing SarthakB11:fix/issue-1027 (88b1224) with main (9ddc0e9)

Open in CodSpeed

Footnotes

  1. 135 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

@Siyet Siyet left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the quick turnaround on the tagged-union fix! I did a deep pass on b5b078a, including an empirical schema diff against 0.21.1 across ~25 type shapes. The ordering change itself looks right and minimal for most shapes, but two behavior changes go beyond reordering and need fixing before this can merge:

1. Meta metadata on a None union member is silently dropped

from typing import Annotated, Union
from msgspec import Meta, json

typ = Union[Annotated[None, Meta(description="explicit null means reset")], int]
json.schema(typ)
  • 0.21.1: {"anyOf": [{"description": "explicit null means reset", "type": "null"}, {"type": "integer"}]}
  • this PR: {"anyOf": [{"type": "integer"}, {"type": "null"}]} - the description (same for title/examples/extra_json_schema) is lost.

The unwrap loop detects NoneType through the Metadata wrapper but then discards the subtype and appends a hardcoded {"type": "null"}. Fix: remember the subtype (e.g. none_member = subtype) and append self.to_schema(none_member) instead of the literal.

2. Optional[tagged union] changes schema shape, not just order

For Union[A, B, None] where A/B are tagged structs and there are no other members:

  • 0.21.1: {"anyOf": [{"type": "null"}, {"anyOf": [ref A, ref B], "discriminator": ...}]} - discriminator scoped to the struct members only.
  • this PR: {"discriminator": ..., "anyOf": [ref A, ref B, {"type": "null"}]} - discriminator hoisted onto a composite that now contains a null branch.

Problems:

  • OpenAPI 3.x expects every schema under a discriminator-bearing composite to carry the discriminator property; an inline {"type": "null"} can't, and it has no mapping entry. Strict tooling (openapi-generator, spectral lints) chokes on this.
  • It's inconsistent within this same PR: Union[A, B, int, None] keeps the nested {anyOf + discriminator} member form, so the same tagged union renders in two different shapes depending on its neighbors.
  • pydantic emits the nested form here, so this diverges exactly where #1027 asks for parity.

Fix: in the len(structs) >= 2 branch with empty options, keep the nested form and emit {"anyOf": [struct_schema, {"type": "null"}]} instead of schema.update(struct_schema) plus a top-level append. test_struct_tagged_union_with_none currently pins the flat shape and would need updating along with this.

Smaller notes:

  • Test gaps worth closing while you're here: Union[A, B, int, None] (the mixed branch isn't covered by any test) and the Annotated[None, Meta(...)] case from point 1.
  • The PR description still mentions the if options or has_none: guard that's no longer in the diff - worth refreshing once the above lands.

Everything else checked out clean: no lost union members, no duplicated nulls, $defs/recursive refs intact, plain unions byte-identical to main, and all outputs pass JSON Schema Draft 2020-12 meta-validation.

…gged union

Address review on the optional-union null-ordering change:

- The union unwrap loop remembered only a has_none flag and appended a
  hardcoded {"type": "null"}, dropping any Meta metadata carried on an
  Annotated[None, Meta(...)] member. Track the actual none_member subtype
  and emit self.to_schema(none_member) so its metadata is preserved.
- Optional[tagged union] flattened the schema shape when no other members
  were present. Keep the nested form and emit
  {"anyOf": [struct_schema, <null>]} so only ordering, not shape, changes.

Update test_struct_tagged_union_with_none to the nested anyOf shape and add
coverage for Union[Point, Point3D, int, None] and Annotated[None, Meta(...)].
@SarthakB11

Copy link
Copy Markdown
Contributor Author

Thanks for the review. Both points are addressed.

  1. None-member metadata is now preserved. The union unwrap loop keeps the actual none_member subtype and emits self.to_schema(none_member) instead of a hardcoded {"type": "null"}, so Annotated[None, Meta(...)] carries its metadata into the schema.

  2. Optional[tagged union] no longer flattens the shape. In the empty-options case it now emits {"anyOf": [struct_schema, ]}, keeping the nested discriminator form so only ordering changes, not the shape.

Tests: updated test_struct_tagged_union_with_none to the nested anyOf shape, and added test_struct_tagged_union_with_none_and_other (Union[Point, Point3D, int, None]) and test_optional_union_null_member_metadata_preserved (Annotated[None, Meta(...)]). Full tests/unit/test_schema.py passes (115 passed, 8 skipped).

One note: the PR description still references an if options or has_none: guard that is no longer in the diff. I will refresh the description to match the current code.

@Siyet Siyet left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Both change requests are resolved. Re-checked empirically against current main across the shapes from the last pass:

  • None-member Meta is preserved now: Union[Annotated[None, Meta(...)], int] keeps description (and title/examples/extra_json_schema) on the null branch, just moved last.
  • Optional[tagged union] keeps the nested form: Union[A, B, None] emits {"anyOf": [{"anyOf": [refs], "discriminator": ...}, {"type": "null"}]} with no top-level discriminator, and Union[A, B, int, None] renders the same nested member, so the two are consistent again.

No regressions: plain unions and tagged unions without None are byte-identical to main, null is only relocated (never duplicated or dropped), $defs/refs intact, and every output passes JSON Schema Draft 2020-12 meta-validation. test_schema.py is green (115 passed) and the branch merges cleanly on current main. Thanks for the thorough follow-up!

@Siyet
Siyet added this pull request to the merge queue Jul 9, 2026
Merged via the queue into msgspec:main with commit e1c6f23 Jul 9, 2026
26 of 27 checks passed
sobolevn pushed a commit to adriangb/msgspec that referenced this pull request Jul 13, 2026
Follow-up to msgspec#1028.

msgspec#1028 introduced `none_member: mi.Type | None` in the union branch of
`_json_schema.py` and guarded the `self.to_schema(none_member)` calls
with a separate `has_none` boolean. mypy can't narrow `none_member`
through the flag, so it errors:

```
_json_schema.py:363: error: Argument 1 to "to_schema" of "_SchemaGenerator" has incompatible type "Type | None"; expected "Type"  [arg-type]
```

This is not caught by current CI because the msgspec source is not
type-checked there (only `tests/typing` is); msgspec#1116 (adding stubtest) is
what surfaces it. Fixing it here so msgspec#1116 can go green.

The fix drops the redundant `has_none` flag and checks `none_member is
not None` at the call sites, which gives mypy the narrowing. No runtime
behavior change (`has_none` and `none_member` were always set together):
`tests/unit/test_schema.py` still passes (115), and stubtest with mypy
2.2.0 is clean.

Co-authored-by: Siyet <[email protected]>
@Siyet Siyet mentioned this pull request Jul 13, 2026
Brohammad pushed a commit to Brohammad/msgspec that referenced this pull request Sep 4, 2026
The 0.21.1 to 0.22.0 changelog batch (msgspec#1112) landed on 2026-07-03. Four
user-visible changes merged between then and the 0.22.0 release on
2026-08-11 never got an entry:

- msgspec#700, overloads on the `Meta` stub, so type checkers reject mixing
`gt` with `ge` and `lt` with `le`
- msgspec#1028, `null` placed last in the `anyOf` generated for optional unions
- msgspec#1114, `__struct_encode_fields__` added to `src/msgspec/__init__.pyi`
(its companion msgspec#813 only touched `tests/typing`, so it needs no entry of
its own)
- msgspec#1118, `json.encode` raising `TypeError` for a `dict` keyed by a plain
`str`-valued `enum.Enum`

I went over every PR merged in that window. The rest need no entry: msgspec#813
and msgspec#1117 are tests only, msgspec#1121 fixes an unreleased regression from
msgspec#1028, msgspec#1145 is an unused variable under free-threaded builds, and msgspec#1146
and msgspec#1148 are CI and tooling. msgspec#1127, msgspec#1135 and msgspec#1080 already have
entries.

Docs build clean with `--fail-on-warning`.

Note for whoever retries the release: the section is still headed
`Version 0.22.0 (2026-08-11)`, and that date will need updating, since
the tag was deleted after the upload failed. Refs msgspec#1134.

Co-authored-by: Siyet <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

field: Schema | None = None should generate anyOf where null is the last item

2 participants