Skip to content

Fix defstruct with bases using a StructMeta subclass - #1164

Open
abhijeet117 wants to merge 5 commits into
msgspec:mainfrom
abhijeet117:fix/defstruct-custom-metaclass
Open

abhijeet117 wants to merge 5 commits into
msgspec:mainfrom
abhijeet117:fix/defstruct-custom-metaclass

Conversation

@abhijeet117

Copy link
Copy Markdown

Fixes #974

Summary

defstruct(bases=...) raised TypeError: Struct types cannot define __slots__ when a base used a custom StructMeta subclass, while equivalent class syntax worked. defstruct always built the class through StructMeta, so type creation re-dispatched through the base's more derived metaclass and re-entered construction with __slots__ already injected into the namespace. We now resolve the most derived metaclass among the bases (mirroring CPython's calculation) and construct through it, matching class definition behavior.

Testing

Added test_defstruct_custom_metaclass_base (constructs, instantiates, JSON round-trips) plus test_defstruct_default_metaclass. Full unit suite: 6380 passed, 145 skipped (CPython 3.13, Windows). ruff check, ruff format --diff, and codespell clean on changed files.

Checklist

  • Bug reproduced before fix
  • Root cause identified
  • Bug fixed
  • Tests passed

@fallenmi fallenmi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The selected metaclass is assigned to the generated type, but its construction hooks are never invoked. StructMeta_new_inner calls PyType_Type.tp_new(metatype, ...) directly, bypassing the selected metaclass's __new__ and __init__. That breaks the documented StructMeta subclass pattern for supplying default struct configuration.

At exact head b11a57d52c3329fa460a1a1d60e7059106796119, using the documented pattern with a metaclass that does config.setdefault("kw_only", True) produces these divergent results:

defstruct: calls=[]; signature=(child: int, *, base: int); Dynamic(2, base=1) succeeds
class syntax: calls=[("new", "Static"), ("init", "Static")]; signature=(*, base: int, child: int); Static(2, base=1) raises TypeError

The added custom-metaclass test only delegates from __new__ and checks the resulting metaclass, so it cannot detect that the hook was skipped. Please preserve the selected metaclass's hooks exactly once (including their configuration/validation behavior) and add a regression covering a hook with an observable effect.

Verification: exact base f51f378335b01dc0026dc6553a0b9e1915a8edae reproduces the reported Struct types cannot define __slots__ failure; the two added tests pass on the head; the complete unit suite passes (6382 passed, 143 skipped). The hook/configuration probe above remains red on the head.

Reviewed with OpenAI Codex assistance

@abhijeet117

Copy link
Copy Markdown
Author

@fallenmi Thanks for the feedback. I've addressed the requested metaclass hook behavior and added regression coverage for the configuration changes. The updated implementation and tests have been verified. Please take another look when you get a chance.

@fallenmi fallenmi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The follow-up still does not preserve the selected metaclass's construction semantics, and the direct tp_init call introduces a regression for ordinary class syntax. At exact head 995ca0b6ffc0ae4cdfeff1de0babce5b3347bc72, PyType_Type.tp_new(type, ...) still bypasses CustomMeta.__new__; type->tp_init is then invoked inside StructMeta_new_inner, before the struct fields/configuration are installed.

For defstruct I observe only [("init", "Dynamic")], with __struct_fields__ still missing during that hook, signature (child: int, *, base: int), and a positional child accepted. Equivalent class syntax observes [("new", "Static"), ("init", "Static"), ("init", "Static")], signature (*, base: int, child: int), and rejects the positional child. A metaclass whose __new__ raises for the generated name is likewise bypassed by defstruct but honored by class syntax.

The added regression test is also red as written because it retains the hooks recorded while creating Base; the complete local unit run is 1 failed, 6382 passed, 143 skipped on CPython 3.13.5. Clearing those setup calls would still expose the missing __new__ and unapplied kw_only policy. The public Build and Test, docs-pr, and CodSpeed workflows for this head are currently action_required with zero jobs.

There is also a deterministic failure-path reference leak in the new code. Py_CLEAR(args) was moved below both the tp_new and tp_init failure branches, while cleanup: never decrefs args. Repeating 25 custom-__init__ validation failures increases the refcount of a namespace sentinel by 25 on this head; exact base increases it by 0.

Please dispatch construction through the selected metaclass on the original namespace/configuration (or otherwise preserve the same lifecycle), so __new__ and __init__ each run exactly once at the class-syntax-equivalent point, validation/config hooks are honored, and all failure-owned references are released. The compatible-MRO and metaclass-conflict behavior did match class syntax in my probes.

Reviewed with OpenAI Codex assistance.

@abhijeet117

Copy link
Copy Markdown
Author

@fallenmi Thanks for the detailed probes, both points were correct. defstruct now builds the class by calling the resolved metaclass directly, the same way a class statement is evaluated, so __new__ and __init__ each run exactly once at the class-syntax-equivalent point and configuration injected or validated by those hooks is honored. The direct tp_init call is gone, which also removes the ordinary-class-syntax regression and the failure-path reference leak; StructMeta_new_inner is back to its original form.

The regression test now clears the hooks recorded while building the base, asserts the hook order for both definition styles, and checks that a kw_only policy set in __new__ actually takes effect. I verified the hook/config probe is green, refcount deltas are 0 on the raising-__new__, raising-__init__, invalid-config and metaclass-conflict paths, and the full unit suite passes (6295 passed, 231 skipped). Ready for another look.

@Siyet Siyet left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The previous class-construction and error-handling issues are fixed. One difference from ordinary class creation remains: configuration flags are converted to booleans twice in defstruct, first during argument parsing and then in StructMeta. With a stateful __bool__, the resulting configuration can differ:

class Flag:
    def __init__(self):
        self.calls = 0
    def __bool__(self):
        self.calls += 1
        return self.calls == 1

flag = Flag()
D = msgspec.defstruct("D", ["x"], frozen=flag)
print(flag.calls, D.__struct_config__.frozen)
# This PR: 2, False. Main and equivalent class syntax: 1, True.

The documented flags are booleans, so this is an edge case. Please remove the duplicate conversion while preserving the distinction between omitted and supplied arguments, so metaclass defaults still apply, and add a regression test.

defstruct always created the new class through msgspec.StructMeta, ignoring the metaclasses of the provided bases. When a base used a custom StructMeta subclass, type creation re-dispatched through that more derived metaclass, running class construction a second time on a namespace msgspec had already injected __slots__ into, and failing with TypeError: Struct types cannot define __slots__.

Resolve the most derived metaclass among the bases before creating the class, mirroring CPython's metaclass calculation, so defstruct constructs classes under the same metaclass as equivalent class definitions.
PyType_Type.tp_new only runs the metaclass __new__ hook, skipping __init__.
This meant metaclass config changes from __init__ were never applied for
defstruct-created classes. Call type->tp_init after tp_new so both hooks
run exactly once, matching class syntax behavior.

Add regression test verifying __new__ and __init__ are each called once
and that a kw_only=True config change from the metaclass hook is honored.
Dispatching through the metaclass runs its __new__ and __init__ hooks once
each, at the same point they run for an equivalent class definition, so a
metaclass that supplies or validates struct configuration now behaves the
same for defstruct. This replaces the direct tp_init call, which ran
__init__ before the struct fields were installed and skipped __new__
entirely.
@abhijeet117
abhijeet117 force-pushed the fix/defstruct-custom-metaclass branch from 93ff1f0 to b07d9b7 Compare September 27, 2026 12:52
@abhijeet117

Copy link
Copy Markdown
Author

@Siyet Thanks for catching this. Fixed in b07d9b7: defstruct now parses the config options as objects rather than with the p conversion, leaving boolean conversion to the selected metaclass. The original kwargs still pass through unchanged, so omitted options remain omitted and metaclass defaults apply. Added regression coverage for stateful bool, custom-metaclass defaults versus explicit values, and omitted flags alongside namespace objects. Ruff, formatting, codespell, and Python syntax checks pass. I could not rebuild the edited C extension or run the suite locally because this Windows environment only has VS 2019, which lacks stdatomic.h; the current GitHub checks do not include a unit-test build.

@fallenmi fallenmi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for fixing the metaclass lifecycle and boolean-option forwarding. One remaining issue in the newly enabled custom-metaclass path: caller-module inference is lost when the metaclass defines __new__ in a different module, which breaks pickling.

Reproduced at ade4eaf70c67e793ecbe1f625af7f388c9387f63 on macOS arm64 with CPython 3.11 and 3.13:

import pickle
import sys
import types
import msgspec

metamod = types.ModuleType("review_metamod")
sys.modules[metamod.__name__] = metamod
exec("""
import msgspec
class Meta(msgspec.StructMeta):
    def __new__(mcls, name, bases, ns, **kw):
        return super().__new__(mcls, name, bases, ns, **kw)
class Base(msgspec.Struct, metaclass=Meta):
    pass
""", metamod.__dict__)

Point = msgspec.defstruct("Point", [("x", int)], bases=(metamod.Base,))
print(Point.__module__)  # review_metamod, instead of the caller's __name__
pickle.dumps(Point(1))  # PicklingError: Point is not defined in review_metamod

module=None has the same failure. Equivalent class syntax, explicit module=__name__, an explicit namespace __module__, and a metaclass without a Python __new__ all pass the cross-module pickle controls.

The new PyObject_Call(metatype, ...) correctly invokes the Python hook, but the omitted-module path in msgspec_defstruct leaves __module__ unset until class creation occurs inside that hook. Please infer the caller's module before invoking the metaclass, while preserving explicit module/namespace behavior, and add a cross-module regression test. This is a defect in functionality newly enabled by this PR, not a claim that the old custom-metaclass path worked.

Verification scope: the existing test_struct.py passes (341 passed, 3 skipped); seven additional local lifecycle/configuration/error-path checks also pass, including stable marker refcounts after repeated construction failures. Those checks do not cover the failing cross-module case above. No full project suite or sanitizer run was performed locally.

Disclosure: AI-assisted review using Claude Code (Opus 5.5) and Codex; the reported failure and controls were independently executed against the pinned head before posting.

@fallenmi

Copy link
Copy Markdown
Contributor

Following up with an apply-ready candidate for the caller-module/pickle issue above, plus an important integration dependency found while testing it.

Please first rebase/merge current main, then apply this candidate. I tested PR head ade4eaf70c67e793ecbe1f625af7f388c9387f63 integrated with main 6d3443f2dae8413a1a56640290c09a6f486f30c2 (conflict-free pre-patch tree 49d5f6ccb7f1ec84887b2b8f8e61f02a0ffae618). The diff below is against that integration, not main alone.

The helper fills an omitted __module__ from the caller's globals before dispatching to the Python metaclass. Explicit module= and namespace __module__ retain their precedence; the supplied namespace is not mutated.

Why the rebase matters: supplying the module also makes string ClassVar annotations resolve like class syntax. On the old PR head plus this candidate, a bounded weakref probe retains all 12 temporary module-namespace sentinels after GC (both ClassVar spellings, default/Python metaclasses, three modules per combination). Class syntax already retains them on that old head. With current main, including the collector ownership fix in #1194, the candidate gives correct fields and retains 0/12 on both routes. Thus I would not apply the caller-module patch alone to the unchanged old head.

Independently executed on macOS arm64:

  • CPython 3.11.11 and 3.13.2: the new caller-module regression fails before the C fix and passes after it.
  • Both versions: test_struct.py 363 passed, 3 skipped, plus 11 passed in separate integration probes for string ClassVar/module lifetime, annotations/mixin-dict lifetime, GC/weakref rejection, and non-type-base rejection.
  • CPython 3.13.2 full tests/unit: 6472 passed, 146 skipped. This includes the patch's regression tests, but not the 11 separate probes.
  • Existing local lifecycle/error-path controls and cross-module pickle controls also pass on both versions.
  • Ruff 0.16.7 lint/format and git diff --check pass. No sanitizer/free-threaded run; no full unit run on 3.11. These are local integration results, not CI results for an upstream commit.
Candidate patch — apply after including current main (47 C lines + 92 test lines)
diff --git a/src/msgspec/_core.c b/src/msgspec/_core.c
index 5f15701..485b361 100644
--- a/src/msgspec/_core.c
+++ b/src/msgspec/_core.c
@@ -6902,6 +6902,50 @@ error:
 }
 
 
+/* Set `namespace["__module__"]` from the calling frame's globals, if not
+ * already present. This mirrors the inference `type.__new__` performs, but
+ * must happen before calling the metaclass: if the metaclass defines
+ * `__new__` in Python, `type.__new__` would otherwise see that frame and
+ * infer the metaclass's module rather than the caller's. Returns 0 on
+ * success, -1 with an exception set on error. */
+static int
+structmeta_infer_module(PyObject *namespace) {
+    PyObject *module_key = NULL, *name_key = NULL, *globals, *module;
+    int contains, status = -1;
+
+    module_key = PyUnicode_InternFromString("__module__");
+    if (module_key == NULL) goto done;
+    contains = PyDict_Contains(namespace, module_key);
+    if (contains != 0) {
+        status = contains < 0 ? -1 : 0;
+        goto done;
+    }
+
+    /* Borrowed reference; NULL without an exception if there's no frame */
+    globals = PyEval_GetGlobals();
+    if (globals == NULL) {
+        status = 0;
+        goto done;
+    }
+    name_key = PyUnicode_InternFromString("__name__");
+    if (name_key == NULL) goto done;
+    module = PyDict_GetItemWithError(globals, name_key);
+    if (module == NULL) {
+        status = PyErr_Occurred() ? -1 : 0;
+        goto done;
+    }
+    /* Hold a strong reference to the borrowed `__name__` while inserting */
+    Py_INCREF(module);
+    status = PyDict_SetItem(namespace, module_key, module);
+    Py_DECREF(module);
+
+done:
+    Py_XDECREF(module_key);
+    Py_XDECREF(name_key);
+    return status;
+}
+
+
 PyDoc_STRVAR(msgspec_defstruct__doc__,
 "defstruct(name, fields, *, bases=None, module=None, namespace=None, "
 "tag_field=None, tag=None, rename=None, omit_defaults=False, "
@@ -7005,6 +7049,9 @@ msgspec_defstruct(PyObject *self, PyObject *args, PyObject *kwargs)
         }
         if (PyDict_SetItemString(namespace, "__module__", module) < 0) goto cleanup;
     }
+    else if (structmeta_infer_module(namespace) < 0) {
+        goto cleanup;
+    }
 
     /* Handle bases */
     if (bases == NULL || bases == Py_None) {
diff --git a/tests/unit/test_struct.py b/tests/unit/test_struct.py
index fb5143c..e6e886a 100644
--- a/tests/unit/test_struct.py
+++ b/tests/unit/test_struct.py
@@ -2461,6 +2461,98 @@ class TestDefStruct:
         Test = defstruct("Test", [], module=None)
         assert Test.__module__ == "tests.unit.test_struct"
 
+    def test_defstruct_module_with_python_metaclass_new(self, monkeypatch):
+        # Infer the module from defstruct's caller, not a Python metaclass hook.
+        meta_globals = {
+            "__name__": "_defstruct_meta_mod",
+            "StructMeta": msgspec.StructMeta,
+        }
+        exec(
+            "class CustomMeta(StructMeta):\n"
+            "    def __new__(mcls, name, bases, ns, **kw):\n"
+            "        return super().__new__(mcls, name, bases, ns, **kw)\n",
+            meta_globals,
+        )
+        CustomMeta = meta_globals["CustomMeta"]
+
+        class Base(Struct, metaclass=CustomMeta):
+            pass
+
+        class Static(Base):
+            pass
+
+        assert Static.__module__ == __name__
+        Test = defstruct("Test", [], bases=(Base,))
+        assert Test.__module__ == __name__
+        Test = defstruct("Test", [], bases=(Base,), module=None)
+        assert Test.__module__ == __name__
+        Test = defstruct("Test", [], bases=(Base,), module="testmod")
+        assert Test.__module__ == "testmod"
+        Test = defstruct("Test", [], bases=(Base,), namespace={"__module__": "nsmod"})
+        assert Test.__module__ == "nsmod"
+        Test = defstruct(
+            "Test",
+            [],
+            bases=(Base,),
+            module="testmod",
+            namespace={"__module__": "nsmod"},
+        )
+        assert Test.__module__ == "testmod"
+
+        Point = defstruct("DefStructPicklePoint", ["x", "y"], bases=(Base,))
+        monkeypatch.setattr(
+            sys.modules[__name__], "DefStructPicklePoint", Point, raising=False
+        )
+        point = Point(1, 2)
+        assert pickle.loads(pickle.dumps(point)) == point
+
+    @pytest.mark.parametrize("module_value", [None, 123, "nsmod"])
+    @pytest.mark.parametrize("module_options", [{}, {"module": None}])
+    def test_defstruct_module_namespace_precedence(self, module_value, module_options):
+        namespace = {"__module__": module_value}
+        Test = defstruct("Test", [], namespace=namespace, **module_options)
+
+        class Static(Struct):
+            __module__ = module_value
+
+        assert Test.__module__ == Static.__module__
+        assert namespace == {"__module__": module_value}
+
+    @pytest.mark.parametrize("custom_meta", [False, True])
+    def test_defstruct_module_missing_globals_name(self, custom_meta):
+        class CustomMeta(msgspec.StructMeta):
+            def __new__(mcls, name, bases, namespace, **kwargs):
+                return super().__new__(mcls, name, bases, namespace, **kwargs)
+
+        class CustomBase(Struct, metaclass=CustomMeta):
+            pass
+
+        scope = {"Base": CustomBase if custom_meta else Struct, "defstruct": defstruct}
+        exec(
+            "Expected = type(Base)('Expected', (Base,), {})\n"
+            "Actual = defstruct('Actual', [], bases=(Base,))\n",
+            scope,
+        )
+        assert "__name__" not in scope
+        missing = object()
+        assert getattr(scope["Actual"], "__module__", missing) == getattr(
+            scope["Expected"], "__module__", missing
+        )
+
+    def test_defstruct_module_namespace_lookup_error(self):
+        error = RuntimeError("module namespace lookup failed")
+
+        class RaisingKey:
+            def __hash__(self):
+                return hash("__module__")
+
+            def __eq__(self, other):
+                raise error
+
+        with pytest.raises(RuntimeError) as raised:
+            defstruct("Test", [], namespace={RaisingKey(): "value"})
+        assert raised.value is error
+
     def test_defstruct_namespace(self):
         Test = defstruct(
             "Test", ["x", "y"], namespace={"add": lambda self: self.x + self.y}

Disclosure: AI-assisted candidate and integration analysis using Claude Code (Opus 5.5); Codex independently executed the regression, integration, and full-suite checks before this follow-up. This is a proposed patch for this PR, not an upstream-accepted fix.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

msgspec.defstruct is not compatible with msgspec.StructMeta

3 participants