Repository navigation
Fix defstruct with bases using a StructMeta subclass - #1164
abhijeet117 wants to merge 5 commits into
Conversation
fallenmi
left a comment
There was a problem hiding this comment.
The selected metaclass is assigned to the generated type, but its construction hooks are never invoked. StructMeta_new_inner calls PyType_Type.tp_new(metatype, ...) directly, bypassing the selected metaclass's __new__ and __init__. That breaks the documented StructMeta subclass pattern for supplying default struct configuration.
At exact head b11a57d52c3329fa460a1a1d60e7059106796119, using the documented pattern with a metaclass that does config.setdefault("kw_only", True) produces these divergent results:
defstruct: calls=[]; signature=(child: int, *, base: int); Dynamic(2, base=1) succeeds
class syntax: calls=[("new", "Static"), ("init", "Static")]; signature=(*, base: int, child: int); Static(2, base=1) raises TypeError
The added custom-metaclass test only delegates from __new__ and checks the resulting metaclass, so it cannot detect that the hook was skipped. Please preserve the selected metaclass's hooks exactly once (including their configuration/validation behavior) and add a regression covering a hook with an observable effect.
Verification: exact base f51f378335b01dc0026dc6553a0b9e1915a8edae reproduces the reported Struct types cannot define __slots__ failure; the two added tests pass on the head; the complete unit suite passes (6382 passed, 143 skipped). The hook/configuration probe above remains red on the head.
Reviewed with OpenAI Codex assistance
|
@fallenmi Thanks for the feedback. I've addressed the requested metaclass hook behavior and added regression coverage for the configuration changes. The updated implementation and tests have been verified. Please take another look when you get a chance. |
fallenmi
left a comment
There was a problem hiding this comment.
The follow-up still does not preserve the selected metaclass's construction semantics, and the direct tp_init call introduces a regression for ordinary class syntax. At exact head 995ca0b6ffc0ae4cdfeff1de0babce5b3347bc72, PyType_Type.tp_new(type, ...) still bypasses CustomMeta.__new__; type->tp_init is then invoked inside StructMeta_new_inner, before the struct fields/configuration are installed.
For defstruct I observe only [("init", "Dynamic")], with __struct_fields__ still missing during that hook, signature (child: int, *, base: int), and a positional child accepted. Equivalent class syntax observes [("new", "Static"), ("init", "Static"), ("init", "Static")], signature (*, base: int, child: int), and rejects the positional child. A metaclass whose __new__ raises for the generated name is likewise bypassed by defstruct but honored by class syntax.
The added regression test is also red as written because it retains the hooks recorded while creating Base; the complete local unit run is 1 failed, 6382 passed, 143 skipped on CPython 3.13.5. Clearing those setup calls would still expose the missing __new__ and unapplied kw_only policy. The public Build and Test, docs-pr, and CodSpeed workflows for this head are currently action_required with zero jobs.
There is also a deterministic failure-path reference leak in the new code. Py_CLEAR(args) was moved below both the tp_new and tp_init failure branches, while cleanup: never decrefs args. Repeating 25 custom-__init__ validation failures increases the refcount of a namespace sentinel by 25 on this head; exact base increases it by 0.
Please dispatch construction through the selected metaclass on the original namespace/configuration (or otherwise preserve the same lifecycle), so __new__ and __init__ each run exactly once at the class-syntax-equivalent point, validation/config hooks are honored, and all failure-owned references are released. The compatible-MRO and metaclass-conflict behavior did match class syntax in my probes.
Reviewed with OpenAI Codex assistance.
|
@fallenmi Thanks for the detailed probes, both points were correct. The regression test now clears the hooks recorded while building the base, asserts the hook order for both definition styles, and checks that a |
There was a problem hiding this comment.
The previous class-construction and error-handling issues are fixed. One difference from ordinary class creation remains: configuration flags are converted to booleans twice in defstruct, first during argument parsing and then in StructMeta. With a stateful __bool__, the resulting configuration can differ:
class Flag:
def __init__(self):
self.calls = 0
def __bool__(self):
self.calls += 1
return self.calls == 1
flag = Flag()
D = msgspec.defstruct("D", ["x"], frozen=flag)
print(flag.calls, D.__struct_config__.frozen)
# This PR: 2, False. Main and equivalent class syntax: 1, True.The documented flags are booleans, so this is an edge case. Please remove the duplicate conversion while preserving the distinction between omitted and supplied arguments, so metaclass defaults still apply, and add a regression test.
defstruct always created the new class through msgspec.StructMeta, ignoring the metaclasses of the provided bases. When a base used a custom StructMeta subclass, type creation re-dispatched through that more derived metaclass, running class construction a second time on a namespace msgspec had already injected __slots__ into, and failing with TypeError: Struct types cannot define __slots__. Resolve the most derived metaclass among the bases before creating the class, mirroring CPython's metaclass calculation, so defstruct constructs classes under the same metaclass as equivalent class definitions.
PyType_Type.tp_new only runs the metaclass __new__ hook, skipping __init__. This meant metaclass config changes from __init__ were never applied for defstruct-created classes. Call type->tp_init after tp_new so both hooks run exactly once, matching class syntax behavior. Add regression test verifying __new__ and __init__ are each called once and that a kw_only=True config change from the metaclass hook is honored.
Dispatching through the metaclass runs its __new__ and __init__ hooks once each, at the same point they run for an equivalent class definition, so a metaclass that supplies or validates struct configuration now behaves the same for defstruct. This replaces the direct tp_init call, which ran __init__ before the struct fields were installed and skipped __new__ entirely.
93ff1f0 to
b07d9b7
Compare
|
@Siyet Thanks for catching this. Fixed in b07d9b7: defstruct now parses the config options as objects rather than with the p conversion, leaving boolean conversion to the selected metaclass. The original kwargs still pass through unchanged, so omitted options remain omitted and metaclass defaults apply. Added regression coverage for stateful bool, custom-metaclass defaults versus explicit values, and omitted flags alongside namespace objects. Ruff, formatting, codespell, and Python syntax checks pass. I could not rebuild the edited C extension or run the suite locally because this Windows environment only has VS 2019, which lacks stdatomic.h; the current GitHub checks do not include a unit-test build. |
fallenmi
left a comment
There was a problem hiding this comment.
Thanks for fixing the metaclass lifecycle and boolean-option forwarding. One remaining issue in the newly enabled custom-metaclass path: caller-module inference is lost when the metaclass defines __new__ in a different module, which breaks pickling.
Reproduced at ade4eaf70c67e793ecbe1f625af7f388c9387f63 on macOS arm64 with CPython 3.11 and 3.13:
import pickle
import sys
import types
import msgspec
metamod = types.ModuleType("review_metamod")
sys.modules[metamod.__name__] = metamod
exec("""
import msgspec
class Meta(msgspec.StructMeta):
def __new__(mcls, name, bases, ns, **kw):
return super().__new__(mcls, name, bases, ns, **kw)
class Base(msgspec.Struct, metaclass=Meta):
pass
""", metamod.__dict__)
Point = msgspec.defstruct("Point", [("x", int)], bases=(metamod.Base,))
print(Point.__module__) # review_metamod, instead of the caller's __name__
pickle.dumps(Point(1)) # PicklingError: Point is not defined in review_metamodmodule=None has the same failure. Equivalent class syntax, explicit module=__name__, an explicit namespace __module__, and a metaclass without a Python __new__ all pass the cross-module pickle controls.
The new PyObject_Call(metatype, ...) correctly invokes the Python hook, but the omitted-module path in msgspec_defstruct leaves __module__ unset until class creation occurs inside that hook. Please infer the caller's module before invoking the metaclass, while preserving explicit module/namespace behavior, and add a cross-module regression test. This is a defect in functionality newly enabled by this PR, not a claim that the old custom-metaclass path worked.
Verification scope: the existing test_struct.py passes (341 passed, 3 skipped); seven additional local lifecycle/configuration/error-path checks also pass, including stable marker refcounts after repeated construction failures. Those checks do not cover the failing cross-module case above. No full project suite or sanitizer run was performed locally.
Disclosure: AI-assisted review using Claude Code (Opus 5.5) and Codex; the reported failure and controls were independently executed against the pinned head before posting.
|
Following up with an apply-ready candidate for the caller-module/pickle issue above, plus an important integration dependency found while testing it. Please first rebase/merge current main, then apply this candidate. I tested PR head The helper fills an omitted Why the rebase matters: supplying the module also makes string Independently executed on macOS arm64:
Candidate patch — apply after including current main (47 C lines + 92 test lines)diff --git a/src/msgspec/_core.c b/src/msgspec/_core.c
index 5f15701..485b361 100644
--- a/src/msgspec/_core.c
+++ b/src/msgspec/_core.c
@@ -6902,6 +6902,50 @@ error:
}
+/* Set `namespace["__module__"]` from the calling frame's globals, if not
+ * already present. This mirrors the inference `type.__new__` performs, but
+ * must happen before calling the metaclass: if the metaclass defines
+ * `__new__` in Python, `type.__new__` would otherwise see that frame and
+ * infer the metaclass's module rather than the caller's. Returns 0 on
+ * success, -1 with an exception set on error. */
+static int
+structmeta_infer_module(PyObject *namespace) {
+ PyObject *module_key = NULL, *name_key = NULL, *globals, *module;
+ int contains, status = -1;
+
+ module_key = PyUnicode_InternFromString("__module__");
+ if (module_key == NULL) goto done;
+ contains = PyDict_Contains(namespace, module_key);
+ if (contains != 0) {
+ status = contains < 0 ? -1 : 0;
+ goto done;
+ }
+
+ /* Borrowed reference; NULL without an exception if there's no frame */
+ globals = PyEval_GetGlobals();
+ if (globals == NULL) {
+ status = 0;
+ goto done;
+ }
+ name_key = PyUnicode_InternFromString("__name__");
+ if (name_key == NULL) goto done;
+ module = PyDict_GetItemWithError(globals, name_key);
+ if (module == NULL) {
+ status = PyErr_Occurred() ? -1 : 0;
+ goto done;
+ }
+ /* Hold a strong reference to the borrowed `__name__` while inserting */
+ Py_INCREF(module);
+ status = PyDict_SetItem(namespace, module_key, module);
+ Py_DECREF(module);
+
+done:
+ Py_XDECREF(module_key);
+ Py_XDECREF(name_key);
+ return status;
+}
+
+
PyDoc_STRVAR(msgspec_defstruct__doc__,
"defstruct(name, fields, *, bases=None, module=None, namespace=None, "
"tag_field=None, tag=None, rename=None, omit_defaults=False, "
@@ -7005,6 +7049,9 @@ msgspec_defstruct(PyObject *self, PyObject *args, PyObject *kwargs)
}
if (PyDict_SetItemString(namespace, "__module__", module) < 0) goto cleanup;
}
+ else if (structmeta_infer_module(namespace) < 0) {
+ goto cleanup;
+ }
/* Handle bases */
if (bases == NULL || bases == Py_None) {
diff --git a/tests/unit/test_struct.py b/tests/unit/test_struct.py
index fb5143c..e6e886a 100644
--- a/tests/unit/test_struct.py
+++ b/tests/unit/test_struct.py
@@ -2461,6 +2461,98 @@ class TestDefStruct:
Test = defstruct("Test", [], module=None)
assert Test.__module__ == "tests.unit.test_struct"
+ def test_defstruct_module_with_python_metaclass_new(self, monkeypatch):
+ # Infer the module from defstruct's caller, not a Python metaclass hook.
+ meta_globals = {
+ "__name__": "_defstruct_meta_mod",
+ "StructMeta": msgspec.StructMeta,
+ }
+ exec(
+ "class CustomMeta(StructMeta):\n"
+ " def __new__(mcls, name, bases, ns, **kw):\n"
+ " return super().__new__(mcls, name, bases, ns, **kw)\n",
+ meta_globals,
+ )
+ CustomMeta = meta_globals["CustomMeta"]
+
+ class Base(Struct, metaclass=CustomMeta):
+ pass
+
+ class Static(Base):
+ pass
+
+ assert Static.__module__ == __name__
+ Test = defstruct("Test", [], bases=(Base,))
+ assert Test.__module__ == __name__
+ Test = defstruct("Test", [], bases=(Base,), module=None)
+ assert Test.__module__ == __name__
+ Test = defstruct("Test", [], bases=(Base,), module="testmod")
+ assert Test.__module__ == "testmod"
+ Test = defstruct("Test", [], bases=(Base,), namespace={"__module__": "nsmod"})
+ assert Test.__module__ == "nsmod"
+ Test = defstruct(
+ "Test",
+ [],
+ bases=(Base,),
+ module="testmod",
+ namespace={"__module__": "nsmod"},
+ )
+ assert Test.__module__ == "testmod"
+
+ Point = defstruct("DefStructPicklePoint", ["x", "y"], bases=(Base,))
+ monkeypatch.setattr(
+ sys.modules[__name__], "DefStructPicklePoint", Point, raising=False
+ )
+ point = Point(1, 2)
+ assert pickle.loads(pickle.dumps(point)) == point
+
+ @pytest.mark.parametrize("module_value", [None, 123, "nsmod"])
+ @pytest.mark.parametrize("module_options", [{}, {"module": None}])
+ def test_defstruct_module_namespace_precedence(self, module_value, module_options):
+ namespace = {"__module__": module_value}
+ Test = defstruct("Test", [], namespace=namespace, **module_options)
+
+ class Static(Struct):
+ __module__ = module_value
+
+ assert Test.__module__ == Static.__module__
+ assert namespace == {"__module__": module_value}
+
+ @pytest.mark.parametrize("custom_meta", [False, True])
+ def test_defstruct_module_missing_globals_name(self, custom_meta):
+ class CustomMeta(msgspec.StructMeta):
+ def __new__(mcls, name, bases, namespace, **kwargs):
+ return super().__new__(mcls, name, bases, namespace, **kwargs)
+
+ class CustomBase(Struct, metaclass=CustomMeta):
+ pass
+
+ scope = {"Base": CustomBase if custom_meta else Struct, "defstruct": defstruct}
+ exec(
+ "Expected = type(Base)('Expected', (Base,), {})\n"
+ "Actual = defstruct('Actual', [], bases=(Base,))\n",
+ scope,
+ )
+ assert "__name__" not in scope
+ missing = object()
+ assert getattr(scope["Actual"], "__module__", missing) == getattr(
+ scope["Expected"], "__module__", missing
+ )
+
+ def test_defstruct_module_namespace_lookup_error(self):
+ error = RuntimeError("module namespace lookup failed")
+
+ class RaisingKey:
+ def __hash__(self):
+ return hash("__module__")
+
+ def __eq__(self, other):
+ raise error
+
+ with pytest.raises(RuntimeError) as raised:
+ defstruct("Test", [], namespace={RaisingKey(): "value"})
+ assert raised.value is error
+
def test_defstruct_namespace(self):
Test = defstruct(
"Test", ["x", "y"], namespace={"add": lambda self: self.x + self.y}
Disclosure: AI-assisted candidate and integration analysis using Claude Code (Opus 5.5); Codex independently executed the regression, integration, and full-suite checks before this follow-up. This is a proposed patch for this PR, not an upstream-accepted fix. |
Fixes #974
Summary
defstruct(bases=...)raisedTypeError: Struct types cannot define __slots__when a base used a customStructMetasubclass, while equivalent class syntax worked.defstructalways built the class throughStructMeta, so type creation re-dispatched through the base's more derived metaclass and re-entered construction with__slots__already injected into the namespace. We now resolve the most derived metaclass among the bases (mirroring CPython's calculation) and construct through it, matching class definition behavior.Testing
Added
test_defstruct_custom_metaclass_base(constructs, instantiates, JSON round-trips) plustest_defstruct_default_metaclass. Full unit suite: 6380 passed, 145 skipped (CPython 3.13, Windows).ruff check,ruff format --diff, andcodespellclean on changed files.Checklist