Skip to content

Clean up module import handling in PyInit__core - #1081

Merged
sobolevn merged 10 commits into
msgspec:mainfrom
provinzkraut:cleanup-module-refs
Oct 16, 2026
Merged

sobolevn merged 10 commits into
msgspec:mainfrom
provinzkraut:cleanup-module-refs

Conversation

@provinzkraut

@provinzkraut provinzkraut commented Jun 17, 2026 •

Copy link
Copy Markdown
Member

Clean up handling of module imports in PyInit__core:

  • Unify error handling; Introduce and always use error label to ensure temporary objects are cleaned up (previously we often did return NULL early, missing Py_XDECREF in a few places)
  • Introduce a new SET_MODULE_REF macro to uniformly handle the "call PyModule_AddObjectRef, check for errors" pattern
  • Introduce a new IMPORT_TEMP_MODULE to uniformly handle the "import module as temp_module, get attribute from temp_module as temp_obj, and assign it to the module state"

Removes 2 leaks found with LSan on (checked locally, as it's turned off in CI)


Based on @sobolevn's suggestion here: #1059 (review)

@provinzkraut
provinzkraut requested a review from sobolevn June 17, 2026 09:45
Comment thread src/msgspec/_core.c Resolved
@codspeed

codspeed Bot commented Jun 17, 2026 •

Copy link
Copy Markdown

Merging this PR will not alter performance

✅ 252 untouched benchmarks
⏩ 2 skipped benchmarks1


Comparing provinzkraut:cleanup-module-refs (bd95764) with main (d3b848c)

Open in CodSpeed

Footnotes

  1. 2 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

Comment thread src/msgspec/_core.c Resolved
Comment thread src/msgspec/_core.c Resolved
@provinzkraut
provinzkraut marked this pull request as draft June 17, 2026 09:49
Comment thread src/msgspec/_core.c Resolved
Comment thread src/msgspec/_core.c Resolved
Comment thread src/msgspec/_core.c Resolved
Comment thread src/msgspec/_core.c Resolved
Comment thread src/msgspec/_core.c Resolved
Comment thread src/msgspec/_core.c Resolved
@provinzkraut
provinzkraut force-pushed the cleanup-module-refs branch from 49a70c5 to 3b22041 Compare June 17, 2026 10:11
@provinzkraut
provinzkraut force-pushed the cleanup-module-refs branch from 3b22041 to d382e12 Compare June 17, 2026 10:11
@provinzkraut
provinzkraut marked this pull request as ready for review June 17, 2026 10:32
@provinzkraut
provinzkraut force-pushed the cleanup-module-refs branch from b44a139 to f7829ec Compare June 17, 2026 15:20
@provinzkraut
provinzkraut requested a review from sobolevn June 18, 2026 11:34

@Siyet Siyet left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did a deep pass on bd95764 (code walk of every failure path + an empirical check). Substance looks good, I'd like to get this in.

Both claimed leaks are real and fixed. On the base, the uuid and decimal blocks are the only two that never Py_DECREF(temp_module) (the reference is overwritten by the next PyImport_ImportModule), and they're exactly what LSan would flag on a plain import. Verified empirically too: with the merge-base and this branch built from the same parent, sys.getrefcount(sys.modules["uuid"]) and ...["decimal"] drop from 4 to 3 while every other module stays identical. Beyond those two, the unified error label also closes the ~20 conditional error paths that used to drop a live temp_module on early return NULL - walked all ten import blocks, the ownership chain converges on every path, no double-DECREF (the SET_TEMP_OBJ/IMPORT_TEMP_MODULE XDECREF-before-replace handles reentry, and the error label's XDECREFs are NULL-safe for the early PyType_Ready gotos). Full unit suite is green on a local build.

On the open items:

  • The conflict with main is a single hunk from #962: keep SET_REF(convert_generic_alias, "convert_generic_alias") in the _utils block and drop the now-redundant Py_DECREF(temp_module). Nothing from #1052/#1097/#1109 touches PyInit__core, so it's a one-line resolution.
  • The EnumMeta/ABCMeta PyType_Check removal you mentioned trying (and ofek pre-approved with "if CI passes... feel free to merge") never got pushed. Either way works for me - push the removal here, or merge as-is and drop them in a follow-up. I'd just rather not let a green refactor sit for another two weeks.

After the rebase this is an approve from me.

@provinzkraut
provinzkraut requested review from sobolevn and removed request for sobolevn October 13, 2026 21:57
@provinzkraut

Copy link
Copy Markdown
Member Author

@sobolevn Can you check this again when you have the time?

@sobolevn sobolevn left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice! Two minor suggestions :)

Comment thread src/msgspec/_core.c Resolved
Comment thread src/msgspec/_core.c Resolved
@sobolevn
sobolevn added this pull request to the merge queue Oct 16, 2026
@sobolevn

Copy link
Copy Markdown
Member

Thank you! This is a very nice and useful change!

Merged via the queue into msgspec:main with commit 529cf3f Oct 16, 2026
24 checks passed
@provinzkraut
provinzkraut deleted the cleanup-module-refs branch October 17, 2026 15:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants