Repository navigation
Crash in specialize_dict_access(): type->ht_cached_keys is NULL on a pybind11 type #96046
Description
Activity
- added a commit that references this issue
on Aug 17, 2022 cd pikepdf/ python -m pip install .If I understood correctly, this command installed pybind11-2.10.0 in a temporary virtual environment to build the pikepdf C extensions.
- I was able to reproduce the stack trace in a fedora:36 docker container.
- I created Undo attr.h, detail/class.h changes made under PR #3923 pybind/pybind11#4142 which I believe will fix the issue.
But I couldn't figure out how to have the pybind11 diff applied when the
python -m pip install .runs. I know next to nothing aboutpipandpyproject.toml. Could someone please help with a hint? What is the best way to handle this situation?E.g., is there a way to run a
patchcommand after pybind11 2.10.0 landed inpip's temporary working directory?Or can I get
pipto use a given working directory that I manually patch, then runpipagain?You should be able to replace the pybind11 requirement here:
https://github.com/pikepdf/pikepdf/blob/3ebb63a1d38c3a4d8a11b5b13b6106a245726f80/pyproject.toml#L9
With a link to https://github.com/rwgk/pybind11/archive/refs/heads/undo_Py_TPFLAGS_MANAGED_DICT.zip
[build-system] requires = [ "setuptools >= 52", "setuptools-scm[toml] >= 7.0.5", "wheel >= 0.35", "https://github.com/rwgk/pybind11/archive/refs/heads/undo_Py_TPFLAGS_MANAGED_DICT.zip", ] build-backend = "setuptools.build_meta"
pikepdf calls pybind11 make_new_python_type() function. Simplified code:
inline PyObject *make_new_python_type(const type_record &rec) { ... auto *heap_type = (PyHeapTypeObject *) metaclass->tp_alloc(metaclass, 0); ... type->tp_name = full_name; type->tp_doc = tp_doc; type->tp_base = type_incref((PyTypeObject *) base); type->tp_basicsize = static_cast<ssize_t>(sizeof(instance)); ... type->tp_as_number = &heap_type->as_number; type->tp_as_sequence = &heap_type->as_sequence; type->tp_as_mapping = &heap_type->as_mapping; type->tp_as_async = &heap_type->as_async; ... type->tp_flags |= Py_TPFLAGS_DEFAULT | Py_TPFLAGS_HEAPTYPE; ... if (PyType_Ready(type) < 0) { pybind11_fail(std::string(rec.name) + ": PyType_Ready failed: " + error_string()); } ... return (PyObject *) type; }
This function calls PyType_Ready() but not type_new().
Maybe type_new() initializing
ht_cached_keyscan be moved from type_new() to PyType_Ready(), so when a project only calls PyType_Type(), we can make sure thatht_cached_keysis initialized. By the way, when PyType_Ready() is called on a heap type which doesn't have Py_TPFLAGS_MANAGED_DICT, PyType_Ready() sets the flag if the parent class has the flag set. For me, it's surprising that the flag can be set, without initializinght_cached_keys.When PyType_Ready() is called, the caller is supposed to be done with initializing the type members, and so it's ok to set (or override) some members like
ht_cached_keys.I hacked locally PyType_Ready() to initialize ht_cached_keys. With this change, building pikepdf documentation with Sphinx no longer crashes.
- added a commit that references this issue
on Aug 18, 2022 1 remaining item
I hacked locally PyType_Ready() to initialize ht_cached_keys. With this change, building pikepdf documentation with Sphinx no longer crashes.
We seem to have two ways of fixing the crash now:
- IIUC the above is a change in Python 3.11
- Undo attr.h, detail/class.h changes made under PR #3923 pybind/pybind11#4142
Should I merge the pybind11 PR? — I'm guessing yes, because that enables testing with Python 3.11rc1. Could you please confirm?
@pablogsal this issue should be fixed. I think it shouldn't block the 3.11 release.
@pablogsal this issue should be fixed. I think it shouldn't block the 3.11 release.
The issue has the "release blocker" label.
I understood that #96047 fixed pybind11.
That was my understanding but nobody closed the issue so it was unclear if something else was missing. We should close it or remove the release blocker label if nothing else is missing here.
I can no longer reproduce the crash. I confirm that pybind11 was fixed and works around in the 3.11 branch. I close the issue.
I ran commands of the first issue comment, and it no longer crashs. Tested versions:
(env) vstinner@mona$ python -m pip freeze alabaster==0.7.12 asttokens==2.0.8 Babel==2.10.3 backcall==0.2.0 certifi==2022.6.15 charset-normalizer==2.1.1 decorator==5.1.1 deprecation==2.1.0 docutils==0.17.1 executing==1.0.0 idna==3.3 imagesize==1.4.1 ipython==8.5.0 jedi==0.18.1 Jinja2==3.1.2 lxml==4.9.1 MarkupSafe==2.1.1 matplotlib-inline==0.1.6 packaging==21.3 parso==0.8.3 pexpect==4.8.0 pickleshare==0.7.5 pikepdf @ file:///home/vstinner/python/3.11/env/pikepdf Pillow==9.2.0 prompt-toolkit==3.0.31 ptyprocess==0.7.0 pure-eval==0.2.2 Pygments==2.13.0 pyparsing==3.0.9 pytz==2022.2.1 requests==2.28.1 six==1.16.0 snowballstemmer==2.2.0 Sphinx==5.1.1 sphinx-issues==3.0.1 sphinx-rtd-theme==1.0.0 sphinx_design==0.3.0 sphinxcontrib-applehelp==1.0.2 sphinxcontrib-devhelp==1.0.2 sphinxcontrib-htmlhelp==2.0.0 sphinxcontrib-jsmath==1.0.1 sphinxcontrib-qthelp==1.0.3 sphinxcontrib-serializinghtml==1.1.5 stack-data==0.5.0 traitlets==5.3.0 urllib3==1.26.12 wcwidth==0.2.5
The crash occurs while building the pikepdf documentation with Sphinx. Reproduce on Fedora 36 with these commands:
gdb traceback:
Frame 0: crash in _PyDictKeys_StringLookup() because dk=NULL.
Frame 1, specialize_dict_access(): call _PyDictKeys_StringLookup(NULL, name), keys is NULL:
Fedora bug report: https://bugzilla.redhat.com/show_bug.cgi?id=2118215