Repository navigation
[2.7.18.16] Update bundled Expat to 2.9.0 (30 expat CVEs) - #104
Conversation
Update the vendored Expat in Modules/expat from 2.6.4 to 2.9.0. The files are taken from 3.10.22.x (95dadd7, the 3.10 backport of pythongh-158876), which fixes these Expat CVEs: CVE-2024-8176 CVE-2025-59375 CVE-2026-24515 CVE-2026-25210 CVE-2026-32776 CVE-2026-32777 CVE-2026-32778 CVE-2026-41080 CVE-2026-45186 CVE-2026-50219 CVE-2026-56131 CVE-2026-56132 CVE-2026-56403 CVE-2026-56404 CVE-2026-56405 CVE-2026-56406 CVE-2026-56407 CVE-2026-56408 CVE-2026-56409 CVE-2026-56410 CVE-2026-56411 CVE-2026-56412 CVE-2026-66046 CVE-2026-72522 CVE-2026-76641 CVE-2026-76956 CVE-2026-76957 CVE-2026-77214 CVE-2026-93990 CVE-2026-102633 Expat 2.9.0 adds xcs.c (and the new headers fallthrough.h, hash_table.h, memory_sanitizer.h and xcs.h). xcs.c is added to setup.py's expat_sources, to Modules/Setup.dist and to PCbuild/_elementtree.vcxproj, which on Windows compiles the vendored sources. The stale macconfig.h entry in that project is replaced, as upstream did. 2.7.18.x keeps the two local changes it already had on top of 2.6.4: - expat_external.h does not include pyexpatns.h. On Windows, pyexpat.vcxproj compiles pyexpat.c against these headers but links the ActiveState libexpat.lib, so the symbols must keep their XML_* names. - xmlparse.c's PREFIX type is renamed to PPREFIX, because PC/pyconfig.h defines a PREFIX macro. The Windows pyexpat therefore builds against 2.9.0 headers. The external expat it links must be 2.9.0 too. pyexpat.c is unchanged. Expat 2.9.0 deprecates the XML_Size position getters, XML_GetInputContext and XML_SetHashSalt, so gcc now warns about them. They still work, and switching pyexpat.c to the *64 replacements would stop it from linking against an Expat older than 2.9.0. test_pyexpat gains test_deeply_nested_entities, which uses a chain of 200000 nested general entities (CVE-2024-8176). It segfaults with the bundled 2.6.4 and passes with 2.9.0. Co-Authored-By: Claude Opus 5.5 <[email protected]>
ezequielp-activestate
left a comment
ezequielp-activestate
left a comment
There was a problem hiding this comment.
Requesting changes: one blocker, the Windows _elementtree build.
Blocker: _elementtree.pyd will not compile on Windows. Expat 2.9.0 moved the rand_s entropy code out of xmlparse.c into random_rand_s.c/random_rand_s.h. xmlparse.c now does #include "random_rand_s.h" under #if defined(_WIN32) && ! defined(XML_POOR_ENTROPY) (Modules/expat/xmlparse.c:153-155, used at :1135). That file isn't vendored here; upstream CPython doesn't vendor it either and instead builds with XML_POOR_ENTROPY. On this branch XML_POOR_ENTROPY is defined only by setup.py. PCbuild/_elementtree.vcxproj:65 compiles xmlparse.c with _CRT_SECURE_NO_WARNINGS;USE_PYEXPAT_CAPI;XML_STATIC, and nothing in PC/, winconfig.h or expat_config.h defines it. Preprocessing xmlparse.c as Windows would:
- this PR,
-D_WIN32:xmlparse.c:154:12: fatal error: random_rand_s.h: No such file or directory - this PR,
-D_WIN32 -DXML_POOR_ENTROPY: fine 2.7.18.16(Expat 2.6.4),-D_WIN32: fine, because 2.6.4 still hadwriteRandomBytes_rand_sinsidexmlparse.c
Our 3.10.22.x line, which these sources came from, defines XML_POOR_ENTROPY in both PCbuild/_elementtree.vcxproj and PCbuild/pyexpat.vcxproj. Adding it to _elementtree.vcxproj's PreprocessorDefinitions fixes this, and matches what Linux already does. Hash salting is unaffected: Python sets the salt explicitly, which is what the stacked hash-salt PR makes strong.
Checked
- Sources: compared with the signed upstream release tarball
expat-2.9.0.tar.xz(sha2561e637186…3051; I couldn't fetch the signing key, so the signature itself is unchecked). Every vendored file is byte-identical tolib/in the tarball, except:- the CPython-only files (
expat_config.h,pyexpatns.h,COPYING, which matches the tarball's); - the
PREFIX→PPREFIXrename this branch already carried; - the
_WIN32 && ! XML_POOR_ENTROPYguards upstream CPython also carries. pyexpatns.his identical to upstream CPython main's after its 2.9.0 update.expat_external.hstill doesn't include it, as on the base branch.
- the CPython-only files (
- CVE list: the 30 IDs in the description are exactly the CVEs in libexpat's
Changesfor 2.7.0–2.9.0. The one extra ID there, CVE-2022-25235, is only cited as "similar to". - Linux build (Python 2.7, gcc): builds cleanly apart from four
-Wdeprecated-declarationswarnings inpyexpat.c, as the description says.pyexpat.EXPAT_VERSIONisexpat_2.9.0. - Tests: test_pyexpat, test_xml_etree, test_xml_etree_c, test_minidom, test_sax and test_xmlrpc all pass on both the base and this branch.
- CVE-2024-8176: a document with 200,000 chained general entities segfaults the base (exit 139) and parses here in 0.55 s. A billion-laughs document is now refused with "limit on input amplification factor … breached". Incremental
cElementTreefeeding of a 200,000-element document still works.
Not assessed: no Windows build, and no build against an external or system libexpat. As the description says, Windows pyexpat.pyd links the AS libexpat.lib, so it only gets these fixes once that ingredient moves to 2.9.0.
Expat 2.9.0 moved the rand_s() entropy code out of xmlparse.c into random_rand_s.c/.h, which xmlparse.c includes under "_WIN32 && ! XML_POOR_ENTROPY". CPython does not vendor that file; it builds Expat with XML_POOR_ENTROPY instead, as setup.py already does on POSIX and as 3.10.22.x's PCbuild/_elementtree.vcxproj does. _elementtree.vcxproj compiles the vendored xmlparse.c, so without the define the Windows build fails with "random_rand_s.h: No such file or directory". Python sets the hash salt itself, so Expat's own entropy source is not used. pyexpat.vcxproj does not compile the vendored sources (it links the AS libexpat), so it needs no change. Co-Authored-By: Claude Opus 5.5 <[email protected]>
|
Thanks, you're right. Fixed in 403699f.
I still can't do a real MSVC build here. The Windows x64 build is still owed for this PR. |
ezequielp-activestate
left a comment
ezequielp-activestate
left a comment
There was a problem hiding this comment.
Re-review of 403699f: the blocker is fixed. Approving.
The previous blocker was that the Windows _elementtree build would not compile: Expat 2.9.0's xmlparse.c:153 includes random_rand_s.h (not vendored) unless XML_POOR_ENTROPY is defined. PCbuild/_elementtree.vcxproj:65 now defines it, matching what the 3.10.22.x line does in its vcxproj files.
_elementtree.vcxproj is the only Windows project that compiles Expat's sources on this branch: pyexpat.vcxproj has xmlparse.c, xmlrole.c and xmltok.c commented out (lines 76-78), so it needs no define. On Windows, XML_POOR_ENTROPY only affects Expat's own fallback salt. #105 sets a 16-byte salt from the OS random source for every parser, so hash-flooding protection doesn't depend on it.
Not assessed: a real Windows build.
Updates the vendored Expat in
Modules/expatfrom 2.6.4 to 2.9.0. Part of 2.7.18.16 (Linear PROD-1588).CVEs fixed by Expat 2.6.4 → 2.9.0 (30)
CVE-2024-8176, CVE-2025-59375, CVE-2026-24515, CVE-2026-25210, CVE-2026-32776, CVE-2026-32777, CVE-2026-32778, CVE-2026-41080, CVE-2026-45186, CVE-2026-50219, CVE-2026-56131, CVE-2026-56132, CVE-2026-56403 through CVE-2026-56412, CVE-2026-66046, CVE-2026-72522, CVE-2026-76641, CVE-2026-76956, CVE-2026-76957, CVE-2026-77214, CVE-2026-93990, CVE-2026-102633
Change
xcs.cand the new headersxcs.h,fallthrough.h,hash_table.handmemory_sanitizer.h.xcs.cbuild wiring: added tosetup.py(expat_sources/expat_depends),Modules/Setup.dist, andPCbuild/_elementtree.vcxproj(.filters). On Windows,_elementtree.pydcompiles the vendored sources, so it would fail to link withoutxcs.c. The stalemacconfig.hentry is replaced, as upstream did.expat_external.hstill does not includepyexpatns.h.xmlparse.c'sPREFIXtype is still renamed toPPREFIX, because it clashes withPC/pyconfig.h.pyexpat.cis unchanged. Expat 2.9.0 marks theXML_Sizeposition getters,XML_GetInputContextandXML_SetHashSaltas deprecated, so gcc now prints warnings. These functions still work. Switching to the*64replacements would make pyexpat require Expat ≥ 2.9.0 at link time; see the Windows note below.Windows: the external expat must move to 2.9.0 as well
PCbuild/pyexpat.vcxproj(from "Switch Expat to the AS Platform Expat") compilespyexpat.cagainst the vendoredModules\expatheaders but links$(AS_DEPENDENCIES_DIR)/lib/libexpat.lib. After this PR the headers say 2.9.0, so the AS expat ingredient used for Windows should be 2.9.0 too. I haven't changedpyexpat.cto depend on new 2.9.0 symbols, so linking against 2.8.x should still work, but the headers and the library will disagree on version until the ingredient moves. On Linux, the platform build uses either the vendored copy or--with-system-expat. If it's the system/external expat (the last platform build used 2.8.4), that ingredient also needs 2.9.0 to pick up these fixes.Testing (Linux x86_64, built against prod OpenSSL 1.1.1w, vendored expat)
pyexpat.EXPAT_VERSIONreportsexpat_2.9.0.test_pyexpat.ParseTest.test_deeply_nested_entitiesbuilds a chain of 200,000 nested general entities (CVE-2024-8176). It segfaults on 2.7.18.15 (expat 2.6.4) and passes here in about 0.5 s.Review follow-up
PCbuild/_elementtree.vcxprojdefinesXML_POOR_ENTROPY, as 3.10.22.x does. Without it, Expat 2.9.0'sxmlparse.cincludes the unvendoredrandom_rand_s.hon Windows.