xml.dom.minidom cannot parse ISO-2022-JP #60081

Activity

1. dcallagh commented on Sep 7, 2012

   

   dcallaghmannequin

   on Sep 7, 2012

   MannequinAuthor

   More actions

   Python 2.7.3 (default, Jul 24 2012, 10:05:38) 
   [GCC 4.7.0 20120507 (Red Hat 4.7.0-5)] on linux2
   Type "help", "copyright", "credits" or "license" for more information.
   >>> c = u'\u65e5\u672c\u8a9e'
   >>> import xml.dom.minidom

   Encoded as UTF-8, everything is fine:

   >>> xml.dom.minidom.parseString('<?xml version="1.0" encoding="UTF-8" ?><x>%s</x>' % c.encode('UTF-8'))
   <xml.dom.minidom.Document instance at 0x7f310d27dcf8>

   but not ISO-2022-JP:

   >>> xml.dom.minidom.parseString('<?xml version="1.0" encoding="ISO-2022-JP" ?><x>%s</x>' % c.encode('ISO-2022-JP'))
   Traceback (most recent call last):
     File "<stdin>", line 3, in <module>
     File "/usr/lib64/python2.7/site-packages/_xmlplus/dom/minidom.py", line 1925, in parseString
       return expatbuilder.parseString(string)
     File "/usr/lib64/python2.7/site-packages/_xmlplus/dom/expatbuilder.py", line 942, in parseString
       return builder.parseString(string)
     File "/usr/lib64/python2.7/site-packages/_xmlplus/dom/expatbuilder.py", line 223, in parseString
       parser.Parse(string, True)
   xml.parsers.expat.ExpatError: not well-formed (invalid token): line 1, column 48

   lxml can handle it fine though:

   >>> import lxml.etree
   >>> lxml.etree.fromstring('<?xml version="1.0" encoding="ISO-2022-JP" ?><x>%s</x>' % c.encode('ISO-2022-JP'))
   <Element x at 0x7f310d284960>
   >>> _.text == c
   True
2. dcallagh

   added

   topic-XML

   on Sep 7, 2012
3. amauryfa commented on Sep 7, 2012

   

   amauryfa

   on Sep 7, 2012

   Contributor

   More actions

   This is similar to bpo-13612: pyexpat does not support multibytes encodings.
4. iritkatriel commented on Sep 30, 2020

   

   iritkatriel

   on Sep 30, 2020

   Member

   More actions

   I don't see this problem on 3.10. Is this still an issue or can this issue be closed?

   Running Release|Win32 interpreter...
   Python 3.10.0a0 (heads/bpo17490-dirty:00eb063b66, Sep 27 2020, 13:20:24) [MSC v.1916 32 bit (Intel)] on win32
   Type "help", "copyright", "credits" or "license" for more information.
   >>> c = u'\u65e5\u672c\u8a9e'
   >>> import xml.dom.minidom
   >>> xml.dom.minidom.parseString('<?xml version="1.0" encoding="UTF-8" ?><x>%s</x>' % c.encode('UTF-8'))
   <xml.dom.minidom.Document object at 0x015FC9E8>
   >>> xml.dom.minidom.parseString('<?xml version="1.0" encoding="ISO-2022-JP" ?><x>%s</x>' % c.encode('ISO-2022-JP'))
   <xml.dom.minidom.Document object at 0x01493208>
   >>>
5. iritkatriel commented on Oct 19, 2020

   

   iritkatriel

   on Oct 19, 2020

   Member

   More actions

   Closing - this now works for me on Python 3.8 and 3.10. It was fixed sometime in the last 8 years.
6. iritkatriel

   closed this as completedon Oct 19, 2020
7. ezio-melotti

   transferred this issue fromon Apr 10, 2022
8. serhiy-storchaka commented on May 27, 2026

   

   serhiy-storchaka

   on May 27, 2026

   Member

   More actions

   It was not fixed. The example just does not work in Python 3. c.encode('ISO-2022-JP') returns the bytes object, and the XML in the example, contains the repr of that object.

   >>> c = u'\u65e5\u672c\u8a9e'
   >>> c.encode('ISO-2022-JP')
   b'\x1b$BF|K\\8l\x1b(B'
   >>> print('<?xml version="1.0" encoding="ISO-2022-JP" ?><x>%s</x>' % c.encode('ISO-2022-JP'))
   <?xml version="1.0" encoding="ISO-2022-JP" ?><x>b'\x1b$BF|K\\8l\x1b(B'</x>

   Limited multibyte encoding support has been added in #62259, but ISO-2022-JP cannot be supported in Expat library. While Python uses Expat as the XML parser, it cannot support encodings like ISO-2022-JP. Try to use third-party XML li=braries not based on Expat.

   There is a workaround. If you know what encoding is used in XML, then you can decode it from bytes to Unicode using Python, and pass the decoded Unicode string to the XML parser.
9. serhiy-storchaka

   reopened this on May 27, 2026
10. serhiy-storchaka

    closed this as not plannedon May 27, 2026