C stack overflow in the Python 2.7 compiler #66773

Activity

1. KevinDyer commented on Oct 8, 2014

   

   KevinDyermannequin

   on Oct 8, 2014

   MannequinAuthor

   More actions

   The following can be used to generate a file called mega_concat.py:

   N = 2**17
   my_array = []
   for i in range(N):
     my_array.append("\"\"")
   print '+'.join(my_array)

   Then, on Ubuntu 14.04, 32-bit:

   $ python mega_concat.py 
   Segmentation fault (core dumped)
   $ python3 mega_concat.py 
   RuntimeError: maximum recursion depth exceeded during compilation

   Seems like this is a simple out-of-memory issue.
2. KevinDyer

   added

   interpreter-core(Objects, Python, Grammar, and Parser dirs)(Objects, Python, Grammar, and Parser dirs)

   type-crashA hard crash of the interpreter, possibly with a core dumpA hard crash of the interpreter, possibly with a core dump

   on Oct 8, 2014
3. MojoVampire commented on Oct 8, 2014

   

   MojoVampiremannequin

   on Oct 8, 2014

   Mannequin

   More actions

   It's not out of memory at all. It's (probably) a blown stack during compilation optimization. Modern Py3 has "fixed" this by simply preventing silly levels of literal concatenation like this causing indefinite call stack expansion; the older ones just allowed the call stack to grow out of control (and the stack can blow long before memory is exhausted).

   I doubt a fix would be backported, and a fix for Py3 seems unnecessary unless you can come up with a scenario where the recursion depth error is unacceptable (eval-ing arbitrary user input doesn't count, for obvious reasons).
4. KevinDyer commented on Oct 9, 2014

   

   KevinDyermannequin

   on Oct 9, 2014

   MannequinAuthor

   More actions

   Gotcha. Thanks for the explanation.

   Two questions:

   1. Given that it's probably not an out-of-memory issue, is it possible that this could turn into something more malicious? (e.g., an arbitrary code execution vulnerability?)
   2. It's not obvious to me why eval-ing arbitrary user input doesn't count. Isn't an interpreter segfault a bad thing, even if it's a somewhat degenerate use case?
5. vstinner commented on Oct 9, 2014

   

   vstinner

   on Oct 9, 2014

   Member

   More actions

   Then, on Ubuntu 14.04, 32-bit:
   $ python mega_concat.py
   Segmentation fault (core dumped)

   What is your Python version? It looks like Ubuntu Trusty provides Python 2.7.6. I'm unable to reproduce the issue on ArchLinux 32-bit with Python 2.7.8. The issue was maybe fixed between 2.7.6 and 2.7.8.

   [haypo@arch32 ~]$ cat megaconcat.py

   N = 2**17
   my_array = []
   for i in range(N):
     my_array.append("\"\"")
   print('+'.join(my_array))

   [haypo@arch32 ~]$ python2 -V
   Python 2.7.8

   [haypo@arch32 ~]$ ulimit -v 8000
   [haypo@arch32 ~]$ python2 megaconcat.py 
   Traceback (most recent call last):
     File "megaconcat.py", line 5, in <module>
       my_array.append("\"\"")
   MemoryError
6. MojoVampire commented on Oct 9, 2014

   

   MojoVampiremannequin

   on Oct 9, 2014

   Mannequin

   More actions

   You're supposed to use that code to create a file (the output file is just ""+""+""+""+""+""+""+""+""+...+"").

   If you want something that won't MemoryError generating the file, this is a "memory free" version of the code to generate the file:

   import itertools, sys
   sys.stdout.writelines(itertools.repeat('""+', 2**17))
   print('""')

   Run that code (saved as whatever file name you like) and pipe the output to mega_concat.py, then run mega_concat.py to repro.
7. vstinner commented on Oct 9, 2014

   

   vstinner

   on Oct 9, 2014

   Member

   More actions

   You're supposed to use that code to create a file (the output file is just ""+""+""+""+""+""+""+""+""+...+"").

   Oh ok, the bug is in Python compiler. It's probably an overflow of the C stack.

   The bug was fixed in Python 3.3 by the issue bpo-5765: see changeset ab02cd145f56. See also the issue bpo-19098 ("sys.setrecursionlimit(2**30) breaks interactive shell") and the changeset c3df31cbcd0a.

   Nick Coghlan wrote in the issue bpo-5765: "However, agreed on the won't fix for 3.2 and 2.7, although I'd consider it at least for 2.7 if someone went through and worked out a patch that applies cleanly."

   $ python2
   Python 2.7.5 (default, Sep 25 2014, 13:57:38) 
   [GCC 4.8.3 20140911 (Red Hat 4.8.3-7)] on linux2>>> compile('1' + '+1'*10**5, '<string>', 'exec')
   <code object <module> at 0x7f8c5ba2c1b0, file "<string>", line 1>
   >>> compile('1' + '+1'*10**6, '<string>', 'exec')
   Erreur de segmentation (core dumped)
   
   $ python3
   Python 3.3.2 (default, Jun 30 2014, 17:20:03) 
   [GCC 4.8.3 20140624 (Red Hat 4.8.3-1)] on linux
   >>> compile('1' + '+1'*10**3, '<string>', 'exec')
   <code object <module> at 0x7f7298d479c0, file "<string>", line 1>
   >>> compile('1' + '+1'*10**4, '<string>', 'exec')
   Traceback (most recent call last):
     File "<stdin>", line 1, in <module>
   RuntimeError: maximum recursion depth exceeded during compilation
8. vstinner

   changed the title [-]Segmentation fault with string concatenation[/-] [+]C stack overflow in the Python 2.7 compiler[/+] on Oct 9, 2014
9. ayeganov commented on Apr 17, 2015

   

   ayeganovmannequin

   on Apr 17, 2015

   Mannequin

   More actions

   This is a backport of https://bugs.python.org/issue5765. I've pretty much taken the patch directly and applied it. I did have to modify the test code to use a different method.
10. zware commented on Apr 27, 2020

    

    zware

    on Apr 27, 2020

    Member

    More actions

    As 2.7 has reached EOL, I'm closing this issue.
11. zware

    closed this as completedon Apr 27, 2020
12. ezio-melotti

    transferred this issue fromon Apr 10, 2022