Cannot Recover From StackOverflow in 3.9 Tests #89964

Activity

1. sweeneyde commented on Nov 15, 2021

   

   sweeneyde

   on Nov 15, 2021

   MemberAuthor

   More actions

   In bpo-30570, David Bolen noticed that "py -3.9 -m test test_pickle" consistently crashes on Windows (even though other methods of running that test do not crash, and the test succeeds when failed tests are retried).

   Curiously, it seems that adding using support.infinite_recursion *introduced* the crash rather than preventing it.

   I'm guessing this would have been fixed by #68689, but that fix was rolled back in #69366 for the sake of 3.9 ABI compatibility.

   As of now, 3.9's pycore_ceval.c reads:

       static inline int _Py_RecursionLimitLowerWaterMark(int limit) {
           if (limit > 200) {
               return (limit - 50);
           }
           else {
               return (3 * (limit >> 2));
           }
       }

   But support.infinite_recursion(max_depth=75) has a default 75, leaving a "low-water-mark" of 54, which the recursion apparently never recovers back to in the right way.

   A couple of solutions could fix this:

   (1) Remove the usage of support.infinite_recursion at the test.pickletester.AbstractPickleTests.test_bad_getattr call site.

   (2) Use a different value max_depth. On my machine at least, it seems 75 was a "perfect storm" to cause this issue. Using infinite_recursion(60) or 61 or ... or 74 or 76 or 77 or 78 all pass, but infinite_recursion(75) in particular fails.

   (3) Use fewer calls after the overflow by inlining something like assertRaises:

                with support.infinite_recursion():
   -                self.assertRaises(RuntimeError, self.dumps, x, proto)
   +                try:
   +                    self.dumps(x, proto)
   +                except RuntimeError:
   +                    pass
   +                else:
   +                    self.fail("RuntimeError not raised")

   (5) Re-visit an ABI-compliant version of #68689, such as #69347

   The output I keep getting without any changes:

   py -3.9 -m test test_pickle -v
   ...
   test_attribute_name_interning (test.test_pickle.CPicklerTests) ... ok
   test_bad_getattr (test.test_pickle.CPicklerTests) ... Fatal Python error: _Py_CheckRecursiveCall: Cannot recover from stack overflow.Python runtime state: initialized

   Current thread 0x000028b0 (most recent call first):
   File "C:\Users\sween\Source\Repos\cpython2\39\lib\test\pickletester.py", line 3300 in __getattr__
   File "C:\Users\sween\Source\Repos\cpython2\39\lib\test\pickletester.py", line 3300 in __getattr__
   File "C:\Users\sween\Source\Repos\cpython2\39\lib\test\pickletester.py", line 3300 in __getattr__
   ...
   File "C:\Users\sween\Source\Repos\cpython2\39\lib\test\pickletester.py", line 3300 in __getattr__
   File "C:\Users\sween\Source\Repos\cpython2\39\lib\test\pickletester.py", line 3300 in __getattr__
   File "C:\Users\sween\Source\Repos\cpython2\39\lib\test\pickletester.py", line 3300 in __getattr__
   File "C:\Users\sween\Source\Repos\cpython2\39\lib\copyreg.py", line 74 in _reduce_ex
   File "C:\Users\sween\Source\Repos\cpython2\39\lib\test\test_pickle.py", line 65 in dumps
   File "C:\Users\sween\Source\Repos\cpython2\39\lib\unittest\case.py", line 201 in handle
   File "C:\Users\sween\Source\Repos\cpython2\39\lib\unittest\case.py", line 739 in assertRaises
   File "C:\Users\sween\Source\Repos\cpython2\39\lib\test\pickletester.py", line 2381 in test_bad_getattr
   File "C:\Users\sween\Source\Repos\cpython2\39\lib\test\support\init.py", line 1770 in wrapper
   File "C:\Users\sween\Source\Repos\cpython2\39\lib\unittest\case.py", line 550 in _callTestMethod
   File "C:\Users\sween\Source\Repos\cpython2\39\lib\unittest\case.py", line 592 in run
   File "C:\Users\sween\Source\Repos\cpython2\39\lib\unittest\case.py", line 651 in __call__
   File "C:\Users\sween\Source\Repos\cpython2\39\lib\unittest\suite.py", line 122 in run
   File "C:\Users\sween\Source\Repos\cpython2\39\lib\unittest\suite.py", line 84 in __call__
   File "C:\Users\sween\Source\Repos\cpython2\39\lib\unittest\suite.py", line 122 in run
   File "C:\Users\sween\Source\Repos\cpython2\39\lib\unittest\suite.py", line 84 in __call__
   File "C:\Users\sween\Source\Repos\cpython2\39\lib\unittest\suite.py", line 122 in run
2. sweeneyde

   added

   3.9 (EOL)end of lifeend of life

   testsTests in the Lib/test dirTests in the Lib/test dir

   type-crashA hard crash of the interpreter, possibly with a core dumpA hard crash of the interpreter, possibly with a core dump

   on Nov 15, 2021
3. jkloth commented on Nov 15, 2021

   

   jkloth

   on Nov 15, 2021

   Contributor

   More actions

   I'll note that it also fails on first run on the Windows 11 builder:

   https://buildbot.python.org/all/#/builders/737/builds/65
4. jkloth

   added

   OS-windows

   on Nov 15, 2021
5. ambv commented on Nov 19, 2021

   

   ambv

   on Nov 19, 2021

   Contributor

   More actions

   New changeset 4296396 by Mark Shannon in branch '3.9':
   [3.9] bpo-45806: Fix recovery from stack overflow for 3.9. Again. (GH-29640)
   4296396
6. ambv commented on Nov 19, 2021

   

   ambv

   on Nov 19, 2021

   Contributor

   More actions

   Thanks! ✨ 🍰 ✨
7. ambv

   closed this as completedon Nov 19, 2021
8. ambv

   closed this as completedon Nov 19, 2021
9. sweeneyde commented on Mar 7, 2022

   

   sweeneyde

   on Mar 7, 2022

   MemberAuthor

   More actions

   Should this be backported to make the 3.8 Buildbots happy?
10. ezio-melotti

    transferred this issue fromon Apr 10, 2022