The copy_reg module becomes unexpectedly empty in test_cpickle

@warsaw

BPO	25698
Nosy	@warsaw, @doko42, @benjaminp, @vadmium, @serhiy-storchaka
Files	issue25698_skip_tests.patch issue25698_fix_remove_module.patch issue25698_fix_add_module.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/serhiy-storchaka'
closed_at = <Date 2016-02-10.08:36:16.709>
created_at = <Date 2015-11-22.18:58:26.441>
labels = ['extension-modules', 'type-bug']
title = 'The copy_reg module becomes unexpectedly empty in test_cpickle'
updated_at = <Date 2017-08-01.06:09:54.294>
user = 'https://github.com/doko42'

bugs.python.org fields:

activity = <Date 2017-08-01.06:09:54.294>
actor = 'Soujanya. Pasam'
assignee = 'serhiy.storchaka'
closed = True
closed_date = <Date 2016-02-10.08:36:16.709>
closer = 'serhiy.storchaka'
components = ['Extension Modules']
creation = <Date 2015-11-22.18:58:26.441>
creator = 'doko'
dependencies = []
files = ['41748', '41750', '41846']
hgrepos = []
issue_num = 25698
keywords = ['patch']
message_count = 38.0
messages = ['255109', '255114', '255118', '255119', '255120', '255138', '255626', '255627', '255628', '255629', '255630', '255631', '255632', '255646', '255660', '255661', '255662', '255667', '255668', '255671', '255672', '255673', '255674', '255929', '255937', '255954', '255967', '255968', '259198', '259200', '259807', '259821', '259834', '259979', '259991', '259992', '260015', '299603']
nosy_count = 7.0
nosy_names = ['barry', 'doko', 'benjamin.peterson', 'python-dev', 'martin.panter', 'serhiy.storchaka', 'Soujanya. Pasam']
pr_nums = []
priority = 'high'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue25698'
versions = ['Python 2.7']

seen with the 2.7.11 release candidate; I'm not yet 100% if this is the fix for bpo-22995 causing these test regressions.

[ 93/395] test_cpickle
test test_cpickle failed -- multiple errors occurred; run in verbose mode for details
[101/395/2] test_decimal
test test_decimal failed -- Traceback (most recent call last):
  File "/home/packages/python/2.7/python2.7-2.7.11~rc1/Lib/test/test_decimal.py", line 1774, in test_pickle
    e = pickle.loads(pickle.dumps(c, proto))
  File "/home/packages/python/2.7/python2.7-2.7.11~rc1/Lib/pickle.py", line 1380, in dumps
    Pickler(file, protocol).dump(obj)
  File "/home/packages/python/2.7/python2.7-2.7.11~rc1/Lib/pickle.py", line 224, in dump
    self.save(obj)
  File "/home/packages/python/2.7/python2.7-2.7.11~rc1/Lib/pickle.py", line 306, in save
    rv = reduce(self.proto)
AttributeError: 'module' object has no attribute '_reduce_ex'

[108/395/4] test_dictviews
test test_dictviews failed -- Traceback (most recent call last):
  File "/home/packages/python/2.7/python2.7-2.7.11~rc1/Lib/test/test_dictviews.py", line 201, in test_pickle
    pickle.dumps, d.viewkeys(), proto)
  File "/home/packages/python/2.7/python2.7-2.7.11~rc1/Lib/unittest/case.py", line 473, in assertRaises
    callableObj(*args, **kwargs)
  File "/home/packages/python/2.7/python2.7-2.7.11~rc1/Lib/pickle.py", line 1380, in dumps
    Pickler(file, protocol).dump(obj)
  File "/home/packages/python/2.7/python2.7-2.7.11~rc1/Lib/pickle.py", line 224, in dump
    self.save(obj)
  File "/home/packages/python/2.7/python2.7-2.7.11~rc1/Lib/pickle.py", line 306, in save
    rv = reduce(self.proto)
AttributeError: 'module' object has no attribute '_reduce_ex'

[177/395/7] test_imp
test test_imp failed -- Traceback (most recent call last):
  File "/home/packages/python/2.7/python2.7-2.7.11~rc1/Lib/test/test_imp.py", line 60, in test_source
    imp.reload(os)
  File "/home/packages/python/2.7/python2.7-2.7.11~rc1/Lib/os.py", line 727, in <module>
    _copy_reg.pickle(stat_result, _pickle_stat_result, _make_stat_result)
AttributeError: 'module' object has no attribute 'pickle'

[184/395/10] test_int_literal
[185/395/10] test_ioctl
test test_ioctl crashed -- <type 'exceptions.AttributeError'>: 'module' object has no attribute 'pickle'
[186/395/11] test_isinstance
[187/395/11] test_iter
[188/395/11] test_iterlen
[189/395/11] test_itertools
[190/395/11] test_json
test test_json crashed -- <type 'exceptions.AttributeError'>: 'module' object has no attribute 'pickle'
[191/395/12] test_kqueue
test_kqueue skipped -- test works only on BSD
[192/395/12] test_largefile
test test_largefile crashed -- <type 'exceptions.AttributeError'>: 'module' object has no attribute 'pickle'
[193/395/13] test_lib2to3
test test_lib2to3 crashed -- <type 'exceptions.AttributeError'>: 'module' object has no attribute 'pickle'
[194/395/14] test_linecache
test test_linecache crashed -- <type 'exceptions.AttributeError'>: 'module' object has no attribute 'pickle'
[195/395/15] test_list
test test_list crashed -- <type 'exceptions.AttributeError'>: 'module' object has no attribute 'pickle'
[196/395/16] test_locale
[197/395/16] test_logging
test test_logging crashed -- <type 'exceptions.AttributeError'>: 'module' object has no attribute 'pickle'
[198/395/17] test_long
[199/395/17] test_long_future
[200/395/17] test_longexp
[201/395/17] test_macos
test test_macos crashed -- <type 'exceptions.AttributeError'>: 'module' object has no attribute 'pickle'
[202/395/18] test_macostools
test test_macostools crashed -- <type 'exceptions.AttributeError'>: 'module' object has no attribute 'pickle'
[203/395/19] test_macpath
[204/395/19] test_macurl2path
[205/395/19] test_mailbox
test test_mailbox crashed -- <type 'exceptions.AttributeError'>: 'module' object has no attribute 'pickle'
[206/395/20] test_marshal
test test_marshal crashed -- <type 'exceptions.AttributeError'>: 'module' object has no attribute 'pickle'
[207/395/21] test_math
test test_math crashed -- <type 'exceptions.AttributeError'>: 'module' object has no attribute 'pickle'
[208/395/22] test_md5
[209/395/22] test_memoryio
test test_memoryio failed -- multiple errors occurred; run in verbose mode for details
[210/395/23] test_memoryview
[211/395/23] test_mhlib
test test_mhlib crashed -- <type 'exceptions.AttributeError'>: 'module' object has no attribute 'pickle'

Duplicate or related to bpo-25601?

Yes, it is a duplicate of bpo-25601. And I can't reproduce it locally.

Matthias, can you reproduce test failure every time when run tests? If yes, could you please run tests in verbose mode and found what tests in test_cpickle were failed first?

looks like re-running the tests in verbose mode using -w is affected by this issue.

the first cpickle subtest failing is:

======================================================================
ERROR: test_recursive_dict_subclass_and_inst (test.test_cpickle.cStringIOCPicklerFastTests)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/home/packages/python/2.7/python2.7-2.7.11~rc1/Lib/test/test_cpickle.py", line 175, in wrapper
    func(self)
  File "/home/packages/python/2.7/python2.7-2.7.11~rc1/Lib/test/pickletester.py", line 819, in test_recursive_dict_subclass_a
nd_inst
    self.check_recursive_collection_and_inst(MyDict.fromkeys)
  File "/home/packages/python/2.7/python2.7-2.7.11~rc1/Lib/test/pickletester.py", line 790, in check_recursive_collection_and
_inst
    s = self.dumps(y, proto)
  File "/home/packages/python/2.7/python2.7-2.7.11~rc1/Lib/test/test_cpickle.py", line 141, in dumps
    p.dump(arg)
AttributeError: 'module' object has no attribute '_reduce_ex'

In your report test_cpickle has number 93. What other 92 tests were ran before? Could you run tests few times and find minimal list of tests that precede failing test_cpickle?

This failure is not related to bpo-22995 because the same failure was reported in bpo-25601 5 commits before first commit in bpo-22995.

This can be related to bpo-25083. Can you look at the size of copy_reg.pyc and copy_reg.pyo files?

This seems entirely reproducible in a local Xenial sbuild using the source packages in https://launchpad.net/~doko/+archive/ubuntu/toolchain/+packages

Interestingly, inspection of the built artifacts in the schroot after all is done yields:

# find . -name copy_reg.py? | xargs ls -l
-rw-rw-r-- 1 barry barry 5597 Nov 30 16:52 ./Lib/copy_reg.pyc
-rw-rw-r-- 1 barry barry 5553 Nov 30 16:53 ./Lib/copy_reg.pyo
-rw-rw-r-- 1 barry barry 5091 Nov 30 17:03 ./debian/tmp-dbg/usr/lib/python2.7/copy_reg.pyc
-rw-rw-r-- 1 barry barry 5047 Nov 30 17:03 ./debian/tmp-dbg/usr/lib/python2.7/copy_reg.pyo
-rw-rw-r-- 1 barry barry 5091 Nov 30 17:02 ./debian/tmp-shdbg/usr/lib/python2.7/copy_reg.pyc
-rw-rw-r-- 1 barry barry 5047 Nov 30 17:02 ./debian/tmp-shdbg/usr/lib/python2.7/copy_reg.pyo

Those are debug builds, so it seems reasonable they are different sizes than the non-debug artifacts in Lib, right? And since they are self-consistent in size, it seems like it wouldn't be caused by a pyc/pyo write bug.

First failure in test run afaict:

======================================================================
ERROR: test_recursive_dict_subclass_and_inst (test.test_cpickle.cStringIOCPicklerFastTests)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/build/python2.7-96oGYh/python2.7-2.7.11~rc1/Lib/test/test_cpickle.py", line 175, in wrapper
    func(self)
  File "/build/python2.7-96oGYh/python2.7-2.7.11~rc1/Lib/test/pickletester.py", line 819, in test_recursive_dict_subclass_and_inst
    self.check_recursive_collection_and_inst(MyDict.fromkeys)
  File "/build/python2.7-96oGYh/python2.7-2.7.11~rc1/Lib/test/pickletester.py", line 790, in check_recursive_collection_and_inst
    s = self.dumps(y, proto)
  File "/build/python2.7-96oGYh/python2.7-2.7.11~rc1/Lib/test/test_cpickle.py", line 141, in dumps
    p.dump(arg)
AttributeError: 'module' object has no attribute '_reduce_ex'

======================================================================

But if I just run test_cpickle.py, it succeeds!

Could you please add debugging output to check that the copy_reg module is empty if tests failed?

I set a breakpoint where the error occurs. I'm in test_cpickle.py dumps(). Doing cPickle.dumps(arg) in the bp, I get the error. arg is:

(Pdb) arg
{<main.H object at 0x7ff8e2a870d0>: None}
(Pdb) type(arg)
<class 'test.pickletester.MyDict'>

Look again at the AttributeError:

(Pdb) cPickle.dumps(arg)
*** AttributeError: 'module' object has no attribute '_reduce_ex'

First, why is that trying to find the attribute of a module object instead of the MyDict instance? I even tried this just to discount the effects of __main__.H:

(Pdb) cPickle.dumps(type(arg)())
*** AttributeError: 'module' object has no attribute '_reduce_ex'

(as you'd expect, type(arg)() is just an instance of a MyDict.

Two questions: why is it trying to get the attribute on a module object, and why is the attribute it's trying to get '_reduce_ex' instead of '__reduce_ex__'?

The attribute to get is taken from the __reduce_ex___str in cPickle.c, which gets defined as:

#define INIT_STR(S) if (!( S ## _str=PyString_InternFromString(#S)))  return -1;
...
    INIT_STR(__reduce_ex__);

This is a compile-time definition, so it makes no sense that it would be trying to find the wrong attribute in some cases (e.g. when running the full regrtest.py) but not in others (e.g. running just test_cpickle.py). But I can find no other references to a string containing "reduce_ex" so the cPickle version has to be it. And that gets initialized in init_stuff(), called from initcPickle() so I don't see any other way for it to get corrupted.

cPickle.dumps({}) works just fine, so it's an artifact of the MyDict. That's all I've figured out for now.

Default __reduce_ex__ and __reduce__ implementations call copy_reg._reduce_ex. cPickle.dumps({}) works because __reduce_ex__ is not called, dict is supported by pickle directly.

... for protocols < 2.

Err, sorry, it's actually PyDict_GetItem which is the problem. Basically
if you try to import with a very deep stack, doing PyDict_GetItem on
sys.modules can return NULL (fro copy_reg). That confuses import a lot.

I haven't exactly figured out why those tests cause failure. One of them
probably puts unicode strings in sys.modules.

On Sat, Dec 5, 2015, at 06:28, Barry A. Warsaw wrote:

Barry A. Warsaw added the comment:

On Dec 05, 2015, at 07:03 AM, Benjamin Peterson wrote:

>The underlying issue is classic: PyDict_SetItem returns NULL if the stack is
>too deep. This confuses import.

Thanks for digging into this Benjamin. I'm not able to investigate
further
atm, but could you please explain your analysis in more detail? Where's
the
faulty setitem? Where is import getting confused? Why does this (or the
combination I found before) cause the problem?

----------

Python tracker <[email protected]>
<http://bugs.python.org/issue25698\>

Err, sorry, it's actually PyDict_GetItem which is the problem. Basically
if you try to import with a very deep stack, doing PyDict_GetItem on
sys.modules can return NULL (fro copy_reg). That confuses import a lot.

I haven't exactly figured out why those tests cause failure. One of them
probably puts unicode strings in sys.modules.

On Sat, Dec 5, 2015, at 06:28, Barry A. Warsaw wrote:

Barry A. Warsaw added the comment:

On Dec 05, 2015, at 07:03 AM, Benjamin Peterson wrote:

>The underlying issue is classic: PyDict_SetItem returns NULL if the stack is
>too deep. This confuses import.

Thanks for digging into this Benjamin. I'm not able to investigate
further
atm, but could you please explain your analysis in more detail? Where's
the
faulty setitem? Where is import getting confused? Why does this (or the
combination I found before) cause the problem?

----------

Python tracker <[email protected]>
<http://bugs.python.org/issue25698\>

One solution is to skip tests caused deep recursion.

The better solution would be to fix import machinery to not produce empty modules.

Here is a patch that makes import machinery to not left empty module in case of recursion error. I prefer this solution.

Benjamin, could you please make a review?

I don't think this patch is the best way. The correct way is to have and
use PyDict_GetWithError. I wouldn't be surprised if you could crash
Python under this patch by putting some function with an import in
__cmp__ in sys.modules.

On Sun, Feb 7, 2016, at 14:23, Serhiy Storchaka wrote:

Serhiy Storchaka added the comment:

Benjamin, could you please make a review?

----------

Python tracker <[email protected]>
<http://bugs.python.org/issue25698\>

Agreed. Here is a patch that makes PyImport_AddModule to use PyDict_GetWithError and fail earlier. I think it should be applied in 3.x too.

That seems fine except you might as well just put the GetWithError prototype in the header file.

New changeset 81dba6e392c3 by Serhiy Storchaka in branch '2.7':
Issue bpo-25698: Importing module if the stack is too deep no longer replaces
https://hg.python.org/cpython/rev/81dba6e392c3

New changeset 1c2de3b0a474 by Serhiy Storchaka in branch '3.5':
Issue bpo-25698: Prevent possible replacing imported module with the empty one
https://hg.python.org/cpython/rev/1c2de3b0a474

New changeset c711c36cf988 by Serhiy Storchaka in branch 'default':
Issue bpo-25698: Prevent possible replacing imported module with the empty one
https://hg.python.org/cpython/rev/c711c36cf988

That seems fine except you might as well just put the GetWithError prototype in the header file.

Yes, of course. It was temporary placed here to avoid rebuilding all sources.

Thank you Barry and Benjamin for investigating this issue.

Thanks for digging deeply and fixing this!

I see test_cpickle failing with NPE while running the tests with IBM Java.

Exception Stack Trace
--------------------------------

<testsuite tests="144" errors="2" failures="0" skipped="0" name="test.test_cpickle.cPickleTests" time="0.049">
  <testcase name="test_attribute_name_interning" time="0.036"/>
  <testcase name="test_bad_input" time="0.786">
    <error type="java.lang.NullPointerException" message="java.lang.NullPointerException"><![CDATA[Traceback (most recent call last):
  File "/jtctest/apps/ascii/jython/80files/jython_2.7.0/Lib/unittest/case.py", line 332, in run
    testMethod()
  File "/jtctest/apps/ascii/jython/80files/jython_2.7.0/Lib/test/pickletester.py", line 1136, in test_bad_input
    self.assertRaises((IndexError, cPickle.UnpicklingError),
  File "/jtctest/apps/ascii/jython/80files/jython_2.7.0/Lib/unittest/case.py", line 476, in assertRaises
    callableObj(*args, **kwargs)
NullPointerException: java.lang.NullPointerException

Jython used
-------------------
2.7.0

IBM Java version
-------------------
java version "1.8.0_141"
Java(TM) SE Runtime Environment (build 8.0.5.0 - pxp3280sr5-20170725_01(SR5 Beta-1)

Please let me know if the issue is similar or it requires a new bug. Thanks.

Navigation Menu

Repository navigation

The copy_reg module becomes unexpectedly empty in test_cpickle #69884

16 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

The copy_reg module becomes unexpectedly empty in test_cpickle #69884

Description

Activity

doko42 commented on Nov 22, 2015

vadmium commented on Nov 22, 2015

serhiy-storchaka commented on Nov 22, 2015

doko42 commented on Nov 22, 2015

serhiy-storchaka commented on Nov 23, 2015

serhiy-storchaka commented on Nov 23, 2015

warsaw commented on Nov 30, 2015

warsaw commented on Nov 30, 2015

warsaw commented on Nov 30, 2015

serhiy-storchaka commented on Nov 30, 2015

warsaw commented on Nov 30, 2015

serhiy-storchaka commented on Nov 30, 2015

serhiy-storchaka commented on Nov 30, 2015

16 remaining items

benjaminp commented on Dec 5, 2015

benjaminp commented on Dec 5, 2015

serhiy-storchaka commented on Jan 29, 2016

serhiy-storchaka commented on Jan 29, 2016

serhiy-storchaka commented on Feb 7, 2016

benjaminp commented on Feb 8, 2016

serhiy-storchaka commented on Feb 8, 2016

benjaminp commented on Feb 10, 2016

python-dev commented on Feb 10, 2016

serhiy-storchaka commented on Feb 10, 2016

warsaw commented on Feb 10, 2016

SoujanyaPasam commented on Aug 1, 2017

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions