Skip to content

test_multiprocessing: pickling failures #52580

Closed
Closed
test_multiprocessing: pickling failures#52580
Labels
release-blockerstdlibStandard Library Python modules in the Lib/ directorytype-bugAn unexpected behavior, bug, or error

Description

@skrah
skrah
mannequin
opened on Apr 7, 2010
BPO 8333
Nosy @loewis, @tjguk, @voidspace, @skrah
Files
  • issue8333.patch
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
closed_at = <Date 2010-04-08.00:44:10.576>
created_at = <Date 2010-04-07.13:34:34.901>
labels = ['type-bug', 'library', 'release-blocker']
title = 'test_multiprocessing: pickling failures'
updated_at = <Date 2010-04-08.00:44:10.575>
user = 'https://github.com/skrah'

    bugs.python.org fields:

    activity = <Date 2010-04-08.00:44:10.575>
actor = 'michael.foord'
assignee = 'none'
closed = True
closed_date = <Date 2010-04-08.00:44:10.576>
closer = 'michael.foord'
components = ['Library (Lib)']
creation = <Date 2010-04-07.13:34:34.901>
creator = 'skrah'
dependencies = []
files = ['16805']
hgrepos = []
issue_num = 8333
keywords = ['patch', 'buildbot']
message_count = 12.0
messages = ['102539', '102542', '102555', '102556', '102561', '102564', '102565', '102573', '102574', '102577', '102579', '102581']
nosy_count = 5.0
nosy_names = ['loewis', 'tim.golden', 'jnoller', 'michael.foord', 'skrah']
pr_nums = []
priority = 'release blocker'
resolution = 'accepted'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue8333'
versions = ['Python 2.7']

    Activity

    1. skrah commented on Apr 7, 2010

      @skrah
      skrahmannequin
      on Apr 7, 2010
      MannequinAuthor

      On Windows/amd64, I get loads of pickling errors in test_multiprocessing.

      Type 1 error:

      Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Users\stefan\svn\trunk\lib\multiprocessing\forking.py", line 347, in main
    self = load(from_parent)
  File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 1378, in load
    return Unpickler(file).load()
  File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 858, in load
    dispatch[key](self)
  File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 880, in load_eof
    raise EOFError
EOFError

      Type 2 error:

      ======================================================================
      ERROR: test_fork (main.WithManagerTestQueue)
      ----------------------------------------------------------------------

      Traceback (most recent call last):
  File "..\..\Lib\test\test_multiprocessing.py", line 485, in test_fork
    p.start()
  File "C:\Users\stefan\svn\trunk\lib\multiprocessing\process.py", line 104, in start
    self._popen = Popen(self)
  File "C:\Users\stefan\svn\trunk\lib\multiprocessing\forking.py", line 244, in __init__
    dump(process_obj, to_child, HIGHEST_PROTOCOL)
  File "C:\Users\stefan\svn\trunk\lib\multiprocessing\forking.py", line 167, in dump
    ForkingPickler(file, protocol).dump(obj)
  File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 224, in dump
    self.save(obj)
  File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 419, in save_reduce
    save(state)
  File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 649, in save_dict
    self._batch_setitems(obj.iteritems())
  File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 681, in _batch_setitems
    save(v)
  File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Users\stefan\svn\trunk\lib\multiprocessing\forking.py", line 40, in dispatcher
    self.save_reduce(obj=obj, *rv)
  File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 401, in save_reduce
    save(args)
  File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 548, in save_tuple
    save(element)
  File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 419, in save_reduce
    save(state)
  File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 649, in save_dict
    self._batch_setitems(obj.iteritems())
  File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 681, in _batch_setitems
    save(v)
  File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 419, in save_reduce
    save(state)
  File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 649, in save_dict
    self._batch_setitems(obj.iteritems())
  File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 681, in _batch_setitems
    save(v)
  File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 396, in save_reduce
    save(cls)
  File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 748, in save_global
    (obj, module, name))
PicklingError: Can't pickle <type 'cStringIO.StringO'>: it's not found as cStringIO.StringO
    2. skrah
      added
      stdlibStandard Library Python modules in the Lib/ directory
      type-bugAn unexpected behavior, bug, or error
      on Apr 7, 2010

    3. tjguk commented on Apr 7, 2010

      @tjguk
      tjguk
      on Apr 7, 2010
      Member

      Seeing the same thing on 32-bit WinXP on x86

      On 07/04/2010 14:34, Stefan Krah wrote:
>
> New submission from Stefan Krah<[email protected]>:
>
> On Windows/amd64, I get loads of pickling errors in test_multiprocessing.
>
> Type 1 error:
>
> Traceback (most recent call last):
>    File "<string>", line 1, in<module>
>    File "C:\Users\stefan\svn\trunk\lib\multiprocessing\forking.py", line 347, in main
>      self = load(from_parent)
>    File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 1378, in load
>      return Unpickler(file).load()
>    File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 858, in load
>      dispatch[key](self)
>    File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 880, in load_eof
>      raise EOFError
> EOFError
>
>
> Type 2 error:
>
>

      ======================================================================

      ERROR: test_fork (main.WithManagerTestQueue)
      ----------------------------------------------------------------------

      > Traceback (most recent call last):
>    File "..\..\Lib\test\test_multiprocessing.py", line 485, in test_fork
>      p.start()
>    File "C:\Users\stefan\svn\trunk\lib\multiprocessing\process.py", line 104, in start
>      self._popen = Popen(self)
>    File "C:\Users\stefan\svn\trunk\lib\multiprocessing\forking.py", line 244, in __init__
>      dump(process_obj, to_child, HIGHEST_PROTOCOL)
>    File "C:\Users\stefan\svn\trunk\lib\multiprocessing\forking.py", line 167, in dump
>      ForkingPickler(file, protocol).dump(obj)
>    File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 224, in dump
>      self.save(obj)
>    File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 331, in save
>      self.save_reduce(obj=obj, *rv)
>    File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 419, in save_reduce
>      save(state)
>    File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 286, in save
>      f(self, obj) # Call unbound method with explicit self
>    File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 649, in save_dict
>      self._batch_setitems(obj.iteritems())
>    File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 681, in _batch_setitems
>      save(v)
>    File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 286, in save
>      f(self, obj) # Call unbound method with explicit self
>    File "C:\Users\stefan\svn\trunk\lib\multiprocessing\forking.py", line 40, in dispatcher
>      self.save_reduce(obj=obj, *rv)
>    File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 401, in save_reduce
>      save(args)
>    File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 286, in save
>      f(self, obj) # Call unbound method with explicit self
>    File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 548, in save_tuple
>      save(element)
>    File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 331, in save
>      self.save_reduce(obj=obj, *rv)
>    File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 419, in save_reduce
>      save(state)
>    File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 286, in save
>      f(self, obj) # Call unbound method with explicit self
>    File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 649, in save_dict
>      self._batch_setitems(obj.iteritems())
>    File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 681, in _batch_setitems
>      save(v)
>    File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 331, in save
>      self.save_reduce(obj=obj, *rv)
>    File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 419, in save_reduce
>      save(state)
>    File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 286, in save
>      f(self, obj) # Call unbound method with explicit self
>    File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 649, in save_dict
>      self._batch_setitems(obj.iteritems())
>    File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 681, in _batch_setitems
>      save(v)
>    File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 331, in save
>      self.save_reduce(obj=obj, *rv)
>    File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 396, in save_reduce
>      save(cls)
>    File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 286, in save
>      f(self, obj) # Call unbound method with explicit self
>    File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 748, in save_global
>      (obj, module, name))
> PicklingError: Can't pickle<type 'cStringIO.StringO'>: it's not found as cStringIO.StringO
>
>

      components: Library (Lib)
      messages: 102539
      nosy: skrah
      priority: high
      severity: normal
      stage: needs patch
      status: open
      title: test_multiprocessing: pickling failures
      type: behavior
      versions: Python 2.7

      Python tracker<[email protected]>
      <http://bugs.python.org/issue8333\>

      Python-bugs-list mailing list
      Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/mail%40timgolden.me.uk

    4. jnoller commented on Apr 7, 2010

      @jnoller
      jnollermannequin
      on Apr 7, 2010
      Mannequin

      When did this problem start?

    5. skrah commented on Apr 7, 2010

      @skrah
      skrahmannequin
      on Apr 7, 2010
      MannequinAuthor

      It was introduced in r79623, and the attached patch makes the problem
      go away. I would be happy to apply it, but I perhaps you and Michael
      could analyze if there are unwanted dependencies between multiprocessing
      and unittest.

    6. voidspace commented on Apr 7, 2010

      @voidspace
      voidspace
      on Apr 7, 2010
      Contributor

      I expect the unittest tests will fail with that patch applied however... No problem with switching to just using StringIO in unittest though - preferably with a comment as to why.

    7. loewis commented on Apr 7, 2010

      @loewis
      loewismannequin
      on Apr 7, 2010
      Mannequin

      Perhaps we could run the multiprocessing tests unbuffered?

    8. loewis commented on Apr 7, 2010

      @loewis
      loewismannequin
      on Apr 7, 2010
      Mannequin

      BTW, this is also causing all Windows buildbots to fail.

    9. loewis
      added
      release-blocker
      on Apr 7, 2010

    10. voidspace commented on Apr 7, 2010

      @voidspace
      voidspace
      on Apr 7, 2010
      Contributor

      TestResult creates its StringIO buffers even if they aren't use. I can create them on first use and use StringIO.StringIO rather than cStringIO. Will fix now.

    11. voidspace commented on Apr 7, 2010

      @voidspace
      voidspace
      on Apr 7, 2010
      Contributor

      Although merely switching to StringIO may not be enough. TestResult objects keep a reference to sys.stdout when they are instantiated and you can't pickle that either.

      It is preferable for the reference to be taken at instantiation time rather than stored as a module level reference to be compatible with test runners (like regrtest) that *themselves* replace sys.stdout...

    12. voidspace commented on Apr 7, 2010

      @voidspace
      voidspace
      on Apr 7, 2010
      Contributor

      Changing unittest.TestResult to use StringIO instead of cStringIO isn't sufficient. regrtest.py also needs to use StringIO...

    13. voidspace commented on Apr 8, 2010

      @voidspace
      voidspace
      on Apr 8, 2010
      Contributor

      I believe this issue to now be fixed on trunk. We can close this when the buildbots are passing again.

    14. voidspace commented on Apr 8, 2010

      @voidspace
      voidspace
      on Apr 8, 2010
      Contributor

      Green build on Windows XP, trunk.

      http://www.python.org/dev/buildbot/all/builders/x86%20XP-5%20trunk/builds/282

    15. voidspace
      closed this as completedon Apr 8, 2010
    16. ezio-melotti
      transferred this issue fromon Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

    Metadata

    Metadata

    Assignees

    No one assigned

      Labels

      release-blockerstdlibStandard Library Python modules in the Lib/ directorytype-bugAn unexpected behavior, bug, or error

      Projects

      No projects

        Milestone

        No milestone

        Relationships

        None yet

        Development

        No branches or pull requests

        Issue actions