Repository navigation
multiprocessing on Windows does not properly manage class attributes #71836
Description
| BPO | 27649 |
|---|---|
| Nosy | @pfmoore, @tjguk, @bitdancer, @zware, @eryksun, @zooba, @sigmavirus24 |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = <Date 2016-07-29.14:23:49.778>
created_at = <Date 2016-07-29.10:54:15.839>
labels = ['type-bug', 'invalid', 'OS-windows']
title = 'multiprocessing on Windows does not properly manage class attributes'
updated_at = <Date 2016-07-29.14:23:49.776>
user = 'https://github.com/sigmavirus24'bugs.python.org fields:
activity = <Date 2016-07-29.14:23:49.776>
actor = 'r.david.murray'
assignee = 'none'
closed = True
closed_date = <Date 2016-07-29.14:23:49.778>
closer = 'r.david.murray'
components = ['Windows']
creation = <Date 2016-07-29.10:54:15.839>
creator = 'icordasc'
dependencies = []
files = []
hgrepos = []
issue_num = 27649
keywords = []
message_count = 4.0
messages = ['271618', '271622', '271627', '271628']
nosy_count = 7.0
nosy_names = ['paul.moore', 'tim.golden', 'r.david.murray', 'zach.ware', 'eryksun', 'steve.dower', 'icordasc']
pr_nums = []
priority = 'normal'
resolution = 'not a bug'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue27649'
versions = ['Python 3.4', 'Python 3.5']
Activity
sigmavirus24 commentedon Jul 29, 2016
In trying to add support for multiprocessing on Windows in Flake8, I found that the behaviour of the module is significantly different than on Unix. If you have a class that has class-level attributes on Unix, and you change them, put the class into a Queue (or have it attached to an object that moves through a Queue) then the values you set will be preserved [1]. On Windows [2], the attributes are reset.
As a minimal reproduction, I've thrown together this project: https://github.com/sigmavirus24/windows-multiprocessing-bug which is running tox on AppVeyor (https://ci.appveyor.com/project/sigmavirus24/windows-multiprocessing-bug) and Travis CI (https://travis-ci.org/sigmavirus24/windows-multiprocessing-bug) so the behaviours can be compared.
For application developers, it would be wonderful if we could rely on the standard library behaving the same way in this case across both operating systems.
[1] https://travis-ci.org/sigmavirus24/windows-multiprocessing-bug/jobs/148276925#L158 https://travis-ci.org/sigmavirus24/windows-multiprocessing-bug/jobs/148276926#L158 https://travis-ci.org/sigmavirus24/windows-multiprocessing-bug/jobs/148276927#L157
[2] https://ci.appveyor.com/project/sigmavirus24/windows-multiprocessing-bug/build/1.0.2#L24
eryksun commentedon Jul 29, 2016
Queuing the class doesn't do anything here. It gets pickled as a module attribute; the queue isn't marshaling the class dict. For example:
>>> pickletools.dis(pickle.dumps(mccabe.McCabeChecker)) 0: \x80 PROTO 3 2: c GLOBAL 'mccabe McCabeChecker' 24: q BINPUT 0 26: . STOP highest protocol among opcodes = 2Unpickling this is basically getattr(sys.modules['mccabe'], 'McCabeChecker'). So you'll see the same result if you only put ('max_complexity', 10) in the queue and hard code the class in the _target function.
Of course for a system that uses fork() the class attribute is the same in the child process. In 3.4+ you can force spawning by initially calling multiprocessing.set_start_method('spawn'). Then you'll see the same result as in Windows.
sigmavirus24 commentedon Jul 29, 2016
Why did you remove Python 3.3? It's still affected by this behaviour.
bitdancer commentedon Jul 29, 2016
Right this is a fork vs spawn issue, not a bug.
I don't know why Eryk changed the versions the way he did, but FYI we use that field to indicate which version we will apply a patch to, so it is pretty much irrelevant when there isn't actually a bug :)