My application needs to pack and unpack workunits all day long and does
this using multiple threading.Threads. I've noticed that the zlib module
seems to use only one thread at a time when using [de]compressobj(). As
the comment in the sourcefile zlibmodule.c already says the module uses
a global lock to protect different threads from accessing the object.
While the c-functions release the GIL while waiting for the global lock,
only one thread at a time can use zlib.
My app ends up using only one CPU to compress/decompress it's workunits...
The patch (svn diff to ) attached here fixes this problem by extending
the compressobj-structure by an additional member to create
object-specific locks and removes the global lock. The lock protects
each compressobj individually and allows multiple python threads to use
zlib in parallel, utilizing all available CPUs. |