Skip to content

concurrent.futures deadlock #80047

Closed
Closed
concurrent.futures deadlock#80047
Labels
3.7 (EOL)end of life3.8 (EOL)end of lifestdlibStandard Library Python modules in the Lib/ directorytype-bugAn unexpected behavior, bug, or error

Description

@jwilk
jwilk
mannequin
opened on Jan 31, 2019
BPO 35866
Nosy @gpshead, @brianquinlan, @pitrou, @vstinner, @jwilk, @MojoVampire, @cagney, @hroncok, @pablogsal
Files
  • cf-deadlock.py
  • gdb-bt-parent.txt
  • gdb-bt-child.txt
  • cf-deadlock-alarm.py: Version of cf-deadlock.py that tries to always exit (eventually)
  • cf-deadlock-1.py
  • gdb.sh
  • stack-python.txt
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
closed_at = <Date 2021-07-03.21:52:31.634>
created_at = <Date 2019-01-31.10:05:11.069>
labels = ['3.7', '3.8', 'type-bug', 'library']
title = 'concurrent.futures deadlock'
updated_at = <Date 2021-07-03.21:53:10.556>
user = 'https://github.com/jwilk'

    bugs.python.org fields:

    activity = <Date 2021-07-03.21:53:10.556>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2021-07-03.21:52:31.634>
closer = 'vstinner'
components = ['Library (Lib)']
creation = <Date 2019-01-31.10:05:11.069>
creator = 'jwilk'
dependencies = []
files = ['48090', '48223', '48224', '48242', '48271', '48272', '48611']
hgrepos = []
issue_num = 35866
keywords = []
message_count = 34.0
messages = ['334618', '334648', '334652', '338376', '338377', '338402', '338501', '338549', '339334', '339361', '339370', '339451', '339464', '340263', '340278', '340343', '340344', '340358', '340359', '340360', '340362', '340430', '340431', '340432', '340442', '340451', '340456', '340459', '340464', '340468', '352707', '396675', '396910', '396929']
nosy_count = 11.0
nosy_names = ['gregory.p.smith', 'bquinlan', 'pitrou', 'vstinner', 'bentoi', 'jwilk', 'josh.r', 'cagney', 'hroncok', 'pablogsal', 'hugh']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue35866'
versions = ['Python 3.7', 'Python 3.8']

    Activity

    1. jwilk commented on Jan 31, 2019

      @jwilk
      jwilkmannequin
      on Jan 31, 2019
      MannequinAuthor

      The attached test program hangs eventually (it may need a few thousand of iterations).

      Tested with Python v3.7.2 on Linux, amd64.

    2. jwilk
      added
      3.7 (EOL)end of life
      stdlibStandard Library Python modules in the Lib/ directory
      type-bugAn unexpected behavior, bug, or error
      on Jan 31, 2019

    3. MojoVampire commented on Feb 1, 2019

      @MojoVampire
      MojoVampiremannequin
      on Feb 1, 2019
      Mannequin

      I've only got 3.7.1 Ubuntu bash on Windows (also amd64) immediately available, but I'm not seeing a hang, nor is there any obvious memory leak that might eventually lead to problems (memory regularly drops back to under 10 MB shared, 24 KB private working set). I modified your code to add a sys.stdout.flush() after the write so it would actually echo the dots as they were written instead of waiting for a few thousand of them to build up in the buffer, but otherwise it's the same code.

      Are you sure you're actually hanging, and it's not just the output getting buffered?

    4. jwilk commented on Feb 1, 2019

      @jwilk
      jwilkmannequin
      on Feb 1, 2019
      MannequinAuthor

      You're right that sys.stdout.flush() is missing in my code; but on Linux it doesn't make a big difference, because multiprocessing flushes stdout before fork()ing.

      And yes, it really hangs.

    5. pablogsal commented on Mar 19, 2019

      @pablogsal
      pablogsal
      on Mar 19, 2019
      Member

      This seem related to https://bugs.python.org/issue35809

    6. pablogsal commented on Mar 19, 2019

      @pablogsal
      pablogsal
      on Mar 19, 2019
      Member

      Could you use gdb/lldb to attach to the process hanging and give us a stack trace?

    7. jwilk commented on Mar 19, 2019

      @jwilk
      jwilkmannequin
      on Mar 19, 2019
      MannequinAuthor

      There are two processes running (parent and child) when the thing hangs.
      I'm attaching GDB backtraces for both.

    8. hugh commented on Mar 20, 2019

      @hugh
      hughmannequin
      on Mar 20, 2019
      Mannequin

      @jwilk: thanks for creating cf-deadlock.py

      I can replicate the test program hang on Fedora 29 with python3-3.7.2-4.fc29.x86_64

      The test program hasn't yet hung on Fedora 29 with older packages, in particular
      python3-3.7.1-4.fc29.x86_64

      My interest is due to the fact that the libreswan.org test suite has started to hang and we don't know why. It might well be this bug.

    9. hugh commented on Mar 21, 2019

      @hugh
      hughmannequin
      on Mar 21, 2019
      Mannequin

      I've filed a Fedora bug report that points to this one: <https://bugzilla.redhat.com/show_bug.cgi?id=1691434\>

    10. vstinner commented on Apr 2, 2019

      @vstinner
      vstinner
      on Apr 2, 2019
      Member

      Any update on this issue? I don't understand why the example hangs.

    11. cagney commented on Apr 2, 2019

      @cagney
      cagneymannequin
      on Apr 2, 2019
      Mannequin

      I've attached a variation on cf-deadlock.py that, should nothing happen for 2 minutes, will kill itself. Useful with git bisect.

    12. cagney commented on Apr 2, 2019

      @cagney
      cagneymannequin
      on Apr 2, 2019
      Mannequin

      I'm seeing cf-deadlock-alarm.py hangs on vanilla python 3.7.[0123] with:

      Linux 5.0.5-100.fc28.x86_64 #1 SMP Wed Mar 27 22:16:29 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
      glibc-2.27-37.fc28.x86_64

      can anyone reproduce this?

      I also wonder if this is connected to bpo-6721 where a recent "fix" made things worse - the fedora versions that work for libreswan don't have the "fix".

    13. cagney commented on Apr 4, 2019

      @cagney
      cagneymannequin
      on Apr 4, 2019
      Mannequin

      More info from adding a faulthandler ...

      15	def f():
16	    import ctypes
17	
18	for i in range(0,50):
19	    sys.stdout.write("\r%d" % i)
20	    sys.stdout.flush()
21	    signal.alarm(60*2)
22	    for j in range(0,1000):
23	        with concurrent.futures.ProcessPoolExecutor() as executor:
24	            ftr = executor.submit(f)
25	            ftr.result()

      Thread 0x00007f1ce7fff700 (most recent call first):
      File "/home/python/v3.7.3/lib/python3.7/threading.py", line 296 in wait
      File "/home/python/v3.7.3/lib/python3.7/multiprocessing/queues.py", line 224 in _feed
      File "/home/python/v3.7.3/lib/python3.7/threading.py", line 865 in run
      File "/home/python/v3.7.3/lib/python3.7/threading.py", line 917 in _bootstrap_inner
      File "/home/python/v3.7.3/lib/python3.7/threading.py", line 885 in _bootstrap

      Thread 0x00007f1cec917700 (most recent call first):
      File "/home/python/v3.7.3/lib/python3.7/selectors.py", line 415 in select
      File "/home/python/v3.7.3/lib/python3.7/multiprocessing/connection.py", line 920 in wait
      File "/home/python/v3.7.3/lib/python3.7/concurrent/futures/process.py", line 354 in _queue_management_worker
      File "/home/python/v3.7.3/lib/python3.7/threading.py", line 865 in run
      File "/home/python/v3.7.3/lib/python3.7/threading.py", line 917 in _bootstrap_inner
      File "/home/python/v3.7.3/lib/python3.7/threading.py", line 885 in _bootstrap

      Current thread 0x00007f1cfd9486c0 (most recent call first):
      File "/home/python/v3.7.3/lib/python3.7/threading.py", line 296 in wait
      File "/home/python/v3.7.3/lib/python3.7/concurrent/futures/_base.py", line 427 in result
      File "cf-deadlock.py", line 25 in <module>

    14. 10 remaining items

    15. cagney commented on Apr 17, 2019

      @cagney
      cagneymannequin
      on Apr 17, 2019
      Mannequin

      Here's a possible stack taken during the fork():

      Thread 1 "python3" hit Breakpoint 1, 0x00007ffff7124734 in fork () from /lib64/libc.so.6

      Thread 1814 (Thread 0x7fffe69d5700 (LWP 23574)):
      #0 0x00007ffff7bc24e5 in __pthread_mutex_unlock_usercnt () from /lib64/libpthread.so.0
      #1 0x00007ffff71928e3 in dl_iterate_phdr () from /lib64/libc.so.6
      #2 0x00007fffe5fcfe55 in _Unwind_Find_FDE () from /lib64/libgcc_s.so.1
      #3 0x00007fffe5fcc403 in uw_frame_state_for () from /lib64/libgcc_s.so.1
      #4 0x00007fffe5fcd90f in _Unwind_ForcedUnwind_Phase2 () from /lib64/libgcc_s.so.1
      #5 0x00007fffe5fcdf30 in _Unwind_ForcedUnwind () from /lib64/libgcc_s.so.1
      #6 0x00007ffff7bc7712 in __pthread_unwind () from /lib64/libpthread.so.0
      #7 0x00007ffff7bbf7e7 in pthread_exit () from /lib64/libpthread.so.0
      #8 0x000000000051b2fc in PyThread_exit_thread () at Python/thread_pthread.h:238
      #9 0x000000000055ed16 in t_bootstrap (boot_raw=0x7fffe8da0e40) at ./Modules/_threadmodule.c:1021
      #10 0x00007ffff7bbe594 in start_thread () from /lib64/libpthread.so.0
      #11 0x00007ffff7157e5f in clone () from /lib64/libc.so.6

      Thread 1 (Thread 0x7ffff7fca080 (LWP 20524)):
      #0 0x00007ffff7124734 in fork () from /lib64/libc.so.6
      #1 0x0000000000532c8a in os_fork_impl (module=<optimized out>) at ./Modules/posixmodule.c:5423
      #2 os_fork (module=<optimized out>, _unused_ignored=<optimized out>) at ./Modules/clinic/posixmodule.c.h:1913

      where, in my source code, dl_iterate_phdr() starts with something like:

      /* Make sure nobody modifies the list of loaded objects. */
      __rtld_lock_lock_recursive (GL(dl_load_write_lock));

      i.e., when the fork occures, the non-fork thread has acquired dl_load_write_lock - the same lock that the child will later try to acquire (and hang)

      no clue as to what that thread is doing though; other than it looks like it is trying to generate a backtrace?

    16. cagney commented on Apr 17, 2019

      @cagney
      cagneymannequin
      on Apr 17, 2019
      Mannequin

      run ProcessPoolExecutor with one fixed child (over ride default of #cores)

    17. cagney commented on Apr 17, 2019

      @cagney
      cagneymannequin
      on Apr 17, 2019
      Mannequin

      script to capture stack backtrace at time of fork, last backtrace printed will be for hang

    18. gpshead commented on Apr 17, 2019

      @gpshead
      gpshead
      on Apr 17, 2019
      Member

      I am unable to get cf-deadlock.py to hang on my own builds of pure CPython 3.7.2+ d7cb203 or 3.6.8+ be77fb7 (versions i had in a local git clone).

      which specific python builds are seeing the hang using? Which specific platform/distro version? "3.7.2" isn't enough, if you are using a distro supplied interpreter please try and reproduce this with a build from the CPython tree itself. distros always apply their own patches to their own interpreters.

      ...

      Do realize that while working on this it is fundamentally *impossible* per POSIX for os.fork() to be safely used at the Python level in a process also using pthreads. That this _ever_ appeared to work is a pure accident of implementations of underlying libc, malloc, system libraries, and kernel behaviors. POSIX considers it undefined behavior. Nothing done in CPython can avoid that. Any "fix" for these kinds of issues is merely working around the inevitable which will re-occur.

      concurrent.futures.ProcessPoolExecutor uses multiprocessing for its process management. As of 3.7 ProcessPoolExecutor accepts a mp_context parameter to specify the multiprocessing start method. Alternatively the default appears
      to be controllable as a global setting https://docs.python.org/3/library/multiprocessing.html#contexts-and-start-methods.

      Use the 'spawn' start method and the problem should go away as it'll no longer be misusing os.fork(). You _might_ be able to get the 'forkserver' start method to work, but only reliably if you make sure the forkserver is spawned _before_ any threads in the process (such as ProcessPoolExecutor's own queue management thread - which appears to be spawned upon the first call to .submit()).

    19. cagney commented on Apr 17, 2019

      @cagney
      cagneymannequin
      on Apr 17, 2019
      Mannequin

      We're discussing vanilla Python, for instance v3.7.0 is:

      git clone .../cpython
      cd cpython
      git checkout v3.7.0
      ./configure --prefix=/home/python/v3.7.0
      make -j && make -j install

      (my 3.6.x wasn't vanilla, but I clearly stated that)

      Like I also mentioned, loading down the machine also helps. Try something like running #cores*2 of the script in parallel?

    20. cagney commented on Apr 17, 2019

      @cagney
      cagneymannequin
      on Apr 17, 2019
      Mannequin

      @gregory.p.smith, I'm puzzled by your references to POSIX and/or os.fork().

      The code in question looks like:

      import concurrent.futures
import sys

def f():
    import ctypes

while True:
    with concurrent.futures.ProcessPoolExecutor() as executor:
        ftr = executor.submit(f)
        ftr.result()

      which, to me, looks like pure Python.

      Are you saying that this code can't work on GNU/Linux systems.

    21. gpshead commented on Apr 17, 2019

      @gpshead
      gpshead
      on Apr 17, 2019
      Member

      concurrent.futures.ProcessPoolExecutor uses both multiprocessing and threading. multiprocessing defaults to using os.fork().

    22. cagney commented on Apr 18, 2019

      @cagney
      cagneymannequin
      on Apr 18, 2019
      Mannequin

      So:

      #1 we've a bug: the single-threaded ProcessPoolExecutor test program should work 100% reliably - it does not

      #2 we've a cause: ProcessPoolExecutor is implemented internally using an unfortunate combination of fork and threads, this is causing the deadlock

      #3 we've got a workaround - something like:
   ProcessPoolExecutor(multiprocessing.get_context('spawn'))
but I'm guessing, the documentation is scant.

      As for a fix, maybe:

      • have ProcessPoolExecutor use 'spawn' by default; this way things always work
      • have ProcessPoolExecutor properly synchronized its threads before "spawning"/"forking"/... so that "single-threaded" code works
      • document that combining ProcessPoolExecutor's "fork" option and user threads isn't a good idea

    23. gpshead commented on Apr 18, 2019

      @gpshead
      gpshead
      on Apr 18, 2019
      Member

      "the single-threaded ProcessPoolExecutor test program"

      I doubt it is single threaded, the .submit() method appears to spawn a thread internally.

    24. bentoi commented on Sep 18, 2019

      @bentoi
      bentoimannequin
      on Sep 18, 2019
      Mannequin

      FYI, I'm getting a similar deadlock in a child Python process which is stuck on locking a mutex from the dl library. See attached stack.

      I'm not using concurrent.futures however, the parent Python process is a test driver that uses threading.Thread and subprocess.Popen to spawn new processes... I'm not using os.fork().

      This occurred on ArchLinux with Python 3.7.4.

    25. vstinner commented on Jun 28, 2021

      @vstinner
      vstinner
      on Jun 28, 2021
      Member

      #6 0x00007ffff7bc7712 in __pthread_unwind () from /lib64/libpthread.so.0
      #7 0x00007ffff7bbf7e7 in pthread_exit () from /lib64/libpthread.so.0
      #8 0x000000000051b2fc in PyThread_exit_thread () at Python/thread_pthread.h:238
      #9 0x000000000055ed16 in t_bootstrap (boot_raw=0x7fffe8da0e40) at ./Modules/_threadmodule.c:1021

      The thread_run() function (previously "t_bootstrap()") which is the low-level C function to run a thread in Python _thread.start_new_thread() no longer calls PyThread_exit_thread(): see bpo-44434.

      I'm able to reproduce the issue using attached cf-deadlock.py with Python 3.8:

      • Run 3 processes running cf-deadlock.py
      • Run the Python test suite to stress the machine: ./python -m test -r -j4 # try different -j values to stess the machine more or less

      In the main branch (with bpo-44434 fix), I can no longer reproduce the issue. I ran cf-deadlock.py in 4 terminals in parallel with "./python -m test -r -j2" in a 5th terminal for 5 minutes. I couldn't reproduce the issue. On Python 3.8, I reproduced the issue in less than 1 minute.

      Can someone please confirm that the issue is now fixed? Can we mark this issue as a duplicate of bpo-44434?

    26. jwilk commented on Jul 3, 2021

      @jwilk
      jwilkmannequin
      on Jul 3, 2021
      MannequinAuthor

      I can no longer reproduce the bug with Python from git.

    27. vstinner
      closed this as completedon Jul 3, 2021

    28. vstinner commented on Jul 3, 2021

      @vstinner
      vstinner
      on Jul 3, 2021
      Member

      Great! I close the issue.

    29. ezio-melotti
      transferred this issue fromon Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

    Metadata

    Metadata

    Assignees

    No one assigned

      Labels

      3.7 (EOL)end of life3.8 (EOL)end of lifestdlibStandard Library Python modules in the Lib/ directorytype-bugAn unexpected behavior, bug, or error

      Projects

      No projects

        Milestone

        No milestone

        Relationships

        None yet

        Development

        No branches or pull requests

        Issue actions