Repository navigation
vmprof causes process to hang #116
Description
When enabling vmprof in a large gevent based server process, it outputs data for a few minutes then causes everything to hang.
when attaching gdb to the process and looking at the backtrace I get the following (confirmed I saw the same trace twice):
#1 0x000000000054a8f6 in find_key.47505 ()
#2 0x000000000054ad3a in PyGILState_GetThisThreadState ()
#3 0x00002b268fa19574 in sigprof_handler (sig_nr=<optimized out>, info=<optimized out>, ucontext=<optimized out>) at src/vmprof_main.h:170
#4 <signal handler called>
#5 0x000000000054a95d in find_key.47505 ()
#6 0x00000000004f3f30 in PyGILState_Release ()
#7 0x00002b26913f0f97 in ev_invoke_pending (loop=0x2b2691644620) at libev/ev.c:3155
#8 0x00002b26914206f8 in ev_run (loop=0x2b2691644620, flags=flags@entry=0) at libev/ev.c:3555
#9 0x00002b2691420fa4 in __pyx_pf_6gevent_8corecext_4loop_14run (__pyx_v_self=0x3abcc20, __pyx_v_once=False, __pyx_v_nowait=<optimized out>)
at gevent/gevent.corecext.c:7017
#10 __pyx_pw_6gevent_8corecext_4loop_15run (__pyx_v_self=<gevent.corecext.loop at remote 0x3abcc20>, __pyx_args=<optimized out>, __pyx_kwds=<optimized out>)
at gevent/gevent.corecext.c:6884
#11 0x0000000000536f94 in PyEval_EvalFrameEx ()
#12 0x00002b268fa189da in vmprof_eval (f=<optimized out>, throwflag=<optimized out>) at src/_vmprof.c:55
#13 0x000000000053e2b0 in PyEval_EvalCodeEx ()
#14 0x000000000051d0a0 in function_call ()
#15 0x00000000004c2646 in PyObject_Call ()
#16 0x0000000000531dea in instancemethod_call.8548 ()
#17 0x00000000004c2646 in PyObject_Call ()
#18 0x00000000004c2aa6 in PyEval_CallObjectWithKeywords ()
#19 0x00002b26911e0b2d in g_initialstub (mark=<optimized out>) at greenlet.c:798
#20 0x00002b26911e0480 in g_switch (target=0x3629370, args=(), kwargs=0x0) at greenlet.c:570
#21 0x00000000004b39f7 in PyObject_CallFunctionObjArgs ()
#22 0x00007ffdde724f61 in ?? ()
#23 0x0000000000000008 in ?? ()
#24 0x000000000050b5a7 in PyString_FromString ()
#25 0x00000000004ef3b4 in import_submodule.39103 ()
#26 0x000000001212c3b0 in ?? ()
#27 0x0000000012132990 in ?? ()
#28 0x000000000051d030 in ?? ()
#29 0x000000001214bb18 in ?? ()
#30 0x0000000012197ae8 in ?? ()
#31 0x0000000000000001 in ?? ()
#32 0x0000000012197b50 in ?? ()
#33 0x00007ffdde726540 in ?? ()
#34 0x000000000051d0a0 in function_call ()
#35 0x00000000004c2646 in PyObject_Call ()
#36 0x0000000000531e06 in instancemethod_call.8548 ()
#37 0x00007ffdde725360 in ?? ()
#38 0x00007ffdde725290 in ?? ()
#39 0x0000000000553507 in tupledealloc.24200 ()
#40 0x000000000089f3e0 in ?? ()
#41 0x00007ffdde7275d0 in ?? ()
#42 0x000000000089f3e0 in ?? ()
#43 0x000000001d0652b0 in ?? ()
#44 0x0000000001fe17f0 in ?? ()
#45 0x0000000000000001 in ?? ()
#46 0x0000000000561ed2 in PyErr_Format ()
#47 0x0000000000484356 in find_module.38892 ()
#48 0x00000000004ef3b4 in import_submodule.39103 ()
#49 0x000000001d1565cf in ?? ()
#50 0x00007ffdde7286b8 in ?? ()
#51 0x000000000050ed81 in PyString_FromFormatV ()
#52 0x0000000000561ed2 in PyErr_Format ()
#53 0x00000000008a9860 in ?? ()
#54 0x00000000ffffffff in ?? ()
#55 0x00000000ffffffff in ?? ()
#56 0x0000000002d6fc36 in ?? ()
#57 0x000000001d2613c0 in ?? ()
#58 0x00000000008a9860 in ?? ()
#59 0x00000000ffffffff in ?? ()
#60 0x00007ffdde728810 in ?? ()
#61 0x00000000008a9860 in ?? ()
#62 0x000000001d2f48dc in ?? ()
#63 0x00000000004efd54 in import_module_level.isra.3.39129 ()
#64 0x0000000000000000 in ?? ()
I am not sure what other information would be useful to debug this issue.
Activity
jogo commentedon Feb 16, 2017
Besides this issue, I have found vmprof to be very useful and look forward to using it more in the future, thanks for such as great tool!
planrich commentedon Feb 16, 2017
Can you post some more information about your system? If have seen this issue once, but could not reproduce it. PyGILState_GetThisThreadState seems not to be able to acquire the lock and hangs forever.
jogo commentedon Feb 16, 2017
Python version: python 2.7.3
OS: Ubuntu 12.04 with 4.2.0 kernel
gevent version: 1.1.2
server: based on http://www.gevent.org/gevent.pywsgi.html
I am trying to get our system running under
python-dbgso I can hopefully get more information about about what is running when it hangs forever. After that I can try to create a self contained code base that can reliably cause the issue.@planrich is there any other information that you want?
jogo commentedon Feb 16, 2017
got python with debug information working:
Also our process is single threaded.
planrich commentedon Feb 17, 2017
Thanks, as I see things now the reason for this is the following:
PyGILState_GetThisThreadState looks up the thread local state which uses a mutex (keymutex in find_key, Python/thread.c). Assuming the following (as shown in the gdb stack trace in your first post):
PyGILState_Release also calls find_key and the signal occurs just after the keymutex has been acuqired. The signal in PyGILState_GetThisThreadState (which also tries to acquire keymutex) will hang, forever.
This could have happened most of the previous releases already. Thinking about a fix for that.
planrich commentedon Feb 17, 2017
so good news, I have worked around the issue described above and uploaded a pre release (version 0.4.2.dev1). Could you install that version an try again? pip install vmprof --pre should do
jogo commentedon Feb 17, 2017
@planrich thanks, I tested out 0.4.2.dev1 and its now working!
jogo commentedon Feb 23, 2017
Fix has been released in 0.4.2, closing
planrich commentedon Feb 24, 2017
Released a fix, closing this issue.