Popen.poll always returns None #46727

Activity

1. jjcogliati commented on Mar 24, 2008

   

   jjcogliatimannequin

   on Mar 24, 2008

   MannequinAuthor

   More actions

   I was trying to use subprocess to run multiple processes, and then wait
   until one was finished. I was using poll() to do this and created the
   following test case:
   #BEGIN
   import subprocess,os

   procs = [subprocess.Popen(["sleep",str(x)]) for x in range(1,11)]
   
   while len(procs) > 0:
       os.wait()
       print [(p.pid,p.poll()) for p in procs]
       procs = [p for p in procs if p.poll() == None]
   #END

   I would have expected that as this program was run, it would remove the
   processes that finished from the procs list, but instead, they stay in
   it and I got the following output:

   #Output
   [(7426, None), (7427, None), (7428, None), (7429, None), (7430, None),
   (7431, None), (7432, None), (7433, None), (7434, None), (7435, None)]
   #above line repeats 8 more times
   [(7426, None), (7427, None), (7428, None), (7429, None), (7430, None),
   (7431, None), (7432, None), (7433, None), (7434, None), (7435, None)]
   Traceback (most recent call last):
     File "./test_poll.py", line 9, in <module>
       os.wait()
   OSError: [Errno 10] No child processes
   #End output

   Basically, even for finished processes, poll returns None.

   Version of python used:
   Python 2.5.1 (r251:54863, Oct 30 2007, 13:45:26)
   [GCC 4.1.2 20070925 (Red Hat 4.1.2-33)] on linux2

   Relevant documentation in Library reference manual 17.1.2
   poll( ) ... Returns returncode attribute.
   ... A None value indicates that the process hasn't terminated yet.
2. jjcogliati

   added

   type-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error

   on Mar 24, 2008
3. roudkerk commented on Apr 3, 2008

   

   roudkerkmannequin

   on Apr 3, 2008

   Mannequin

   More actions

   The problem is that os.wait() does not play nicely with subprocess.py.
   Popen.poll() and Popen.wait() use os.waitpid(pid, ...) which will
   raise OSError if pid has already been reported by os.wait().
   Popen.poll() swallows OSError and by default returns None.

   You can (sort of) fix your program by using
   "p.popen(_deadstate='dead')" in place of "p.popen()". This will make
   poll() return 'dead' instead of None if OSError gets caught, but this
   is undocumented.

   Maybe a subprocess.wait() function could be added which would return a
   tuple

   (pid, exitcode, popen_object)
   

   where popen_object is None if the process is "foreign" (i.e. it was
   not created by the subprocess module).

   It would not be hard to implement this on unix if you don't care about
   thread safety. (On unix Popen.wait() is not thread-safe either, so
   maybe thread safety does not matter.)

   To implement something similar on windows you would probably need to
   use WaitForMultipleObjects() to check whether any process handles are
   signalled, but that would involve patching _subprocess.c or using
   ctypes or pywin32.
4. jjcogliati commented on Apr 4, 2008

   

   jjcogliatimannequin

   on Apr 4, 2008

   MannequinAuthor

   More actions

   Hm. Well, after filing the bug, I created a thread for each subprocess,
   and had that thread do an wait on the process, and that worked fine.
   So, I guess at minimum it sounds like the documentation for poll could
   be improved to mention that it will not catch the state if something
   else does. I think a better fix would be for poll to return some kind
   of UnknownError instead of None if the process was finished, but python
   did not catch it for some reason (like using os.wait() :)
5. lisanke commented on Jun 15, 2008

   

   lisankemannequin

   on Jun 15, 2008

   Mannequin

   More actions

   Isn't this a critical problem. The .poll() function serves as a means to
   check the status of the process started. When it continues to report
   'None' to a process which has already terminated, it creates a false
   positive of a hung process. Dealing with recovery from an actual hung
   process is difficult enough. Having to deal with a bad detection that
   the process ran to completion on top of this, makes the use of
   subprocess difficult.

   Maybe I'm miss applying the .poll() function. I'm trying to detect that
   a process has hung, prior to calling .stdout.readlines(). The
   readlines() will hang my python script if the process is hung. Is there
   another way I should be doing this?

   Thanks,

   Mike
6. reiko commented on Jul 28, 2008

   

   reikomannequin

   on Jul 28, 2008

   Mannequin

   More actions

   I have also run into this problem. If you only use p.poll() and never
   p.wait(), returncode will always remain None.

   roudkerk's workaround doesn't seem to work with the new Popen objects,
   at least in python 2.4. ("unexpected keyword argument '_deadstate'")
   Does anyone have a workaround for subprocess.Popen? Do I have to switch
   to the deprecated popen function(s)?
7. terryjreedy commented on Jun 17, 2010

   

   terryjreedy

   on Jun 17, 2010

   Member

   More actions

   Should this be closed or is this still a problem in 2.7 (release candidate out now, final soon) or 3.1?
8. lisanke commented on Jun 19, 2010

   

   lisankemannequin

   on Jun 19, 2010

   Mannequin

   More actions

   Terry,

   I had long since coded around the problem. At this point, I no longer have
   the test environment to cause the intermittent conditions of the process
   hang. I could code something up, but; your question is the first response to
   this bug report since "reiko <[email protected]>" on 7/28/2008. I
   suggest it can be closed, and would be reopened when encountered again. I
   appears nobody is working in that area of code.

   On Wed, Jun 16, 2010 at 8:13 PM, Terry J. Reedy <[email protected]>wrote:

   Terry J. Reedy <[email protected]> added the comment:

   Should this be closed or is this still a problem in 2.7 (release candidate
   out now, final soon) or 3.1?

   ----------
   nosy: +tjreedy

   ---

   Python tracker <[email protected]>
   <http://bugs.python.org/issue2475\>

   ---
9. terryjreedy

   closed this as completedon Jun 19, 2010
10. ezio-melotti

    transferred this issue fromon Apr 10, 2022
11. norswap commented on Jan 12, 2024

    

    norswap

    on Jan 12, 2024

    More actions

    This issue is still alive and well for me.

    I'm running Python 3.10.13 on Apple Silicon Mac.

    I'm sending Popen.terminate() to a subprocess, and I want to ensure that it does terminate within a certain delay, otherwise I send it Popen.kill(). To check if the process is terminated, I use Popen.poll() — which always returns None, even when the process is not actually terminated (in some case, it doesn't even terminate later).

    A workaround that works: using Popen._internal_poll(_deadstate="dead") — as when the thread is dead it will return "dead" instead of None.

    A potential complication is that I'm running all of this from a signal handler or an atexit handler. I want to make sure that all sub-processes have exited when the main process exits.

    I tried using Popen.wait(0.001) right before the poll call, given the implications of some messages in this thread, however this caused the process to terminate right there and then without any diagnostic whatsoever as to what was happening.

    Can I suggest reopening this and actually fixing the behaviour of poll? Is seems fundamentally broken if it does not actually return the exit code of a completed subprocess.