Repository navigation
"Restart Shell" command leaves pythonw.exe processes running #56749
Description
| BPO | 12540 |
|---|---|
| Nosy | @loewis, @birkenfeld, @terryjreedy, @kbkaiser, @tjguk, @ned-deily, @briancurtin |
| Files |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = 'https://github.com/ned-deily'
closed_at = <Date 2011-08-05.06:43:12.172>
created_at = <Date 2011-07-12.09:21:40.622>
labels = ['expert-IDLE', 'performance']
title = '"Restart Shell" command leaves pythonw.exe processes running'
updated_at = <Date 2011-08-09.08:17:30.823>
user = 'https://bugs.python.org/PeterCaven'bugs.python.org fields:
activity = <Date 2011-08-09.08:17:30.823>
actor = 'Aneesh'
assignee = 'ned.deily'
closed = True
closed_date = <Date 2011-08-05.06:43:12.172>
closer = 'ned.deily'
components = ['IDLE']
creation = <Date 2011-07-12.09:21:40.622>
creator = 'Peter.Caven'
dependencies = []
files = ['22799', '22830', '22837']
hgrepos = []
issue_num = 12540
keywords = ['patch', '3.2regression']
message_count = 36.0
messages = ['140179', '140474', '140486', '140487', '140555', '140773', '140775', '140786', '140872', '140896', '140900', '140933', '140938', '140948', '140954', '140965', '140968', '140970', '140971', '140973', '140974', '140975', '141010', '141037', '141060', '141330', '141344', '141420', '141422', '141595', '141611', '141647', '141648', '141656', '141672', '141810']
nosy_count = 12.0
nosy_names = ['loewis', 'georg.brandl', 'terry.reedy', 'kbk', 'tim.golden', 'ned.deily', 'eli.bendersky', 'brian.curtin', 'sunqiang', 'python-dev', 'Peter.Caven', 'Aneesh']
pr_nums = []
priority = 'high'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'resource usage'
url = 'https://bugs.python.org/issue12540'
versions = ['Python 3.2', 'Python 3.3']
Activity
PeterCaven commentedon Jul 12, 2011
On Windows Vista (x64) the IDLE "Restart Shell" command leaves a "pythonw.exe" process running each time that the command is used.
Observed in Python 3.2.1 release and RC2.
terryjreedy commentedon Jul 15, 2011
I just 'upgraded' to 3.2.1 on my XP machine and I see the same with F5-run, which restarts before running the saved file. This appears to be a nasty regression from 3.2.0 that should have been a release blocker if caught earlier. I believe it merits a quick re-release, once fixed, of either the Windows binary or Python itself depending on whether this is windows specific or not.
The normal behavior when starting IDLE is 2 pythonw.exe processes -- one to run IDLE itself and the other for the attached process that runs user code. Restart/Run starts a third process with a fresh interpreter for user code and the old, orphaned user process should and usually does disappear in 2-3 seconds (on my old, slow machine). (There are tracker issues about this not always happening when a runaway process is stopped with ^C, but is has always worked otherwise.) Each process appears to take about 10+Mb, so anyone doing rapid code/test iterations, as I sometimes do, could easily overfill physical memory.
Closing the IDLE windows does not stop the detached processes, so they have to be killed 1 at a time with 3 clicks each or by rebooting. Neither is pleasant.
Although I should have 3.2.1 loaded at least for reviewing issues, I plan to revert to 3.2.0.
Victor: do you know of any way to test for process extinction on Windows?
loewis commentedon Jul 16, 2011
Terry, please don't overreact. Nobody has noticed it during the *long* rc period of 3.2.1, so it can't be that bad. Actually, I *did* notice, but didn't have the time to submit a bug report.
loewis commentedon Jul 16, 2011
FWIW, it only happens with IDLE; python.exe seems to terminate fine when done.
ned-deily commentedon Jul 18, 2011
This appears to be a Windows-only issue; I'm not able to readily reproduce it on either Linux or OS X. Taking a quick look at diffs between 3.2 and 3.2.1, there aren't a lot of changes in IDLE (Lib/idlelib) and nothing obviously related. There are a number of changes elsewhere in signal handling and process handling, though. The restart_subprocess code is in Lib/idlelib/PyShell.py. If there isn't anything obvious elsewhere, perhaps someone can try to hg bisect it on Windows.
terryjreedy commentedon Jul 21, 2011
This nasty bug really can cause severe problems. If a zombie process ran a tkinter (tk) window, then attempting to logout/restart/shutdown eventually brings up a window I have never seen before: End Process -- EmbeddedMenuWindow. The message window shows a countdown timer that implies that it will shutdown the process automatically when the timer reaches 0. But it does not because the process does not respond. It will sit there indefinitely. One has to click a shutdown button to get rid of it. Then a few seconds later, another window for the next zombie appears. And so on. It there are 50 zombies, then one would have to repeat 50 times. Not acceptible to me.
Since IDLE is my Python workspace and since I plan to soon start running lots of tkinter tests and experiments soon, I am reverting to 3.2.0 until there is a fixed Windows binary.
Adding the two listed Windows experts.
terryjreedy commentedon Jul 21, 2011
Restart is not required to create a zombie. Just start IDLE and quit, and the second, user process does not disappear.
Reverting completely does not seem possible. I first just ran the 3.2 installer and it did not complain, that I noticed, about replacing a newer version, but apparently only replaced .exe and .dll and a few other files but not Lib/*.py. Idle or command prompt window said it was still 3.2.1 and the bug remained.
So I uninstalled, deleted everything left except my .py files in site-packages and another subdirectory, and reinstalled 3.2. Now everything seems to predate 2/22/2011. BUT IDLE and command prompt window *still* report "Python 3.2.1 (default, Jul 10 2011, 21:51:15) [MSC v.1500 32 bit (Intel)] on win32". This is sys.version. Something somewhere (registry?) seems to not be deleted by uninstall. And the bug remains. Could a registry entry possibly affect this?
My system is 7 years old with updated win xp 32 bit.
I then checked the never updated 64 bit Py 3.2 on an 18 month old 64 bit Win 7 laptop and detached user processes *do* disappear as I remember on this machine. It did, however, take 8 sec over restart and 12 after closing, which is longer than I remember for my older and definitely slower machine.
Aneesh commentedon Jul 21, 2011
We are also noticed same issue and reverted to Python 3.1. Whenever we run a script, two new pythonw.exe process is started and it is really irritating to see all in Windows Task Manager.
Last day I killed around 14 Pythonw.exe to clean up everything.
elibendersky commentedon Jul 22, 2011
Is the problem happening only on 64-bit Windows, or 32-bit as well?
sunqiang commentedon Jul 22, 2011
I can reproduce this in 32-bit Windows XP Pro. SP3
terryjreedy commentedon Jul 22, 2011
Good question.
Peter, you said Vista x64. Are you running 32 or 64 bit Python?
My system with the apparently irreversible problem is 32 bit xp home.
I am reluctant to test on my daughter's 64 bit laptop as I do not know that I would be able to revert successfully and I want her to be able to write small programs for schoolwork without extra hassle.
elibendersky commentedon Jul 23, 2011
OK, I can reproduce the problem on a clean virtual machine running a pristine XP home SP2. No Python was previously installed.
Steps to reproduce:
Running steps 2-3 N times leaves N pythonw.exe processes alive
elibendersky commentedon Jul 23, 2011
Am I missing something, or is there no explicit command to kill the subprocess on Windows in PyShell.py
The kill_subprocess method (which does get invoked) of ModifiedInterpreter is:
The subprocess is started with:
Could it be that in earlier versions this ensured the subprocess exited with its parent, but this somehow got modified?
Note that the same code existed in PyShell.py for ages, so it's unlikely that the culprit is there.
11 remaining items
terryjreedy commentedon Jul 24, 2011
I mentioned pipes because half of the subprocess chapter, it seems, talks about them. ASo I got the mis-impression that they are special for subprocess-started processes. But if the subprocess gets the args it needs to connect to a socket, it should not care how it is started.
Anyway, some experiments:
3.1.3 and 3.1.4 freshly installed do not seem to have the zombie problem. This seems to rule out the possibility that the problem is due to recent patches from Microsoft.
I redeleted 3.2 installation and re-installed 3.2. Sys.version still mistakingly say 3.2.1, July 10., so there is something different about the relationship between 3.2 and 3.2.1 and that between 3.1.3 and 3.1.4. And the 3.2 re-install has the zombie problem while I do not believe the fresh 3.2 install did. (And it does not on my other machine.) But I do not see how something stuck in the registry could affect process killing.
In the notepad example, changing
changed the 'pid' from a constant (across multiple runs) to a variable (across multiple runs) and changed the result of the kill from a zombie and exception to proper termination.
When I tried the same fix in idlelib/PyShell.py, adding 'import subprocess' and changing
self.rpcpid = os.spawnv(os.P_NOWAIT, sys.executable, args)
to
self.rpcpid = subprocess.Popen(args).pid
(args begins with sys.executable) IDLE failed to start. The only evidence that it had been invoked was a brief (1/4 second?) appearance of 1 pythonw process in task manager. On a subsequent tries, without touching the file, I do not see even that. Is there any obvious mistake in the above?
elibendersky commentedon Jul 29, 2011
Terry,
""" When I tried the same fix in idlelib/PyShell.py, adding 'import subprocess' and changing
self.rpcpid = os.spawnv(os.P_NOWAIT, sys.executable, args)
to
self.rpcpid = subprocess.Popen(args).pid
(args begins with sys.executable) IDLE failed to start. The only evidence that it had been invoked was a brief (1/4 second?) appearance of 1 pythonw process in task manager. On a subsequent tries, without touching the file, I do not see even that. Is there any obvious mistake in the above? """
No, when I do the same, things seem to go fine. No zombie is left running after IDLE is closed, and even "Restart shell" works without leaving a zombie.
Maybe you had other modifications in your idlelib sources?
Anyway, this wouldn't be a complete fix, because in:
os.waitpid on Windows also expects a process handle, not pid.
I think the complete solution, in addition to replacing os.spawnv by subprocess.Popen, would be to use Popen.kill and then Popen.wait instead of os.kill and then os.wait in the code above. This would require keeping the Popen object somewhere in self.
terryjreedy commentedon Jul 29, 2011
I was working with the freshly reinstalled 3.2 which is not the same as a pristine 3.2 install because it still had the problem that 3.2.1 has and the 3.2.1 sys.version. 3.2.1 uninstall in not complete (a different issue). So I should reinstall 3.2.1 again and try again.
I agree about fully switching t. subprocess. If self.rpcpid is not used anywhere other than unix_terminate (which I think should be renamed)), then self.subproc could replace self.rpcpid.
elibendersky commentedon Jul 30, 2011
Attaching an initial patch for Lib/idlelib/PyShell.py
It uses subprocess.Popen instead of spawn&kill, in the way discussed in earlier messages.
As far as I can tell, IDLE opens and restarts shells successfully, without leaving zombies behind. I only tested it on Windows, however, and due to the lack of unit tests for idlelib there wasn't much verification done.
elibendersky commentedon Jul 30, 2011
I've now tested this on Ubuntu Linux as well. IDLE works, no zombies left behind.
ned-deily commentedon Aug 3, 2011
I've added a couple of review comments to the one Peter already made in Rietveld. Here is an updated patch that addresses all of the comments. I've tested in briefly on Windows and on OS X and it seems to work OK. Eli, if you're OK with it, feel free to commit or I will do it.
PeterCaven commentedon Aug 3, 2011
Terry, sorry about the delay in responding: I'm using 32bit Python. I haven't had a chance yet to try the 64 bit release.
python-dev commentedon Aug 5, 2011
New changeset cc86f4ca5020 by Ned Deily in branch '3.2':
Issue bpo-12540: Prevent zombie IDLE processes on Windows due to changes
http://hg.python.org/cpython/rev/cc86f4ca5020
New changeset c2fd1ce1c6d4 by Ned Deily in branch 'default':
Issue bpo-12540: Prevent zombie IDLE processes on Windows due to changes
http://hg.python.org/cpython/rev/c2fd1ce1c6d4
ned-deily commentedon Aug 5, 2011
With Eli's concurrence, I have applied the updated patch to 3.2 (for 3.2.2) and to default (for 3.3).
elibendersky commentedon Aug 5, 2011
On Fri, Aug 5, 2011 at 09:43, Ned Deily <[email protected]> wrote:
Tested this on Windows XP with Python 3.2 installed into "Program Files".
Works fine (including shell restart).
It would be great to get more people to test it though, especially people
who ran into the problem originally - Peter? Terry? Anish? Qiang Sun?
terryjreedy commentedon Aug 5, 2011
The replacement file, for anyone without a dev setup, is
http://hg.python.org/cpython/file/cc86f4ca5020/Lib/idlelib/PyShell.py
After renaming PyShell to PyShellBak and replacing with the above,
IDLE seems to run better than ever. On my XP system, the several second delay for the old process dying is gone and I never see 3 pythonw processes, even temporarily, as before. I can no longer tell from TaskManager that one process has been killed and another started. Testing included running tk windows. Will test later on new Win7 system.
Pending some unforeseen new problem, thank you Ned and Eli.
Aneesh commentedon Aug 9, 2011
I too retested this on a Windows 7 32 and 64 bit machines and is working fine when the provided PyShell.py is used.
As Terry mentioned, IDLE seems be running better. The process in Task Manager disappears quickly after I close the IDLE.