Repository navigation
shutil rmtree fails on readonly files in Windows #63842
Description
| BPO | 19643 |
|---|---|
| Nosy | @pfmoore, @tjguk, @bitdancer, @zware |
| Files |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = 'https://github.com/tjguk'
closed_at = <Date 2014-05-07.17:34:58.251>
created_at = <Date 2013-11-18.12:11:11.635>
labels = ['type-feature', 'library', 'OS-windows']
title = 'shutil rmtree fails on readonly files in Windows'
updated_at = <Date 2014-10-16.17:37:23.941>
user = 'https://bugs.python.org/ivanradic'bugs.python.org fields:
activity = <Date 2014-10-16.17:37:23.941>
actor = 'paul.moore'
assignee = 'tim.golden'
closed = True
closed_date = <Date 2014-05-07.17:34:58.251>
closer = 'tim.golden'
components = ['Library (Lib)', 'Windows']
creation = <Date 2013-11-18.12:11:11.635>
creator = 'ivan.radic'
dependencies = []
files = ['35168', '35170']
hgrepos = []
issue_num = 19643
keywords = ['patch']
message_count = 27.0
messages = ['203285', '203297', '203298', '203301', '203308', '203313', '203314', '203315', '203316', '208647', '208651', '208654', '208657', '208662', '213867', '217997', '218021', '218023', '218044', '218051', '218052', '218053', '218054', '218063', '229538', '229541', '229542']
nosy_count = 8.0
nosy_names = ['paul.moore', 'tim.golden', 'r.david.murray', 'python-dev', 'zach.ware', 'Jovik', 'ivan.radic', 'sushisource']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue19643'
versions = ['Python 3.5']
Activity
ivanradic commentedon Nov 18, 2013
shutil.rmtree works nice on Windows until it hits file with read only attribute set. Workaround is to provide a onerror parameter as a function that checks and removes file attribute before attempting to delete it. Can option to delete read_only files be integrated in shutil.rmtree?
Example output in In Python 2.7:
shutil.rmtree("C:\\2")
bitdancer commentedon Nov 18, 2013
You are essentially asking that we have an option to make the windows behavior mirror the posix behavior? (A read only file in a writable directory can be deleted in posix, since only the directory entry, not the file, is being deleted.)
That makes some sense to me. I wonder what the windows devs think?
tjguk commentedon Nov 18, 2013
This, unfortunately, is the classic edge-case where intra-platform consistency and inter-platform consistency clash.
I (on Windows) would certainly be surprised if a tree delete removed read-only files without my specifying some kind of override. I understand why cross-platform behaviour might be preferred, and I'm open to being convinced, but I start at -0.
bitdancer commentedon Nov 18, 2013
Well, it would *definitely* need to be a new explicit option whose default value was the current behavior.
ivanradic commentedon Nov 18, 2013
Actually, your explanation is perfect.
I want to be able to remove some directory after I am done using it. When similar operation is done through file manager, dialog pops up asking for confirmation, I would like to have function parameter equivalent of "yes to all" dialog that file manager gives me.
The thing is, anyone working with files is used to think in "rm -rf" kind of way, and on Windows read_only files break this workflow. I discovered this problem few days ago when I was working on custom backup script that needs to work both on Linux (at home) and Windows (at work). Currently, I need to have some extra *windows only* code just to be able to successfully remove a directory.
Quick Google search discovered the workaround (http://stackoverflow.com/questions/1889597/deleting-directory-in-python), so I am set, but the original
question: "Why oh why is this such a pain?"
and the comment: "Maybe nobody has taken the five minutes to file a bug at bugs.python.org"
resonated in my head long enough to give it a try.
For me it makes sense to have this option configurable. And it make a ton of sense to support one line equivalent of "rm -rf".
zware commentedon Nov 18, 2013
I like the idea of a remove_readonly flag. I was going to say that I'm a bit worried about the fact that shutil.rmtree already has a couple of keyword arguments, but it's nowhere near what, say, copytree has. Call me +0.75.
bitdancer commentedon Nov 18, 2013
That's not a good name for the flag. The problem is that 'read-only' means different things on Windows than it does on Unix. Which is why I suggested that the flag control whether or not it acts "posix like" on Windows. So perhaps 'posix_compat'? That feels a bit odd, though, since on posix it does nothing...so it's really about behavioral consistency across platforms....
Hmm. It's really hard to think of a name that conveys succinctly what we are talking about here.
A more radical notion would be something like 'delete_control' and have it be tri-valued: 'unixlike', 'windowslike', and 'native', with the default being native. Bad names, most likely, but you get the idea. The disadvantage is that it would be even more code to implement ;)
tjguk commentedon Nov 18, 2013
TBH I'm still fairly -0 and edging towards -0.5. If we didn't already
have two keyword args I might be convinced towards a jfdi=True flag.
But, as the OP described, the current params already allow for a
workaround of sorts and another param of the semantics we're discussing
would surely just be confusing?
(Or you could just "preprocess" on Windows, eg attrib -r /s)
7 remaining items
tjguk commentedon May 6, 2014
Ok, so to move this forward we have essentially two proposals:
Add a remove_readonly flag
Add a doc example which shows how to use the onerror handler to remove a recalcitrant file.
I'm -0.5 on (1) because it feels like Windows-specific clutter; and +0 on (2).
zware commentedon May 6, 2014
I'm good with just adding an example to the docs, along the lines of Paul's del_rw. I think it would be better to use a more conservative example though, something like:
pfmoore commentedon May 6, 2014
Checking the exact error could be a bit fragile. I have a feeling I recently saw an issue somewhere with code that stopped working on Python 3.4 because the precise error raised for a read-only file changed. I don't recall where the issue was, unfortunately.
It's also worth noting that trapping too broad a set of errors won't actually matter much, because the retry will simply fail again if the actual problem wasn't a read-only file...
tjguk commentedon May 7, 2014
The attached patch adds an example to the shutil documentation showing how to use an onerror handler to reattempt the removal of a read-only file. It's deliberately low-tech and simply removes the attribute and retries. If there's some other obstacle, it will continue to fail as it would have done in any case.
zware commentedon May 7, 2014
Fair point, Paul.
Patch looks good to me, Tim, barring a couple of nits pointed out on Rietveld.
tjguk commentedon May 7, 2014
Thanks, Zach. Updated patch.
zware commentedon May 7, 2014
LGTM!
tjguk commentedon May 7, 2014
Thanks. I'll hold off pushing until I've had a chance to run it on a
Unix system. I'm not 100% whether it will operate in the same way there.
python-dev commentedon May 7, 2014
New changeset 31d63ea5dffa by Tim Golden in branch 'default':
bpo-19643 Add an example of shutil.rmtree which shows how to cope with readonly files on Windows
http://hg.python.org/cpython/rev/31d63ea5dffa
New changeset a7560c8f38ee by Tim Golden in branch 'default':
bpo-19643 Fix whitespace
http://hg.python.org/cpython/rev/a7560c8f38ee
sushisource commentedon Oct 16, 2014
Although this is closed, I stumbled across it while looking to see if this behavior had changed at all recently, and I have a suggestion I think might work.
How about we take Tim's example error function which was added to the docs, and it's bound to something like shutil.REMOVE_WINDOWS_READONLY, so it can be used in the following way (probably with a better name):
Good idea? Too weird?
bitdancer commentedon Oct 16, 2014
I think it is an interesting idea. Probably worth opening a new enhancement request with the suggestion.
pfmoore commentedon Oct 16, 2014
Not a bad idea. I would want the implementation to remain in the documentation as well, as code that wants to be portable back to earlier versions of Python will need it. But for code that targets Python 3.5+ only, having it available "out of the box" is a nice addition.