Repository navigation
On Unix, shutil.which() and subprocess no longer look for the executable in the current directory if PATH environment variable is not set #79936
Description
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = <Date 2019-04-17.16:53:09.123>
created_at = <Date 2019-01-16.23:41:46.541>
labels = ['type-security', '3.8', 'library']
title = 'On Unix, shutil.which() and subprocess no longer look for the executable in the current directory if PATH environment variable is not set'
updated_at = <Date 2019-04-24.10:54:33.118>
user = 'https://github.com/vstinner'bugs.python.org fields:
activity = <Date 2019-04-24.10:54:33.118>
actor = 'koobs'
assignee = 'none'
closed = True
closed_date = <Date 2019-04-17.16:53:09.123>
closer = 'vstinner'
components = ['Library (Lib)']
creation = <Date 2019-01-16.23:41:46.541>
creator = 'vstinner'
dependencies = []
files = ['48063', '48064', '48065']
hgrepos = []
issue_num = 35755
keywords = ['patch', 'patch', 'patch']
message_count = 29.0
messages = ['333801', '333804', '333807', '333826', '333827', '333830', '333832', '333834', '333835', '333837', '333905', '340352', '340357', '340366', '340384', '340399', '340400', '340401', '340402', '340403', '340404', '340410', '340416', '340418', '340419', '340420', '340422', '340446', '340485']
nosy_count = 8.0
nosy_names = ['gregory.p.smith', 'vstinner', 'giampaolo.rodola', 'christian.heimes', 'jwilk', 'serhiy.storchaka', 'koobs', 'izbyshev']
pr_nums = ['11586', '11586', '11586', '12858', '12861', '12862']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'security'
url = 'https://bugs.python.org/issue35755'
versions = ['Python 3.8']
Activity
vstinner commentedon Jan 16, 2019
Currently, posixpath.defpath is equal to:
defpath = ':/bin:/usr/bin'
It gives 3 directories:
where the empty string means "the current directory". Trying to locate an executable from the current directory can be security issue when an attacker tries to execute arbitrary command.
The Linux exec(3) manual page contains an interesting note about the removal of the empty string from glibc 2.24 by accident:
http://man7.org/linux/man-pages/man3/execvp.3.html
NOTES
Context of this issue: This discussion started from my PR 11579 which modifies the subprocess module to use posix_spawnp():
#11579 (review)
So I propose to replace defpath = ':/bin:/usr/bin' with defpath = '/bin:/usr/bin' which gives 2 directories:
>>> '/bin:/usr/bin'.split(posixpath.pathsep) ['/bin', '/usr/bin']This change would only affect os.get_exec_path(), and so indirectly the subprocess module (when the executable contains no directory), *when the PATH environmant variable is not set*.
vstinner commentedon Jan 16, 2019
I wrote attached execv_curdir.py to check if os.execv() tries to find the executable in the current directory if it doesn't contain a directory: yes, it does.
I also wrote attached subprocess_curdir.py which confirms that subprocess runs a program from the current directory if it exists.
$ python3 subprocess_curdir.py defpath = :/bin:/usr/bin subprocess searchs in the current directoryMoreover, the current directory has the priority over /bin and /usr/bin.
izbyshev commentedon Jan 17, 2019
Would it make sense to use os.confstr('CS_PATH') instead of a hardcoded path, or is identical behavior on all POSIX platforms preferred to that?
gpshead commentedon Jan 17, 2019
ntpath and macpath appear to have the same potential issue.
They've basically had this defpath value forever.
keep following it and you find a commit from 1994 2979b01
Changing them only alters behavior of users of os.defpath or in the (odd?) situation when the PATH environment variable is unset, anything using of shutil.which() or distutils.spawn.find_executabnle(), the only stdlib users of os.defpath.
gpshead commentedon Jan 17, 2019
I'm not arguing against this change, just trying to figure out where it came from in the first place. We should fix the value on all OSes.
It would be a behavior change so probably only good for 3.8+.
tiran commentedon Jan 17, 2019
+1, /usr/bin:/bin sounds good to me.
My /usr/include/paths.h has #define _PATH_DEFPATH "/usr/bin:/bin" and #define _PATH_STDPATH "/usr/bin:/bin:/usr/sbin:/sbin". The file is pretty old and has copyright from 89 and 93, https://code.woboq.org/gcc/include/paths.h.html
vstinner commentedon Jan 17, 2019
I'm working on PR but I found an issue.
shutil.which() behaves differently than subprocess, distutils.spawn.find_executable() and os.execv() when PATH is set but set to an empty string:
Using PATH=":", they all run the program and os.get_exec_path() returns ['', ''].
Who is right? Which behavior do we want for Python?
Note: When PATH is set to an empty string, shutil.which() and distutils.spawn.find_executable() use the empty string, they don't use os.defpath.
vstinner commentedon Jan 17, 2019
I wrote PR 11586 to remove the current directory from os.defpath.
I would prefer to first decide how the os, subprocess, shutil and distutils modules have to handle a PATH variable set to an empty string, before merging my PR. I would prefer to have the same behavior for these modules.
Gregory P. Smith:
Oh, I didn't know that Windows had the same behavior... I didn't know that Windows has a default search path! C:\bin? Who has such directory?
I agree that it's better to have the same behavior on all platforms.
Gregory P. Smith:
I concur. It's a backward incompatible change.
Christian Heimes:
On my Fedora 29, this file comes from the glibc:
According to execvp() manual page, the current directory has only been removed from glibc 2.24 (2016-08-05).
serhiy-storchaka commentedon Jan 17, 2019
This is how the which command behaves:
I think shutil.which() should behave similarly unless there are good reasons for difference.
vstinner commentedon Jan 17, 2019
Alexey Izbyshev:
I didn't know this variable. man confstr says:
_CS_PATH: A value for the PATH variable which indicates where all the POSIX.2 standard utilities can be found.
On my Fedora 29, it only returns '/usr/bin':
On Fedora 29, /bin is a symlink to /usr/bin:
$ ls -ld /bin lrwxrwxrwx. 1 root root 7 13 juil. 2018 /bin -> usr/bin/So it makes sense to omit /bin from the default search path :-)
On Debian Sid where /bin is still a distinct directory than /usr/bin, CS_PATH is equal to "/bin:/usr/bin".
On Fedora, using confstr() would have the advantage of avoiding one useless syscall stat("/bin/program") in addition to stat("/usr/bin/program") in shutil.which() if the program doesn't exist... It's really a micro optimization which has no impact on the correctness, for the specific case of Fedora.
About the correctness, FreeBSD has a different value:
>>> os.confstr("CS_PATH") '/usr/bin:/bin:/usr/sbin:/sbin'Not only it also includes /usr/sbin and /sbin, but /usr/bin has the preference over /bin (posixpath of Python 3 checks /bin before /usr/bin). I'm not sure if the different order has an impact about correctness. I only found two programs which are in /bin and /usr/bin, but the programs in /usr/bin are symbolic links to a program with the same name in /bin :-)
vstinner@freebsd$ file /usr/bin/pkill
/usr/bin/pkill: symbolic link to ../../bin/pkill
izbyshev commentedon Jan 17, 2019
Thanks for the info on CS_PATH, Victor. IMHO it'd make sense to use the libc-provided default PATH at least in shutil.which() since its intent is to emulate "which" from the default shell.
vstinner commentedon Apr 16, 2019
I wrote PR 12858 to os.confstr("CS_PATH") if available in shutil.which() and distutils.spawn.find_executable(), but also change the behavior when the PATH environment variable is set to an empty string: use an empty string, don't use os.confstr("CS_PATH") nor os.defpath.
7 remaining items
vstinner commentedon Apr 17, 2019
I don't understand this point. Your example is consistent with what I saw on my Fedora 29 and the Python implementation that I just merged. Would you mind to elaborate which corner case is handled differently and describe how?
vstinner commentedon Apr 17, 2019
Anyway, I wrote PR 12861 to remove "to mimick Unix which command behavior".
vstinner commentedon Apr 17, 2019
New changeset 197f044 by Victor Stinner in branch 'master':
bpo-35755: Don't say "to mimick Unix which command behavior" (GH-12861)
197f044
vstinner commentedon Apr 17, 2019
New changeset 394b991 by Victor Stinner in branch '3.7':
[3.7] bpo-35755: shutil.which() uses os.confstr("CS_PATH") (GH-12862)
394b991
[-]Remove current directory from posixpath.defpath to enhance security[/-][+]shutil.which() and subprocess no longer looks the executable in the current directory if PATH environment variable is not set[/+][-]shutil.which() and subprocess no longer looks the executable in the current directory if PATH environment variable is not set[/-][+]shutil.which() and subprocess no longer look for the executable in the current directory if PATH environment variable is not set[/+]vstinner commentedon Apr 17, 2019
Gregory:
In the meanwhile, I reverted the ntpath change. I'm not sure that it's ok to change the Windows case.
shutil.which() *always* starts by checking if the searched program is the current directory:
If someone cares about changing Windows, please open a separated issue.
vstinner commentedon Apr 17, 2019
I'm still confused by distutils.spawn.find_executable() function which *always* first look the current directory. I don't know the rationale for this behavior, so I made the conservation choice of keeping it.
If someone wants to change distutils.spawn.find_executable(), please open a separated issue.
vstinner commentedon Apr 17, 2019
I modified posixpath.defpath, shutil.which() and distutils.spawn.find_executable() in 3.7 and master (future Python 3.8) branches. I close the issue. Thanks everybody for the review and helping me to collect info about corner cases!
I chose to also change Python 3.7. IMHO there is a low risk of breaking applications: I expect that few users run Python with no PATH environment variable *and* expect that Python looks for programs in the current directory. But it enhances the security a little bit.
For Python 2.7... well, I don't think that this issue is important enough to justify a backport. I prefer to do nothing rather than having to deal with unhappy users complaining that Python 2.7 changed broke their application in a minor 2.7.x release :-) Even if, again, the risk of regression is very low.
[-]shutil.which() and subprocess no longer look for the executable in the current directory if PATH environment variable is not set[/-][+]On Unix, shutil.which() and subprocess no longer look for the executable in the current directory if PATH environment variable is not set[/+]jwilk commentedon Apr 17, 2019
When PATH is empty string:
I suspect that the former is implementation accident. I can't imagine why would anyone want this behavior.
NB, POSIX says that when PATH is unset or empty, the path search is implementation-defined.
vstinner commentedon Apr 17, 2019
Not thank you POSIX for being clueless. Let's say that Python is
opiniatied, as when we decided that file descriptors must be created
non-inheritable by default even if "it goes against POSIX".
If you want to look if the current directory, you now have to ask for it
kindly and explicitly ;-)
IHMO Using CS_PATH rather than hardcoded os.defpath is also a major step
forward ;-)
vstinner commentedon Apr 18, 2019
Same rationale for Python 3.6. While I would call this change related to security, I'm not comfortable to backport the change. The issue is not important enough compared to the risk of regression.