__code__. co_filename should always be an absolute path

@gpshead

BPO	20443
Nosy	@gpshead, @ncoghlan, @vstinner, @bitdancer, @ambv, @ammaraskar, @tirkarthi, @isidentical
PRs	bpo-20443: make code objects filename an absolute path #13527 bpo-20443: _PyConfig_Read() gets the absolute path of run_filename #14053 bpo-20443: Fix calculate_program_full_path() warning #14446 bpo-20443: No longer make sys.argv[0] absolute for script #17534 bpo-20443: Update What's New In Python 3.9 #17986

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2019-12-31.04:56:03.820>
created_at = <Date 2014-01-30.02:15:15.725>
labels = ['type-bug', '3.9']
title = '__code__. co_filename should always be an absolute path'
updated_at = <Date 2020-01-13.13:57:20.518>
user = 'https://github.com/1st1'

bugs.python.org fields:

activity = <Date 2020-01-13.13:57:20.518>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2019-12-31.04:56:03.820>
closer = 'ncoghlan'
components = []
creation = <Date 2014-01-30.02:15:15.725>
creator = 'yselivanov'
dependencies = []
files = []
hgrepos = []
issue_num = 20443
keywords = ['patch']
message_count = 32.0
messages = ['209699', '345501', '345521', '345585', '346207', '346262', '346524', '346789', '346800', '346823', '354925', '354999', '355055', '355085', '355088', '355104', '355105', '355106', '355218', '355219', '355278', '355334', '355335', '355339', '355340', '358094', '358114', '358116', '358128', '358661', '359098', '359902']
nosy_count = 9.0
nosy_names = ['gregory.p.smith', 'ncoghlan', 'vstinner', 'r.david.murray', 'lukasz.langa', 'Michel Desmoulin', 'ammar2', 'xtreak', 'BTaskaya']
pr_nums = ['13527', '14053', '14446', '17534', '17986']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue20443'
versions = ['Python 3.9']

There are many issues on tracker related to the relative paths in co_filename. Most of them are about introspection functions in inspect module--which are usually broken after 'os.chdir'.

Test case: create a file t.py:

   def foo(): pass
   print(foo.__code__.co_filename)

Execute it with '$ python t.py' -> it will print 't.py'.

Ideally, when executing a python file, interpreter should expand all relative paths for __file__ and __code__.co_filename attributes.

PR 14053 is a different approach than PR 13527: compute the absolute path to the script filename in PyConfig_Read() just after parsing the command line.

One of the side effect of my PR 14053 is that tracebacks are more verbose: the filename is now absolute rather than relative.

Currently:

$ python3 x.py 
Traceback (most recent call last):
  File "x.py", line 4, in <module>
    func()
  File "x.py", line 2, in func
    bug
NameError: name 'bug' is not defined

With my PR:

$ ./python x.py 
Traceback (most recent call last):
  File "/home/vstinner/prog/python/master/x.py", line 4, in <module>
    func()
  File "/home/vstinner/prog/python/master/x.py", line 2, in func
    bug
NameError: name 'bug' is not defined

The site module tries to compute the absolute path of __file__ and __cached__ attributes of all modules in sys.modules:

def abs_paths():
    """Set all module __file__ and __cached__ attributes to an absolute path"""
    for m in set(sys.modules.values()):
        if (getattr(getattr(m, '__loader__', None), '__module__', None) not in
                ('_frozen_importlib', '_frozen_importlib_external')):
            continue   # don't mess with a PEP 302-supplied __file__
        try:
            m.__file__ = os.path.abspath(m.__file__)
        except (AttributeError, OSError, TypeError):
            pass
        try:
            m.__cached__ = os.path.abspath(m.__cached__)
        except (AttributeError, OSError, TypeError):
            pass

The __path__ attribute isn't updated.

Another approach would be to hack importlib to compute the absolute path before loading a module, rather than trying to fix it *afterwards*.

One pratical problem: posixpath and ntpath are not available when importlib is setup, these modules are implemented in pure Python and so must be imported.

Maybe importlib could use a naive implementation of os.path.abspath(). Maybe the C function _Py_abspath() that I implemented in PR 14053 should be exposed somehow to importlib as a private function using a builtin module like _imp, so it can be used directly.

Perhaps os._abspath_for_import? If importlib finds it, then it can handle making early paths absolute itself, otherwise it will defer doing that until it has the full external import system bootstrapped. (importlib does try to make all paths absolute as it loads modules - it's just that some early modules aren't loaded that way)

Effect of my PR 14053 using script.py:

import sys
print(f"{__file__=}")
print(f"{sys.argv=}")
print(f"{sys.path[0]=}")

Before:

$ ./python script.py 
__file__=script.py
sys.argv=['script.py']
sys.path[0]=/home/vstinner/prog/python/master

With the change:

$ ./python script.py 
__file__=/home/vstinner/prog/python/master/script.py
sys.argv=['/home/vstinner/prog/python/master/script.py']
sys.path[0]=/home/vstinner/prog/python/master

New changeset 3939c32 by Victor Stinner in branch 'master':
bpo-20443: _PyConfig_Read() gets the absolute path of run_filename (GH-14053)
3939c32

Example of case where a module path is still relative:
---

import sys
import os
modname = 'relpath'
filename = modname + '.py'
sys.path.insert(0, os.curdir)
with open(filename, "w") as fp:
    print("import sys", file=fp)
    print("mod = sys.modules[__name__]", file=fp)
    print("print(f'{__file__=}')", file=fp)
    print("print(f'{mod.__file__=}')", file=fp)
    print("print(f'{mod.__cached__=}')", file=fp)
__import__(modname)
os.unlink(filename)

Output:
---
__file__='./relpath.py'
mod.__file__='./relpath.py'
mod.__cached__='./pycache/relpath.cpython-39.pyc'
---

__file__ and mod.__file__ are relative paths: not absolute paths.

This change seems to produce a new warning on my Mac.

./Modules/getpath.c:764:23: warning: incompatible pointer types passing 'char [1025]' to parameter of type 'const wchar_t *' (aka 'const int *') [-Wincompatible-pointer-types]
            _Py_isabs(execpath))
                      ^~~~~~~~
./Include/fileutils.h:158:42: note: passing argument to parameter 'path' here
PyAPI_FUNC(int) _Py_isabs(const wchar_t *path);
                                         ^
1 warning generated.

New changeset 3029035 by Victor Stinner in branch 'master':
bpo-20443: Fix calculate_program_full_path() warning (GH-14446)
3029035

Isn't that change breaking compat ?

I'm assuming many scripts in the wild sys.argv[0] and play with it assuming it's relative.

I would expect such a change to be behind a flag or a __future__ import.

I think that's a valid point regarding sys.argv[0] - it's the import system and code introspection that wants(/needs) absolute paths, whereas sys.argv[0] gets used in situations (e.g. usage messages) where we should retain whatever the OS gave us, since that best reflects what the user entered.

That's straightforward to selectively revert, though: remove the "config->run_filename != NULL" clause at

cpython/Python/initconfig.c

Line 2201 in 24dc2f8

else if (config->run_filename != NULL) {

and add a comment with the reason for the deliberate omission.

That way the OS-provided argv entry will continue to be passed through to sys.argv[0], while everywhere else will get the absolute path.

What is the use case for having a relative sys.argv[0]?

Isn't that change breaking compat ?

Right. It has been made on purpose. The rationale can be found in the first message of this issue.

To help address the original report filing issue, we could add a notion of .co_cwd to code objects for use in resolving co_filename. Allow it to be '' if getcwd failed at source load time. Code could check if co_cwd exists and join it with the co_filename. Also let co_cwd remain empty when there is no valid co_filename for the code.

Do you mean that co_filename attribute of code objects must remain relative?

Preaching: Code that calls os.chdir() is always a potential problem as it alters process global state. That call is best avoided as modifying globals is impolite.

I thought that calling os.chdir("/") is a good practice when launching a program as a daemon. And thne call os.fork() twice to detach the process from its parent and run it in background.

I commonly use os.chdir() in my programs for various reasons. A classical use case is to mimick a shell script doing something like (cd $DIRECTORY; command): temporary change the current directory.

cwd parameter of subprocess.Popen helps to avoid changing the currently directory, but sometimes subprocess is not used at all.

Maybe avoiding os.chdir() is a good practice, but it's hard to prevent users to call it. I don't see how os.chdir() is bad by design. I'm not sure that it's really useful to debate this neither :-)

I think sys.argv[0] is the important one as program logic often tends to use that as an input.

I'm honestly unsure of if this will be as much of a problem for .co_filename or sys.path[0].

(read that as: feel free to keep the behavior for co_filename and path[0] and lets see what happens :)

Anything new here? 3.9.0a2 is due next week.

Anything new here? 3.9.0a2 is due next week.

I had it in my TODO list but I lost the bpo number. Thanks for the reminder :-) I wrote PR 17534 that I plan to merge as soon as the CI tests pass.

New changeset a1a99b4 by Victor Stinner in branch 'master':
bpo-20443: No longer make sys.argv[0] absolute for script (GH-17534)
a1a99b4

I am not sure that if co_filename can break backwards compability or not (because until CodeType.replace; it was usual to break code object construction, people who are using code objects may take this too) but yes PR 17534 was necessary.

My initial patch (PR 13527) proposed a future directive to do this (and poorly implemented the concept) which can be helpful (yet it needs a PEP).

I reverted my change, so I remove "release blocker" priority.

With the sys.argv[0] change reverted, I think this overall issue is fixed now - code objects will get absolute paths, while sys.argv[0] will continue to reflect how __main__ was identified.

New changeset c1ee6e5 by Victor Stinner in branch 'master':
bpo-20443: Update What's New In Python 3.9 (GH-17986)
c1ee6e5

This says that Module __file__ attributes (and related values) should now always contain absolute paths by default, with the sole exception of __main__.__file__, and this says that the __file__ attribute of the __main__ module became an absolute path; does this mean that __file__ is always an absolute path?

Is there anything in the official Python documentation that specifies whether __file__ is absolute or relative?

This issue is closed. I suggest you to open a new issue.

Navigation Menu

Repository navigation

code. co_filename should always be an absolute path #64642

10 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

__code__. co_filename should always be an absolute path #64642

Description

Activity

1st1 commented on Jan 30, 2014

vstinner commented on Jun 13, 2019

vstinner commented on Jun 13, 2019

vstinner commented on Jun 14, 2019

ncoghlan commented on Jun 21, 2019

vstinner commented on Jun 21, 2019

vstinner commented on Jun 25, 2019

vstinner commented on Jun 28, 2019

tirkarthi commented on Jun 28, 2019

vstinner commented on Jun 28, 2019

MichelDesmoulin commented on Oct 18, 2019

ncoghlan commented on Oct 20, 2019

vstinner commented on Oct 21, 2019

10 remaining items

vstinner commented on Oct 24, 2019

gpshead commented on Oct 24, 2019

gpshead commented on Oct 24, 2019

ambv commented on Dec 9, 2019

vstinner commented on Dec 9, 2019

vstinner commented on Dec 9, 2019

isidentical commented on Dec 9, 2019

vstinner commented on Dec 19, 2019

ncoghlan commented on Dec 31, 2019

vstinner commented on Jan 13, 2020

bvd0 commented on Sep 6, 2023

vstinner commented on Sep 6, 2023

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions

code. co_filename should always be an absolute path #64642