Repository navigation
Shared modules built with Modules/Setup are not found when run from build directory #61297
Closed as not planned
Description
| BPO | 17095 |
|---|---|
| Nosy | @loewis, @Yhg1s, @vstinner, @larryhastings, @ned-deily, @matrixise |
| Files |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = 'https://github.com/ned-deily'
closed_at = None
created_at = <Date 2013-01-31.21:54:56.549>
labels = ['build']
title = 'Shared modules built with Modules/Setup are not found when run from build directory'
updated_at = <Date 2014-10-18.09:09:12.698>
user = 'https://github.com/Yhg1s'bugs.python.org fields:
activity = <Date 2014-10-18.09:09:12.698>
actor = 'matrixise'
assignee = 'ned.deily'
closed = False
closed_date = None
closer = None
components = ['Build']
creation = <Date 2013-01-31.21:54:56.549>
creator = 'twouters'
dependencies = []
files = ['28928', '35344']
hgrepos = []
issue_num = 17095
keywords = ['patch']
message_count = 12.0
messages = ['181048', '219070', '219554', '219555', '225978', '226548', '226894', '229421', '229490', '229637', '229638', '229639']
nosy_count = 9.0
nosy_names = ['loewis', 'twouters', 'vstinner', 'larry', 'ned.deily', 'Arfrever', 'python-dev', 'matrixise', 'John.Malmberg']
pr_nums = []
priority = 'low'
resolution = None
stage = 'needs patch'
status = 'open'
superseder = None
type = None
url = 'https://bugs.python.org/issue17095'
versions = ['Python 3.4', 'Python 3.5']
Activity
Yhg1s commentedon Jan 31, 2013
At some point (probably in 3.2) the support for shared modules built using Modules/Setup was broken, for two reasons:
Python no longer considers 'foomodule.so' when looking for a module called 'foo', but Modules/makesetup still appends 'module.so' for a 'foo' module.
Python no longer considers Modules/ to be the exec_prefix (and instead uses what was loaded from the build-info .txt written by setup.py) but the build process still leaves these Modules/Setup-built shared modules in the Modules directory (and it doesn't know about the directory setup.py will make, so it can't really do anything else.)
This patch fixes both problems, by making Modules/makesetup name shared library modules 'foo.so' and by adding the Modules subdirectory to sys.path (after the setup.py-provided directory) when running from the build directory. (The existing build process has no problem _installing_ the Modules/Setup-built extension modules, it's just Python that doesn't know how to find them.)
ned-deily commentedon May 25, 2014
Your original patch LGTM. However, since it was written, Setup *shared* support became even more broken due to changes in Makefile macros. Attached is an updated patch. I've never tried customizing Setup before so I don't have a lot of confidence that it now works as expected in all cases, but I tried a few cases (like building _ssl with a non-standard OpenSSL on OS X) and they did work.
python-dev commentedon Jun 2, 2014
New changeset 6c468df214dc by Ned Deily in branch '3.4':
Issue bpo-17095: Fix Modules/Setup *shared* support.
http://hg.python.org/cpython/rev/6c468df214dc
New changeset 227ce85bdbe0 by Ned Deily in branch 'default':
Issue bpo-17095: Fix Modules/Setup *shared* support.
http://hg.python.org/cpython/rev/227ce85bdbe0
ned-deily commentedon Jun 2, 2014
Committed for release in 3.4.2 and 3.5.0.
vstinner commentedon Aug 27, 2014
I reopen the issue because these changes broke the decimal module when the _decimal module is missing: see the issue bpo-22285.
Martin von Loewis proposed to build modules in the build directory, not in the Modules directory directly:
http://bugs.python.org/issue22285#msg225961
ned-deily commentedon Sep 8, 2014
Clearly this breaks running from the build directory for various modules when their optional C extension module build fails. I'll either change the Setup shared builds to use a build directory or revert the path changes prior to the upcoming 3.4.2.
python-dev commentedon Sep 15, 2014
New changeset 781454f792c4 by Ned Deily in branch '3.4':
Issue bpo-17095: Temporarily revert getpath.c change that added the Modules
https://hg.python.org/cpython/rev/781454f792c4
New changeset d3939f602e1f by Ned Deily in branch 'default':
Issue bpo-17095: merge from 3.4
https://hg.python.org/cpython/rev/d3939f602e1f
matrixise commentedon Oct 15, 2014
Hi Ned,
What are the news for this issue?
ned-deily commentedon Oct 15, 2014
Stéphane, as I understand it, this issue covers two problems:
(1) Building and installing shared modules using Modules/Setup was broken for current Python 3 releases;
(2) Using those shared modules when running Python from a build directory (rather than from a normal installed Python) is broken.
Problem (1) is now fixed. Problem (2) remains to be properly fixed; it's a little tedious to get right as a fix needs to take into account the SO_ABI and ensure built modules get properly cleaned and all the right Makefile dependencies work. I'd be happy to review a patch if someone gets to it before I do. But it seems to me that (1) was the serious problem; (2) is more of a minor annoyance for developers with easy workarounds, e.g. use PYTHONPATH or install Python to a temporary location for testing.
matrixise commentedon Oct 18, 2014
Hi Ned,
For the first point, maybe close this issue and open a new bug for the second.
What do you think?
ned-deily commentedon Oct 18, 2014
Sorry, I don't see the point of opening another issue, especially since most of the discussion in this issue has been about the second problem. I have updated the issue title to be more specific, though.
[-]Modules/Setup *shared* support broken[/-][+]Shared modules built with Modules/Setup are not found when run from build directory[/+]matrixise commentedon Oct 18, 2014
Thanks Ned
ned-deily commentedon Oct 2, 2024
There has been a lot of work in the Python initialization area including Setup since this issue was opened. And there hasn't been much interest in it including from me. Let's just close it.