Repository navigation
Python modules not linking to libpython causes issues for RTLD_LOCAL system-wide #80934
Description
| BPO | 36753 |
|---|---|
| Nosy | @doko42, @vstinner, @ericvw, @pablogsal, @furiel, @MrAnno |
| Files |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = None
created_at = <Date 2019-04-29.16:16:00.782>
labels = ['3.8', 'type-bug', 'library']
title = 'Python modules not linking to libpython causes issues for RTLD_LOCAL system-wide'
updated_at = <Date 2020-04-06.21:57:11.332>
user = 'https://bugs.python.org/reimar'bugs.python.org fields:
activity = <Date 2020-04-06.21:57:11.332>
actor = 'Joshua Merchant'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2019-04-29.16:16:00.782>
creator = 'reimar'
dependencies = []
files = ['48291']
hgrepos = []
issue_num = 36753
keywords = []
message_count = 7.0
messages = ['341094', '341096', '341105', '349232', '349255', '349736', '349739']
nosy_count = 8.0
nosy_names = ['doko', 'vstinner', 'ericvw', 'pablogsal', 'reimar', 'furiel', 'Joshua Merchant', 'MrAnno']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue36753'
versions = ['Python 3.8']
Activity
reimar commentedon Apr 29, 2019
Most affected platforms: RedHat and Debian, but with the changes from bpo-21536 probably all Linux distributions will be affected.
bpo-34814 and bpo-21536 and https://bugzilla.redhat.com/show_bug.cgi?id=1585201 make statements along the lines of "In short, RTLD_LOCAL is not supported."
This might have been considered a reasonable stance because of the specific example opening libpython directly.
However Python modules not linking to libpython also breaks things when libpython is loaded in the most indirect ways via dlopen.
E.g. dlopen("libA.so", RTLD_LOCAL | RTLD_NOW)
libA might have linked against libB, libB against libC and libC might optionally link against libpython.
As a developer generally cannot really know if some library might ever pull in a most indirect reference to libpython, not supporting RTLD_LOCAL in Python essentially means RTLD_LOCAL can NEVER EVER be used safely.
A test-case that will fail the import command when modules have not been linked against libpython is attached (demonstrating only one layer of indirection, but much more complex cases are of course possible).
You will need to adjust the (include, lib) paths in test.sh for your Python version, it was written to demonstrate the issue against RedHat's modifications of Python 2.7 (to my knowledge, RedHat and Debian has been affected by this issue much longer than mainline Python).
While dlmopen is an alternative with similar behaviour to RTLD_LOCAL on recent Linux versions for this case, it is not portable.
doko42 commentedon Apr 29, 2019
why is this an issue on Debian? Debian is already not linking with libpython.
reimar commentedon Apr 29, 2019
The modules not linking against libpython CAUSES this issue, thus Debian being affected even for old Python versions before bpo-21536.
MrAnno commentedon Aug 8, 2019
I'd like to mention a real-world example for this issue: syslog-ng with its
plugin system.
Plugins are loaded using dlopen() after the initial plugin discovery. RTLD_LOCAL
would be a reasonable choice, since symbols in plugins should not pollute the
global namespace (so plugins using different versions of OpenSSL, Python, etc.
would be possible).
Since we have Python-based plugins, we had to use RTLD_GLOBAL instead (with RTLD_LAZY to
reduce the possibility of conflicts) as a workaround.
Note: The latest Python 3 version seems to be working with RTLD_LOCAL as everything
under lib-dynload/ linked against libpython.
pablogsal commentedon Aug 8, 2019
In the What's New section of 3.8 (https://docs.python.org/3.8/whatsnew/3.8.html#changes-in-the-c-api) it indicates that:
So if you are embedding python by dlopen'ing libpython.so you should use RTLD_GLOBAL to make sure everyone is using the same symbols. You have more information on the rationale and decisions in bpo-21536.
Is your use-case not cover by the arguments in bpo-21536? What is the problem of embedding by dlopening libpython using RTLD_GLOBAL?
reimar commentedon Aug 14, 2019
I'm not sure how I can explain it much better, I even wrote example code after all, but I'll try...
Neither me nor Laszlo are using/embedding or otherwise involving Python (directly/intentionally at least).
We just want to load some .so file.
That .so itself might then use libpython. Or use a library that uses libpython. Or uses a library that uses a library .... that uses libpython.
And how could we know whether SOME library down that dependency chain uses libpython or not?
The result is that now EVERY SINGLE LIBRARY IN THE WHOLE SYSTEM needs to be loaded with RTLD_GLOBAL.
Because a library 50 dependencies down that uses python might break otherwise and there's not really any way to know.
Just to try be very clear: We are not users of libpython, we do not write or use any python code ourselves (except through indirect dependencies) and we are still hit by this issue.
reimar commentedon Aug 14, 2019
I guess one way this could be "solved" would be by libpython doing a dlopen on itself with RTLD_GLOBAL on Python initialization.
I wouldn't bet that that wouldn't end up with some very interesting unintended consequences as well though.