OpenSSL provides required ssl module APIs... no #57

Activity

1. solvingj commented on Feb 3, 2025

   

   solvingj

   on Feb 3, 2025

   Last edited by solvingj

   Author

   More actions

   I tried downgrading openssl to 1.1.1w, and then 1.1.1u and get the same issue.
   Also tried Python 3.11.10 and same issue.
2. solvingj commented on Feb 3, 2025

   

   solvingj

   on Feb 3, 2025

   Last edited by solvingj

   Author

   More actions

   Just as a sanity check, i removed:
   cpython-modules: openssl

   And now it reports yes to these checks:

   checking whether compiling and linking against OpenSSL works... yes
   checking for --with-openssl-rpath... 
   checking whether OpenSSL provides required ssl module APIs... yes
   checking whether OpenSSL provides required hashlib module APIs... yes
   

   So then I restored it and switched to 1.1.1n which is the same version the system has and they still report no.

   The .pc files all look right so I'm running out of ideas.
3. solvingj commented on Feb 3, 2025

   

   solvingj

   on Feb 3, 2025

   Author

   More actions

   @zsimic am I misunderstanding something fundamental?

   My expectation is that:

   The net effect of this statement is that OpenSSL will be compiled from source and statically linked in:

   cpython-modules: openssl
   

   configure should report yes in these lines when doing so, and that it's a problem that it reports no:

   checking whether OpenSSL provides required ssl module APIs... no
   checking whether OpenSSL provides required hashlib module APIs... no
   

   Because it seems to me that everything is happening properly.

   OpenSSL's build is installing the static libs and the .pc files:

   install libcrypto.a -> /home/root/build/deps/lib/libcrypto.a
   install libssl.a -> /home/root/build/deps/lib/libssl.a
   created directory `/home/root/build/deps/lib/pkgconfig'
   install libcrypto.pc -> /home/root/build/deps/lib/pkgconfig/libcrypto.pc
   install libssl.pc -> /home/root/build/deps/lib/pkgconfig/libssl.pc
   install openssl.pc -> /home/root/build/deps/lib/pkgconfig/openssl.pc
   

   And we know Python's configure script is using the .pc files because it finds the headers successfully:

   checking for include/openssl/ssl.h in /home/root/build/deps... yes
   

   And it seems to me the problem is that it's not finding the symbols inside:

   libcrypto.a
   libssl.a
   

   I feel I must be missing something because otherwise static linking like this should never have worked properly for anyone.
4. solvingj commented on Feb 3, 2025

   

   solvingj

   on Feb 3, 2025

   Author

   More actions

   Ok bit of a surprise here. I found the following in components/cpython/config.log indicating it's not even the openssl symbols it's failing to find, but the dl symbols:

   configure:28246: checking whether OpenSSL provides required ssl module APIs
   configure:28279: gcc -pthread -o conftest  -I/home/root/build/deps/include   -L/home/root/build/deps/lib  conftest.c -lpthread -ldl  -lutil -lssl -lcrypto >&5
   /home/root/build/deps/lib/libcrypto.a(dso_dlfcn.o): In function `dlfcn_globallookup':
   dso_dlfcn.c:(.text+0x18): undefined reference to `dlopen'
   dso_dlfcn.c:(.text+0x2c): undefined reference to `dlsym'
   dso_dlfcn.c:(.text+0x38): undefined reference to `dlclose'
   /home/root/build/deps/lib/libcrypto.a(dso_dlfcn.o): In function `dlfcn_bind_func':
   dso_dlfcn.c:(.text+0x224): undefined reference to `dlsym'
   dso_dlfcn.c:(.text+0x2c0): undefined reference to `dlerror'
   /home/root/build/deps/lib/libcrypto.a(dso_dlfcn.o): In function `dlfcn_load':
   dso_dlfcn.c:(.text+0x340): undefined reference to `dlopen'
   dso_dlfcn.c:(.text+0x3bc): undefined reference to `dlclose'
   dso_dlfcn.c:(.text+0x400): undefined reference to `dlerror'
   /home/root/build/deps/lib/libcrypto.a(dso_dlfcn.o): In function `dlfcn_pathbyaddr':
   dso_dlfcn.c:(.text+0x4e0): undefined reference to `dladdr'
   dso_dlfcn.c:(.text+0x558): undefined reference to `dlerror'
   /home/root/build/deps/lib/libcrypto.a(dso_dlfcn.o): In function `dlfcn_unload':
   dso_dlfcn.c:(.text+0x758): undefined reference to `dlclose'
   collect2: error: ld returned 1 exit status
   configure:28279: $? = 1
   configure: failed program was:
   

   It's got -ldl which is the common reason for these errors, so will see what else i can find.
5. solvingj commented on Feb 3, 2025

   

   solvingj

   on Feb 3, 2025

   Author

   More actions

   Ok, I traced this down and have some feedback.

   This does appear to be a problem specific to openssl and static linking, which will be platform specific. So far i've only tested on linux arm (aarch64), later today i'll test linux x86.

   The fundamental thing I had wrong was that it was actually missing openssl symbols, specifically the ones which are loaded dynamically, as part of the DSO/Engines system.

   This seems to be a known potential problem with static linking openssl into python based on this comment in the Python configure.ac file:

   https://github.com/python/cpython/blob/main/configure.ac#L7314-L7333

   I was able to workaround this issue by disabling openssl's dso feature, by patching the code to adding no-dso here:

   https://github.com/codrsquad/portable-python/blob/main/src/portable_python/external/xcpython.py#L161

   yield "no-shared", "no-idea", "no-tests", "no-dso"
   

   I don't believe additional configure flags for deps is exposed to the config the way the version is, but that would be great. Would that be supportable in the future?
6. zsimic

   closed this as completedin #59on Feb 14, 2025
7. zsimic commented on Feb 14, 2025

   

   zsimic

   on Feb 14, 2025

   Contributor

   More actions

   Release with portable-python v1.9.8