Building with pip fails with missing headers #17

Activity

1. wbarnha commented on Sep 8, 2023

   

   wbarnha

   on Sep 8, 2023

   Member

   More actions

   You need to install and link the RocksDB header files, it's a major pain ☹️, which is why I created a workflow to build wheel files to get around this issue.

   Alternatively, you may find this SO answer of interest: https://stackoverflow.com/a/70501371/9852671
2. GodTamIt commented on Sep 9, 2023

   

   GodTamIt

   on Sep 9, 2023

   Author

   More actions

   @wbarnha I tried building RocksDB as a shared library, but it didn't seem to like that either.

   I'm doing the following in a Dockerfile:

   RUN apt-get install -y --no-install-recommends libsnappy-dev zlib1g-dev libbz2-dev libgflags-dev
   RUN git clone https://github.com/facebook/rocksdb --depth 1 --branch "v6.29.5" rocksdb
   RUN cd rocksdb && make install-shared -j 32
   RUN ls /usr/local/lib && test -f /usr/local/lib/librocksdb.so.6.29
   
   # Remove temporary dir
   RUN rm -rf rocksdb
   
   RUN pip install git+https://github.com/faust-streaming/python-rocksdb.git@9b81bed3ffa55a1662811e9ad9fea652f42047f6
   
   RUN python -c 'import rocksdb'

   I'm getting:

   ImportError: librocksdb.so.6.29: cannot open shared object file: No such file or directory
   
3. wbarnha commented on Sep 10, 2023

   

   wbarnha

   on Sep 10, 2023

   Last edited by wbarnha

   Member

   More actions

   Out of curiosity, what's the motivation for building this library from source? Are you trying to support another OS or architecture? I'm considering marking this project as deprecated since rocksdict has taken over.
4. GodTamIt commented on Sep 10, 2023

   

   GodTamIt

   on Sep 10, 2023

   Author

   More actions

   I'm trying to build it for ARM64. I want to move to rocksdict, but there are a few legacy instances that rocksdict isn't binary compatible with, I don't think.

   👍React with 👍1Reacted by William Barnhart
5. wbarnha commented on Sep 10, 2023

   

   wbarnha

   on Sep 10, 2023

   Last edited by wbarnha

   Member

   More actions

   Got it. The easiest solution to be honest would to be extending the CI/CD pipeline for more architectures. The complexity of this library's build process is horrific and I apologize.
6. wbarnha commented on Sep 15, 2023

   

   wbarnha

   on Sep 15, 2023

   Last edited by wbarnha

   Member

   More actions

   Got your ARM wheels built in #18. Holy crap, I can't believe QEMU slowed it down to take 5h8m.

   To be honest I'm contemplating this being my final release and then archiving the project. This has been an unwieldy library for some time and maintaining it is incredibly difficult. I've tried reviewing the internals of RocksDB to see if I can improve build times or reduce wheel sizes, but there's not much I can currently achieve with my skillset at the moment. Plus not many other people have actively stepped up to help me maintain this project, and I think I've been able to move more attention to rocksdict, which in my opinion, is substantially easier to maintain. Unless other people volunteer to support this project, I will be likely archiving it due to lack of additional support and outdated functionality.
7. wbarnha

   linked a pull request that will close this issueAdd QEMU setup to dist.yml to create arm64 wheels #18on Sep 15, 2023
8. wbarnha

   closed this as completedin #18on Sep 15, 2023