Hadoop on UBI10

## Description

Part of https://github.com/stackabletech/docker-images/issues/1349
Related to [building boost from source](https://github.com/stackabletech/docker-images/pull/1475) which is now parked.

The native code (written in C) part of `hadoop-common` doesn't build on UBI10. See the build error message at the bottom.

The reason is that the OpenSSL library shipped with UBI10 has removed ENGINE support as described [here](https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/10/html-single/considerations_in_adopting_rhel_10/index)

> The support for OpenSSL ENGINE has been removed. You must no longer use the SSLCryptoDevice configuration directive.

It seems that other Linux distros also build newer OpenSSL 3 versions with the `no-engine` configuration option.

## Upstream issue

Created [this issue](https://issues.apache.org/jira/browse/HADOOP-19872) upstream with the Apache developers.

Issue has been reopened. ~Unfortunately, it was wrongly marked as duplicate and closed.~

## Investigation

Summary: the native part of the hadoop-common Maven project cannot be built without moving the implementation away from the engine usage.

These are my findings during the investigation of this issue, in no particular order.

1. Haddop 3.4.3 and 3.5.0 also fail to compile on UBI10 as they have the same problem.
2. On UBI 10, the `openssl/engine.h` header exists __but is empty__. On Fedora 43, it's missing completely.
3. Updating the `wildfly` dependency version as done in [HADOOP-19719](https://issues.apache.org/jira/browse/HADOOP-19719) __does not__ fix the problem in any way.
4. There is no way to pass command line arguments to the compiler via the Maven build.
5. Search for the package [pkcs11-provider](https://github.com/openssl-projects/pkcs11-provider) which might cover some of the engine functionality. There is no such package (provider) in EPEL10. There is a `pkcs11-helper` package that provides a different engine implementation and is not a drop in replacement of the `openssl/engine.h` from UBI9.
6. Unccessfully tried to build with additional macros as described in [here](https://openssl-library.org/post/2025-12-18-remove-engines/) because `OPENSSL_ENGINE_STUBS` doesn't exist in the UBI10 openssl headers. Also it looks like it would only fix the compilation but may still break at runtime.
7. Hadoop builds successfully if the `-Pnative` profile is removed from the Maven build command.

## Build error message

```
727.1 [WARNING] make[2]: Leaving directory '/build/src/hadoop/hadoop/patchable-work/worktree/3.4.2/hadoop-common-project/hadoop-common/target/native'
727.1 [WARNING] make[2]: Leaving directory '/build/src/hadoop/hadoop/patchable-work/worktree/3.4.2/hadoop-common-project/hadoop-common/target/native'
727.1 [WARNING] make[1]: Leaving directory '/build/src/hadoop/hadoop/patchable-work/worktree/3.4.2/hadoop-common-project/hadoop-common/target/native'
727.1 [WARNING] /build/src/hadoop/hadoop/patchable-work/worktree/3.4.2/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/OpensslCipher.c:184:13: warning: 'loadSm4Ctr' defined but not used [-Wunused-function]
727.1 [WARNING]   184 | static void loadSm4Ctr(JNIEnv *env)
727.1 [WARNING]       |             ^~~~~~~~~~
727.1 [WARNING] /build/src/hadoop/hadoop/patchable-work/worktree/3.4.2/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/OpensslCipher.c:184:13: warning: 'loadSm4Ctr' defined but not used [-Wunused-function]
727.1 [WARNING]   184 | static void loadSm4Ctr(JNIEnv *env)
727.1 [WARNING]       |             ^~~~~~~~~~
727.1 [WARNING] /build/src/hadoop/hadoop/patchable-work/worktree/3.4.2/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/random/OpensslSecureRandom.c: In function 'locks_setup':
727.1 [WARNING] /build/src/hadoop/hadoop/patchable-work/worktree/3.4.2/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/random/OpensslSecureRandom.c:250:33: warning: implicit declaration of function 'dlsym_CRYPTO_num_locks'; did you mean 'dlsym_CRYPTO_malloc'? [-Wimplicit-function-declaration]
727.1 [WARNING]   250 |   lock_cs = dlsym_CRYPTO_malloc(dlsym_CRYPTO_num_locks() *  \
727.1 [WARNING]       |                                 ^~~~~~~~~~~~~~~~~~~~~~
727.1 [WARNING]       |                                 dlsym_CRYPTO_malloc
727.1 [WARNING] /build/src/hadoop/hadoop/patchable-work/worktree/3.4.2/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/random/OpensslSecureRandom.c:257:3: warning: implicit declaration of function 'dlsym_CRYPTO_set_id_callback'; did you mean 'CRYPTO_set_id_callback'? [-Wimplicit-function-declaration]
727.1 [WARNING]   257 |   dlsym_CRYPTO_set_id_callback((unsigned long (*)())pthreads_thread_id);
727.1 [WARNING]       |   ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
727.1 [WARNING]       |   CRYPTO_set_id_callback
727.1 [WARNING] /build/src/hadoop/hadoop/patchable-work/worktree/3.4.2/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/random/OpensslSecureRandom.c:258:3: warning: implicit declaration of function 'dlsym_CRYPTO_set_locking_callback'; did you mean 'CRYPTO_set_locking_callback'? [-Wimplicit-function-declaration]
727.1 [WARNING]   258 |   dlsym_CRYPTO_set_locking_callback((void (*)())pthreads_locking_callback);
727.1 [WARNING]       |   ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
727.1 [WARNING]       |   CRYPTO_set_locking_callback
727.1 [WARNING] /build/src/hadoop/hadoop/patchable-work/worktree/3.4.2/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/random/OpensslSecureRandom.c: In function 'openssl_rand_init':
727.1 [WARNING] /build/src/hadoop/hadoop/patchable-work/worktree/3.4.2/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/random/OpensslSecureRandom.c:329:40: error: 'ENGINE_METHOD_RAND' undeclared (first use in this function)
727.1 [WARNING]   329 |     rc = dlsym_ENGINE_set_default(eng, ENGINE_METHOD_RAND);
727.1 [WARNING]       |                                        ^~~~~~~~~~~~~~~~~~
727.1 [WARNING] /build/src/hadoop/hadoop/patchable-work/worktree/3.4.2/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/random/OpensslSecureRandom.c:329:40: note: each undeclared identifier is reported only once for each function it appears in
727.1 [WARNING] /build/src/hadoop/hadoop/patchable-work/worktree/3.4.2/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/random/OpensslSecureRandom.c: At top level:
727.1 [WARNING] /build/src/hadoop/hadoop/patchable-work/worktree/3.4.2/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/random/OpensslSecureRandom.c:261:13: warning: 'locks_cleanup' defined but not used [-Wunused-function]
727.1 [WARNING]   261 | static void locks_cleanup(void)
727.1 [WARNING]       |             ^~~~~~~~~~~~~
727.1 [WARNING] /build/src/hadoop/hadoop/patchable-work/worktree/3.4.2/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/random/OpensslSecureRandom.c:247:13: warning: 'locks_setup' defined but not used [-Wunused-function]
727.1 [WARNING]   247 | static void locks_setup(void)
727.1 [WARNING]       |             ^~~~~~~~~~~
727.1 [WARNING] make[2]: *** [CMakeFiles/hadoop_static.dir/build.make:104: CMakeFiles/hadoop_static.dir/main/native/src/org/apache/hadoop/crypto/random/OpensslSecureRandom.c.o] Error 1
727.1 [WARNING] make[2]: *** Waiting for unfinished jobs....
727.1 [WARNING] /build/src/hadoop/hadoop/patchable-work/worktree/3.4.2/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/random/OpensslSecureRandom.c: In function 'locks_setup':
727.1 [WARNING] /build/src/hadoop/hadoop/patchable-work/worktree/3.4.2/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/random/OpensslSecureRandom.c:250:33: warning: implicit declaration of function 'dlsym_CRYPTO_num_locks'; did you mean 'dlsym_CRYPTO_malloc'? [-Wimplicit-function-declaration]
727.1 [WARNING]   250 |   lock_cs = dlsym_CRYPTO_malloc(dlsym_CRYPTO_num_locks() *  \
727.1 [WARNING]       |                                 ^~~~~~~~~~~~~~~~~~~~~~
727.1 [WARNING]       |                                 dlsym_CRYPTO_malloc
727.1 [WARNING] /build/src/hadoop/hadoop/patchable-work/worktree/3.4.2/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/random/OpensslSecureRandom.c:257:3: warning: implicit declaration of function 'dlsym_CRYPTO_set_id_callback'; did you mean 'CRYPTO_set_id_callback'? [-Wimplicit-function-declaration]
727.1 [WARNING]   257 |   dlsym_CRYPTO_set_id_callback((unsigned long (*)())pthreads_thread_id);
727.1 [WARNING]       |   ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
727.1 [WARNING]       |   CRYPTO_set_id_callback
727.1 [WARNING] /build/src/hadoop/hadoop/patchable-work/worktree/3.4.2/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/random/OpensslSecureRandom.c:258:3: warning: implicit declaration of function 'dlsym_CRYPTO_set_locking_callback'; did you mean 'CRYPTO_set_locking_callback'? [-Wimplicit-function-declaration]
727.1 [WARNING]   258 |   dlsym_CRYPTO_set_locking_callback((void (*)())pthreads_locking_callback);
727.1 [WARNING]       |   ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
727.1 [WARNING]       |   CRYPTO_set_locking_callback
727.1 [WARNING] /build/src/hadoop/hadoop/patchable-work/worktree/3.4.2/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/random/OpensslSecureRandom.c: In function 'openssl_rand_init':
727.1 [WARNING] /build/src/hadoop/hadoop/patchable-work/worktree/3.4.2/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/random/OpensslSecureRandom.c:329:40: error: 'ENGINE_METHOD_RAND' undeclared (first use in this function)
727.1 [WARNING]   329 |     rc = dlsym_ENGINE_set_default(eng, ENGINE_METHOD_RAND);
727.1 [WARNING]       |                                        ^~~~~~~~~~~~~~~~~~
727.1 [WARNING] /build/src/hadoop/hadoop/patchable-work/worktree/3.4.2/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/random/OpensslSecureRandom.c:329:40: note: each undeclared identifier is reported only once for each function it appears in
727.1 [WARNING] /build/src/hadoop/hadoop/patchable-work/worktree/3.4.2/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/random/OpensslSecureRandom.c: At top level:
727.1 [WARNING] /build/src/hadoop/hadoop/patchable-work/worktree/3.4.2/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/random/OpensslSecureRandom.c:261:13: warning: 'locks_cleanup' defined but not used [-Wunused-function]
727.1 [WARNING]   261 | static void locks_cleanup(void)
727.1 [WARNING]       |             ^~~~~~~~~~~~~
727.1 [WARNING] /build/src/hadoop/hadoop/patchable-work/worktree/3.4.2/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/random/OpensslSecureRandom.c:247:13: warning: 'locks_setup' defined but not used [-Wunused-function]
727.1 [WARNING]   247 | static void locks_setup(void)
727.1 [WARNING]       |             ^~~~~~~~~~~
727.1 [WARNING] make[2]: *** [CMakeFiles/hadoop.dir/build.make:104: CMakeFiles/hadoop.dir/main/native/src/org/apache/hadoop/crypto/random/OpensslSecureRandom.c.o] Error 1
727.1 [WARNING] make[2]: *** Waiting for unfinished jobs....
727.1 [WARNING] make[1]: *** [CMakeFiles/Makefile2:113: CMakeFiles/hadoop_static.dir/all] Error 2
727.1 [WARNING] make[1]: *** Waiting for unfinished jobs....
727.1 [WARNING] make[1]: *** [CMakeFiles/Makefile2:87: CMakeFiles/hadoop.dir/all] Error 2
727.1 [WARNING] make: *** [Makefile:91: all] Error 2
727.1 [INFO] ------------------------------------------------------------------------
727.1 [INFO] Reactor Summary for Apache Hadoop Main 3.4.2-stackable0.0.0-dev:
727.1 [INFO]
727.1 [INFO] Apache Hadoop Main ................................. SUCCESS [ 44.354 s]
727.1 [INFO] Apache Hadoop Build Tools .......................... SUCCESS [ 22.983 s]
727.1 [INFO] Apache Hadoop Project POM .......................... SUCCESS [ 17.289 s]
727.1 [INFO] Apache Hadoop Annotations .......................... SUCCESS [ 14.348 s]
727.1 [INFO] Apache Hadoop Assemblies ........................... SUCCESS [  0.761 s]
727.1 [INFO] Apache Hadoop Project Dist POM ..................... SUCCESS [  9.856 s]
727.1 [INFO] Apache Hadoop Maven Plugins ........................ SUCCESS [ 50.921 s]
727.1 [INFO] Apache Hadoop MiniKDC .............................. SUCCESS [ 22.615 s]
727.1 [INFO] Apache Hadoop Auth ................................. SUCCESS [01:08 min]
727.1 [INFO] Apache Hadoop Auth Examples ........................ SUCCESS [ 21.367 s]
727.1 [INFO] Apache Hadoop Common ............................... FAILURE [01:04 min]
727.1 [INFO] Apache Hadoop NFS .................................. SKIPPED
...

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Hadoop on UBI10 #845

Description

Upstream issue

Investigation

Build error message

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

Hadoop on UBI10 #845

Description

Description

Upstream issue

Investigation

Build error message

Metadata

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

Issue actions