[v24.x backport] backend split, KT12 limit, ByteSource MemoryRetainer + fixes and improvements - #65087

This separates the native crypto backend paths for OpenSSL >= 3,
BoringSSL, and legacy OpenSSL. The OpenSSL >= 3 path now builds with
`OPENSSL_API_COMPAT=30000` and `OPENSSL_NO_DEPRECATED`, moving normal
crypto/TLS code away from APIs that OpenSSL 3.0.0 marks deprecated.
BoringSSL remains on its own branch, and OpenSSL < 3 remains the legacy
fallback.

The exception is ENGINE support. ENGINE APIs are isolated into a
dedicated compatibility target so they can remain available while the
JS-facing engine APIs are runtime-deprecated in 27.x. That gives us a
clear removal point for 28.x, without letting ENGINE usage leak back
into the strict OpenSSL 3 path.

The split also makes the eventual OpenSSL 1.1.1 removal easier to reason
about. Once support for OpenSSL < 3 is dropped, the legacy branch can be
removed in a focused follow-up, possibly targeting 27.x, instead of
untangling mixed version guards throughout the crypto implementation.

No public crypto or TLS API behavior is intentionally changed.

Assisted-by: Codex:gpt-5
Signed-off-by: Filip Skokan <[email protected]>
PR-URL: #64211
Backport-PR-URL: #65087
Refs: #56733
Reviewed-By: Matteo Collina <[email protected]>
Reviewed-By: Richard Lau <[email protected]>
Reviewed-By: James M Snell <[email protected]>
Reviewed-By: Antoine du Hamel <[email protected]>

To align with a future OpenSSL-imposed limit.

Signed-off-by: Filip Skokan <[email protected]>
PR-URL: #64557
Backport-PR-URL: #65087
Reviewed-By: Luigi Pinca <[email protected]>
Reviewed-By: James M Snell <[email protected]>
Reviewed-By: Antoine du Hamel <[email protected]>

Limit the g >= p rejection needed by the provider backend to OpenSSL 3.
Preserve the legacy DiffieHellman object and expose the validation
result through verifyError as before.

Signed-off-by: Filip Skokan <[email protected]>
PR-URL: #64547
Backport-PR-URL: #65087
Refs: #64211
Reviewed-By: James M Snell <[email protected]>
Reviewed-By: Yagiz Nizipli <[email protected]>
Reviewed-By: Antoine du Hamel <[email protected]>

Treat missing private RSA parameters as an export failure instead of
passing null BIGNUM pointers to the JWK encoder. Also stop constructing
a usable RSA view when reading an optional parameter itself fails.

Signed-off-by: Filip Skokan <[email protected]>
PR-URL: #64547
Backport-PR-URL: #65087
Refs: #64211
Reviewed-By: James M Snell <[email protected]>
Reviewed-By: Yagiz Nizipli <[email protected]>
Reviewed-By: Antoine du Hamel <[email protected]>

Clear provider-exported RSA, EC, and DH private BIGNUMs before freeing
them. Also cleanse OSSL_PARAM builder copies and the plaintext DER
intermediate used for encrypted traditional PEM output.

Signed-off-by: Filip Skokan <[email protected]>
PR-URL: #64547
Backport-PR-URL: #65087
Refs: #64211
Reviewed-By: James M Snell <[email protected]>
Reviewed-By: Yagiz Nizipli <[email protected]>
Reviewed-By: Antoine du Hamel <[email protected]>

Recreate the historical rsaEncryption AlgorithmIdentifier used by
X509Certificate legacy pubkey output for restricted and unrestricted
RSA-PSS keys without relying on deprecated RSA APIs.

Signed-off-by: Filip Skokan <[email protected]>
PR-URL: #64547
Backport-PR-URL: #65087
Refs: #64211
Reviewed-By: James M Snell <[email protected]>
Reviewed-By: Yagiz Nizipli <[email protected]>
Reviewed-By: Antoine du Hamel <[email protected]>

Report DH failures instead of aborting or returning an empty secret.

Signed-off-by: Filip Skokan <[email protected]>
PR-URL: #64851
Backport-PR-URL: #65087
Reviewed-By: James M Snell <[email protected]>
Reviewed-By: Luigi Pinca <[email protected]>
Reviewed-By: Antoine du Hamel <[email protected]>

Create the mutex before sharing key data so every copy uses one lock.

Signed-off-by: Filip Skokan <[email protected]>
PR-URL: #64851
Backport-PR-URL: #65087
Reviewed-By: James M Snell <[email protected]>
Reviewed-By: Luigi Pinca <[email protected]>
Reviewed-By: Antoine du Hamel <[email protected]>

Report the correct parameter name for invalid optional inputs.

Signed-off-by: Filip Skokan <[email protected]>
PR-URL: #64852
Backport-PR-URL: #65087
Reviewed-By: James M Snell <[email protected]>
Reviewed-By: Yagiz Nizipli <[email protected]>
Reviewed-By: Antoine du Hamel <[email protected]>

Reject null options with ERR_INVALID_ARG_TYPE.

Signed-off-by: Filip Skokan <[email protected]>
PR-URL: #64852
Backport-PR-URL: #65087
Reviewed-By: James M Snell <[email protected]>
Reviewed-By: Yagiz Nizipli <[email protected]>
Reviewed-By: Antoine du Hamel <[email protected]>

Avoid a second native call when reading the issuer certificate.

Signed-off-by: Filip Skokan <[email protected]>
PR-URL: #64852
Backport-PR-URL: #65087
Reviewed-By: James M Snell <[email protected]>
Reviewed-By: Yagiz Nizipli <[email protected]>
Reviewed-By: Antoine du Hamel <[email protected]>

Explain that a cipher is required when a passphrase is provided.

Signed-off-by: Filip Skokan <[email protected]>
PR-URL: #64852
Backport-PR-URL: #65087
Reviewed-By: James M Snell <[email protected]>
Reviewed-By: Yagiz Nizipli <[email protected]>
Reviewed-By: Antoine du Hamel <[email protected]>

KEM encapsulation produces separate ciphertext and shared-secret
allocations. The existing DeriveBitsJob path packs both values into an
intermediate buffer, then copies them again into separate buffers.

Instead, this uses a dedicated KEMEncapsulateJob to retain both outputs
across the worker boundary and convert each directly through ByteSource.
This removes the intermediate allocation and at least one complete round
of copies.

Signed-off-by: Filip Skokan <[email protected]>
PR-URL: #64553
Backport-PR-URL: #65087
Reviewed-By: James M Snell <[email protected]>
Reviewed-By: Yagiz Nizipli <[email protected]>
Reviewed-By: Antoine du Hamel <[email protected]>

Signed-off-by: Filip Skokan <[email protected]>
PR-URL: #64660
Backport-PR-URL: #65087
Reviewed-By: James M Snell <[email protected]>
Reviewed-By: Antoine du Hamel <[email protected]>

Return an operation error when XOF output allocation fails.

Signed-off-by: Filip Skokan <[email protected]>
PR-URL: #64851
Backport-PR-URL: #65087
Reviewed-By: James M Snell <[email protected]>
Reviewed-By: Luigi Pinca <[email protected]>
Reviewed-By: Antoine du Hamel <[email protected]>

The ncrypto KDF helpers cleared the OpenSSL error queue on return, and
the traits insert their own message, which makes DeriveBitsJob skip
errors->Capture(). Argon2, HKDF, PBKDF2 and scrypt failures were
therefore bare Errors with no code and no opensslErrorStack.

Drop the guard, which DeriveBitsJob already provides, and capture
before inserting since Capture() clears the store.

Signed-off-by: Filip Skokan <[email protected]>
PR-URL: #64776
Backport-PR-URL: #65087
Reviewed-By: Yagiz Nizipli <[email protected]>
Reviewed-By: Antoine du Hamel <[email protected]>