fix(opensearch-operator): raise vm.max_map_count on every node from a DaemonSet - #4152

…anager (#2682)

## What this PR does

The OpenSearch HTTP API (9200) and Dashboards (5601) already served TLS,
from the operator's own private CA and with a SAN built from
`cluster.local` while the credentials Secret advertised the release on
the platform domain, so nothing a client was handed could verify. Both
certificates now come from a per-release cert-manager CA in the tenant
namespace, and the CA reaches tenants as a key-free trust anchor.

- `tls.issuer` selects who issues the HTTP server certificate,
`cert-manager` or `operator`, and is named for the issuer because TLS is
served under both. Unset resolves from `external` (cert-manager for
externally published services, the operator for cluster-internal); an
explicit value always wins.
- Renders a self-contained cert-manager chain in the tenant namespace: a
self-signed Issuer, a CA Certificate, a CA Issuer, and two leaf
Certificates, one for the HTTP API and one for Dashboards, each with the
SANs of its own Service and the external hostname when `external: true`.
Leaf keys are written as PKCS8, since the JVM loader rejects the
PKCS1/SEC1 encoding cert-manager emits by default, and the CA subject is
pinned so the operator's signature check accepts the chain.
- The CA Secret is `<release>.http-ca`. The dot separates the
chart-chosen suffix from release names, so no other release can produce
that name; the two leaf Secrets stay hyphenated because none of the
chart's suffixes is a tail of another. The operator's own transport CA
remains at `<release>-ca` and is not touched.
- Transport mTLS (9300) stays operator-managed (`transport.generate:
true`, `perNode: true`). The security plugin requires it and the chart
cannot turn it off.
- `hack/e2e-chainsaw/opensearch/` runs the move on a cluster: it
installs the app on the operator's HTTP listener, removes `tls.issuer`
so it resolves from `external`, and asserts the cert-manager chain, the
CR dropping `http.generate`, the admin certificate reissued under the
chart CA, the StatefulSet rolling onto the three subPath mounts, the
tenant anchor, and the certificate the endpoint actually presents to a
client holding that anchor.

### Sysctl carrier and single-node rolls

The E2E suite for this PR was red because the chart set
`spec.general.setVMMaxMapCount: true`, which makes the operator prepend
a privileged `init-sysctl` container to every OpenSearch pod. Tenant
namespaces run at the cluster's default PodSecurity level, baseline on
Talos, so admission refused the container and the StatefulSet never
produced a pod (#4131). The carrier for the sysctl is now on main: #4152
added a DaemonSet to the `opensearch-operator` package that raises
`vm.max_map_count` to a floor of 262144 on every node, from a namespace
the platform labels privileged. This branch is rebased onto that main
and sets the flag to an explicit `false`, with a template comment saying
why and a unit test pinning it, so a release no longer asks the operator
for a container admission refuses. The value stays with the DaemonSet
until the node contract carries it (#4144); the DaemonSet's own template
comment, which said the chart still asks, now says it does not.

Three more things came out of the same pass. The chart asked the
operator to drain data nodes before every rolling restart, and with a
single data node the drain never finishes: at v2.8.0
`PreparePodForDelete` excludes the node from allocation and then waits
for it to hold no shards, which the only data node can never satisfy, so
a one-replica release could not roll for a version bump or for the
certificate move this PR relies on. The chart now drains only when a
second data node exists, pinned both ways in the unit tests. The chart
maps an empty `storageClass` to `replicated` at one replica, and the PR
lane runs LINSTOR without DRBD and creates only `local`, so the E2E
fixture takes its class from the `storageClass` value the harness passes
with `--set-string` from `COZY_E2E_STORAGE_CLASS`, the way the
vminstance fixtures do; on the nightly QEMU lane that still resolves to
`replicated`. And the external Dashboards Service selected labels no pod
carries. `pkg/builders/dashboards.go` at v2.8.0 stamps
`opensearch.cluster.dashboards: <cluster name>` on the Deployment, its
selector and the pod template, and nothing else; the chart asked for
`opster.io/opensearch-cluster` together with
`app.kubernetes.io/component: dashboards`, so the LoadBalancer came up
with no endpoints and no error to show for it. It selects the label the
operator sets now.

### securityadmin cross-trust, still blocked on the DNS base

Chart-managed HTTP TLS previously broke the operator's securityadmin
reconciliation: the admin certificate was signed by the transport CA
while the HTTP listener trusted the cert-manager CA, so the two never
verified against each other and users and roles silently never applied.
This chart puts one CA under both sides, handing the HTTP CA to the
operator through `spec.security.tls.http.caSecret` so that it signs the
admin certificate with the CA the listener trusts. That closes the CA
mismatch. The other half, the job resolving a name the operator built
from `cluster.local` on a cluster that does not use it, was closed
separately and is on main, so the security configuration applies once
this lands. No operator bump is required; the API exists in the pinned
version.

That path is version-gated upstream at OpenSearch 2.0.0: below it the
operator falls back to the transport CA and the mismatch returns with no
error anywhere. The chart therefore refuses chart-managed HTTP TLS on
`version: v1` with an explicit render-time failure that names the
mechanism and offers the alternatives, rather than letting it fail
silently at runtime.

### Trust anchor for tenants

The HTTP server's private key is not exposed to tenants. The chart
renders a `TenantProjection` sentinel that names `<release>.http-ca` as
its source; the CA-extraction controller publishes a `ca.crt`-only copy
as `<release>.tenant-ca`, and the ResourceDefinition selects it with the
engine-agnostic `internal.cozystack.io/tenant-ca` label. The sentinel is
gated on the same condition as the chain, so a plaintext release renders
no sentinel instead of one stuck at `SourceNotFound`. Tenants get what
they need to verify the server and nothing more.

## Known limitations

These are stated rather than left to be discovered:

1. **securityadmin needed both trust and reachability, and both are
closed now.** This branch put one CA under the admin certificate and the
HTTP listener. The other half was the operator resolving
`<svc>.<ns>.svc.<DNS_BASE>` with `DNS_BASE` left at the vendored
`cluster.local`, so the job waited on a name that never resolved
whichever CA had signed; the platform now renders that value from
`networking.clusterDomain` and it is on main, which is what unblocks
this branch.
2. **The e2e suite reaches the move, not the chart bump that triggers
it.** The suite above arrives at the chart-managed chain by removing
`tls.issuer` from a running release, which leaves exactly the values an
upgraded release arrives with. The one step it cannot run is the one
where the previous chart version wrote the CR, because there is no
second chart version to install from in this tree. Everything downstream
of that write is exercised on a cluster.
3. **A reissue does not reach a running pod.** Neither service reloads
its certificate, and cert-manager reissues on any change to `external`,
the tenant host or the cluster domain, so an ordinary edit leaves the
pods serving material that no longer matches their own names with no
error, event or condition to show for it. Completing such a change is
manual. So is removing the CA private key cert-manager leaves behind
when TLS is turned off again, since the platform runs with
`enableCertificateOwnerRef: false` and deleting a `Certificate` does not
delete its `Secret`. `packages/apps/opensearch/README.md` carries both,
with the commands.

### Downstream repositories

`packages/apps/opensearch/values.schema.json` gains the `tls` object, so
`cozystack/terraform-provider-cozystack` needs the field modelled for
`cozystack_opensearch`; tracked in
cozystack/terraform-provider-cozystack#41.

### Upgrade impact

An existing `external: true` release with `tls.issuer` unset serves
operator-issued HTTP TLS today and moves to chart-managed TLS on its
first reconcile after this lands: the cluster rolls once and is
re-anchored on a new per-release CA, so any client pinned to the
operator's CA has to be repointed at the key-free `<release>.tenant-ca`.
Setting `tls.issuer: operator` before the upgrade keeps that half as it
is. Releases that are not external are untouched. Every existing release
rolls once regardless of `tls.issuer` and `external`, because the pod
template loses the operator's `init-sysctl` container; on a
single-replica release that roll is the first one the operator can
complete, since draining the only data node used to block every rolling
restart.

The published external-dns name changes on the same upgrade whatever
`tls.issuer` says, because the Services are gated on `external` alone:
they move from `<release>.<namespace>.<cluster-domain>`, an in-cluster
name that never resolved outside the cluster, to
`<release>.<tenant-host>` and `<release>-dashboards.<tenant-host>`.
external-dns is configured upsert-only here, so the old record is not
withdrawn and keeps pointing at the LoadBalancer until it is removed
from the zone by hand. That cleanup is per zone rather than per release
and nothing enumerates the records, so the README shows how to derive
the list from the cluster.

### Release note

```release-note
feat(opensearch)!: the OpenSearch HTTP API and Dashboards already served TLS, from the operator's own private CA and with a SAN built from cluster.local while the credentials Secret advertised the release on the platform domain; both certificates now come from a per-release cert-manager CA whose anchor the platform publishes into the tenant as <release>.tenant-ca. Any client that worked against the old endpoint was skipping verification, because no hostname it was handed could have matched that SAN. Existing external releases move from the operator's CA to the chart's on their first reconcile, and their published external-dns name moves from the in-cluster domain to <release>.<tenant-host> with the old record left in place; set tls.issuer: operator beforehand to opt out of the CA move. With dashboards.enabled the application name is limited to 41 characters, and to 32 when external is on as well; past either bound the chart fails the render and withholds every object in the release. A release already above a bound is not broken by the guard, it is told why it has not been upgrading: the Dashboards Service the operator builds from that name exceeds the 63-character limit for a DNS label, the operator returns that failure on every pass, and the reconcilers queued behind Dashboards, which are upgrade and restart and snapshot repository, have not run since the release was created. Set dashboards.enabled to false to drop the bound. The external Dashboards Service also selected labels the operator does not put on Dashboards pods, so it stood up with no endpoints; it selects opensearch.cluster.dashboards now. The drain of data nodes is now requested only with the data role on and more than two replicas, since at two the operator gives up on a full drain itself. The chart stops asking the operator for its privileged init-sysctl container, since the opensearch-operator package now sets vm.max_map_count from a DaemonSet, so every existing release rolls once on upgrade; single-replica releases no longer ask the operator to drain their only data node, which used to block every rolling restart
```