[Bug]: Re-registered services ignore DSTACK_ACME_SERVER after a gateway loses its state #4385

Steps to reproduce

1. Start the dstack server with a non-default ACME server, for example Let's Encrypt staging:

   DSTACK_ACME_SERVER=https://acme-staging-v02.api.letsencrypt.org/directory dstack server

2. Create a gateway on the Kubernetes backend with a domain, for example gateway.example.com,
   and run a service through it, for example my-service. Check the certificate issuer:

   openssl s_client -connect my-service.gateway.example.com:443 \
     -servername my-service.gateway.example.com </dev/null 2>/dev/null \
     | openssl x509 -noout -issuer

   The issuer is a Let's Encrypt staging CA ((STAGING) ...), as configured.

3. Make the gateway container restart in place, without deleting the pod. This is what happens
   after an OOM kill or a node reboot, and it can be triggered by sending SIGTERM to the
   container's PID 1, for example with
   kubectl exec -n <namespace> <gateway-pod> -- kill -TERM 1. Kubernetes restarts the
   container on a fresh filesystem, and the gateway is installed again. dstack keeps the replica
   running, reconnects, sees that the gateway has no services and registers them again.

4. Check the issuer again once the service answers. The certificate now comes from Let's Encrypt
   production.

Actual behaviour

After the gateway has lost its state, the re-registered services get their certificates from
certbot's default ACME server, Let's Encrypt production, instead of DSTACK_ACME_SERVER.

With staging configured, this happened on all four recoveries of a Kubernetes gateway that we
ran. Some were container restarts in place, and in the others the pod was recreated under the
same name. It happened even in recoveries where we also restarted the dstack server, which
we expected to resend the settings through init_gateways.

Consequences:

• Staging does not protect the production rate limits when a gateway recovers.
• With a CA that requires EAB (DSTACK_ACME_EAB_KID, DSTACK_ACME_EAB_HMAC_KEY), the
  certificates come from Let's Encrypt instead of that CA. We infer this from the code and have
  not tried it.

Expected behaviour

Re-registered services get certificates from the configured ACME server, with the configured
EAB credentials, just like services registered on a freshly provisioned gateway.

dstack version

0.22.4. The code references below are for master at fe6fe22e.

Server logs

Nothing is logged as an error. The re-registration looks like a normal registration.

Additional information

Cause (master fe6fe22e):

• The gateway keeps the ACME settings that the server sends to POST /api/config in its state
  file, /home/ubuntu/dstack/state-v2.json (src/dstack/_internal/proxy/gateway/app.py:35).
  When that file is lost, ACMESettings falls back to server=None
  (proxy/gateway/models.py:18-19), and run_certbot adds --server and the EAB options only
  when they are set (proxy/gateway/services/nginx.py:147-152).
• The server sends the settings (GatewayClient.submit_gateway_config,
  server/services/gateways/client.py:190, through configure_gateway_replica,
  server/services/gateways/__init__.py:885) in only two places:
  • when a replica is provisioned
    (server/background/pipeline_tasks/gateway_replicas.py:622);
  • in init_gateways (server/services/gateways/__init__.py:740, call at 796). This is a
    one-off job at server start (server/background/scheduled_tasks/__init__.py:43,
    DateTrigger), and it runs concurrently with the pipeline below, so a server restart
    doesn't reliably help.
• The pipeline's sync for running replicas (gateway_replicas.py) lists the gateway's services
  (667) and registers the missing ones (_perform_state_sync at 711, registration at 849-860).
  It never resends the settings first.

This is most visible on Kubernetes, where every container restart starts on a fresh
filesystem. It applies to any gateway that loses its state file while its replica stays
running.

Proposed fix (I have a patch and will open a PR that links this issue): in the running-replica
sync, call submit_gateway_config() before registering services, and only on ticks that
register something. The call is idempotent and cheap. If it fails, the tick skips registration
and retries on the next tick, instead of requesting a certificate from the wrong server.

An alternative is for the gateway to report whether it has received its config, for example in
the service list response, so the server resends it only when needed.