[Bug]: Requests that nginx rejects with 400 count as service hits and scale services up from zero #4384

Steps to reproduce

1. Create a gateway with a domain, for example gateway.example.com, and run this service through
   it (authentication is on by default):

   type: service
   name: my-service
   image: python:3.12-slim
   commands:
     - python3 -m http.server 8000
   port: 8000
   replicas: 0..1
   scaling:
     metric: rps
     target: 1

2. Wait until the service is registered on the gateway with no replicas running.

3. Without any token, open a TLS connection to the service's domain and send a bare newline.
   This is what a common certificate check does:

   echo | openssl s_client -connect my-service.gateway.example.com:443 \
     -servername my-service.gateway.example.com

   Malformed requests from internet scanners had the same effect in our tests.

4. Within about 30 seconds, dstack ps shows a replica of my-service being provisioned,
   even though no request reached the service and none was authenticated.

On the gateway, nginx answered with 400 Bad Request, and the request's line in
/var/log/nginx/dstack.access.log has no hit flag. The line ends in a space where the
flag should be:

2026-01-01T12:00:00+00:00 my-service.gateway.example.com 400 0.000 

Actual behaviour

The stats collector counts such requests as replica hits. With rps scaling and zero
replicas, the autoscaler scales up as soon as it sees any hit, so a single malformed request
starts a replica, which then runs at least until scale_down_delay passes.

Service domains are public through Certificate Transparency logs. On our public gateway, a
newly certified service domain received 11 malformed requests from unknown clients within
35 minutes, and two of them started replicas of a scale-to-zero service. On a GPU service,
each of those starts rents an instance.

Expected behaviour

Requests that nginx rejects before routing them never reach a replica, so they should not
count toward scaling, just as requests rejected by authentication (403) or not found (404)
don't count.

dstack version

0.22.4. The code references below are for master at fe6fe22e.

Server logs

The server logs show only the ordinary scale-up and job submission for the service.

Additional information

Cause (master fe6fe22e):

• src/dstack/_internal/proxy/gateway/resources/nginx/service.jinja2 sets
  $dstack_replica_hit 0 at server level (line 17) and 1 in the locations that proxy to
  replicas (54, 77).
• nginx rejects a malformed request, or one with oversized headers, while it is still reading
  the request header. That happens before the server-level set runs, so the variable is
  uninitialized. nginx logs an uninitialized set variable as an empty string, so the
  dstack_stat format
  ($time_iso8601 $host $status $request_time $dstack_replica_hit, 00-log-format.conf:1)
  ends the line in a trailing space. By the same mechanism, the other errors nginx returns at
  that stage, such as 408, 414 and 494, should log an empty flag too; we have only seen 400.
• src/dstack/_internal/proxy/gateway/services/stats.py, _read_access_log (124-127), splits
  the line with line.split(), gets 4 cells and takes it for a pre-0.19.11 log line. In that
  case every status except 403 and 404 counts as a hit. The shim came with Fix request filtering for service stats #2678.
• src/dstack/_internal/server/services/services/autoscalers.py (90-92): with zero replicas
  the RPS autoscaler returns the new count immediately, without scale_up_delay.

Proposed fix (I have a patch and will open a PR that links this issue): split the line on
single spaces after stripping the newline. A current-format line with an empty flag then keeps
5 cells, and an empty flag means "not a hit". Legacy 4-field lines are parsed as before, and
the existing tests pass unchanged.

An alternative is to drop the pre-0.19.11 shim. Stats only use the last 5 minutes of the log,
so legacy lines would matter only in the first minutes after a very old gateway is updated.

Workarounds: check certificates with openssl s_client ... </dev/null, which sends nothing and
logs nothing, and don't use replicas: 0..N with rps scaling on a public gateway.