* Add Lium backend

* Refine Lium SSH access and volume support

* Use released gpuhunt and simplify Lium docs and tests

---------

Co-authored-by: Andrey Cheptsov <[email protected]>

`_prepare_startup_context` ran on every PROVISIONING and PULLING tick
and returned early whenever the runner submission had to wait for other
jobs: replica provisioning, the router IP, or referenced group IPs. The
early return also skipped the status handlers' own work: copying the
instance provisioning data into the job, failing the job when its
instance is terminated, the provisioning timeout, and shim polling.
With replica IP refs on newly provisioned instances, jobs waited for
IPs that only the skipped copy would set, so the run stayed in
provisioning forever.

* Replace `_prepare_startup_context` with `_prepare_runner_submission`,
  called right before the runner submission once the runner is up.
* Load volumes and interpolate secrets in the shim branch, which needs
  the interpolated `registry_auth`. The shim no longer waits for other
  jobs, so the image pull starts earlier.
* Terminate workers whose router failed or did not get an IP in time
  in `_terminate_if_router_unavailable`, before both statuses as
  before.
* Wait for every job in the replica to have a hostname, not only
  provisioning data, since shim submission no longer ensures it.
* Drop the unused `secrets` parameter of `_submit_job_to_runner`.

Fixes: dstackai#4331
Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]>

* Show owned and joined projects on user pages

* Simplify user project regression tests

---------

Co-authored-by: Andrey Cheptsov <[email protected]>

Pin uv to 0.13.3. uv embeds the list of Python builds it can install,
so previously the patch versions available to runs depended on the
latest uv at image build time. With a pinned uv, rebuilding the image
yields the same Python builds.

Install Python 3.11-3.15 into /opt/uv/python instead of a single
default version into root's home. Unlike /root, the shared directory is
readable by non-root users. Callers opt in by setting
UV_PYTHON_INSTALL_DIR; the job configurator will be updated separately
once the image is tested.

Drop --default: the python/python3 symlinks it created in
~/.local/bin were never on PATH.

A prerequisite for Python 3.15 support

Allow `python: "3.15"` in run configurations. The base image 0.16
preinstalls Python 3.15.0. This also fixes servers running on Python
3.15: runs without `python` default to the server's Python version,
which was rejected as unsupported.

Pods, clusters, network volumes, and container registry credentials
are managed via REST API v2 instead of the deprecated GraphQL API,
with the following exceptions and changes:

* Community Cloud pods are still created via GraphQL, since v2 does
  not support placement by country code. gpuhunt also still uses
  GraphQL.
* Clusters with `registry_auth` are no longer supported: v2 cluster
  creation does not accept registry credentials, and cluster member
  pods cannot be updated via `PATCH /v2/pods/{id}`.
* Spot offers are rejected, since v2 has no interruptible pods.
  gpuhunt does not produce Runpod spot offers anyway.
* The cluster internal IP is set in `update_provisioning_data()`,
  since v2 may assign it after the cluster is created.
* A cluster is deleted if its pods cannot be listed after creation or
  their count does not match, as dstack does not track it yet.
* A failed container registry credentials cleanup no longer fails
  provisioning of an already created pod.

Handle InvalidGroup.Duplicate and
InvalidPermission.Duplicate errors when instances
in the same fleet provision at once.

…i#4387)

Nginx answers some requests, e.g. ones with a malformed request line,
with 400 before the server-level `set $dstack_replica_hit 0` runs, so
the flag is logged empty. The stats collector read these 4-field lines as
pre-0.19.11 logs and counted them as replica hits, so stray internet
traffic could scale a service up from zero.

Tell the two cases apart by the trailing separator that the current log
format leaves when the flag is empty, and treat an empty flag as no hit.

Co-authored-by: Claude Opus 5.5 <[email protected]>

The running gateway replica pipeline re-registers services after a
gateway loses its state, e.g. when the gateway is reinstalled, but it
never resends the global config. Certificates for the re-registered
services are then requested from certbot's default ACME server instead
of the configured one (staging, or an EAB CA). Only a server restart
resent the config, through init_gateways.

Submit the config before any service registration. The call is
idempotent and happens only on ticks that register services. If it
fails, skip the sync until the next tick rather than issue a
certificate from the wrong server.

Co-authored-by: Claude Opus 5.5 <[email protected]>

* Prototype OrphanCleanupPipeline

* Clean up

* Fix orphan classification order

* Batch owner record lookups in orphan cleanup

* Log orphan resources of stuck pending instances

* Emit events for orphaned resources

* Add migrations for orphan cleanup

* Recommend separate migrations for concurrent indexes

* Clarify resource listing contract

* Register OrphanCleanupPipeline

* Pass provider_ref to backends

* Match orphaned resources of failed job provisioning by job

* Add provider_ref naming helpers

* Implement orphan resource listing for AWS

* Move AWS Elastic IP helpers to resources.py

* Enroll backends in orphan cleanup

* Mention orphan cleanup when instance termination gives up

* Document orphan cleanup settings

* Reword resource listing contract

* List AWS instances without Elastic IP permissions

* Detect orphan cleanup support of registered backends

* Report partial resource listings in orphan cleanup

* Ignore unsupported owner types in orphan cleanup

* Test OrphanCleanupPipeline fetching and processing

* Document orphan cleanup support for backends

* Fix AWS resource listing edge cases

* Simplify orphan cleanup result handling and logging

* Tidy orphan cleanup docstrings and tests

* Check orphan cleanup activity in worker instead of fetcher

* Skip orphan groups of owners changed around listing and drop backoff for skips

* Report orphan owner resources whose deletion is requested

* Load orphan owner jobs before instances

* Keep superseded job submissions until orphan cleanup

* Bound how long superseded submissions wait for orphan cleanup

* Clarify orphan cleanup filter docstring

* Fix compute group provider_ref TODO

* Replace em dash in release_eip docstring

* Derive superseded submission max wait from orphan cleanup interval

* Document orphan cleanup in contributing

* Record issued provider refs in a ledger

Placeholder instances carry the provider_ref of run_job() instead of jobs,
and ProviderRefModel keeps refs after owners are deleted, so superseded
job submissions no longer need to wait for orphan cleanup.

* Wrap AWS errors when deleting orphaned resources

* Support orphaned resources cleanup for RunPod

* Fix RunPod orphan cleanup enrollment and pods pagination

* Delete RunPod orphan pods without a region

* Use parse_datetime for API, logs and CLI datetimes

* Make orphan cleanup enrollment interval configurable

* Document that orphan reports are not deduplicated

Update the Dynamo example and the Dynamo tabs in concepts/services.md
to run etcd as its own replica group that the frontend and workers
reach via ${{ groups[0].replicas[0].IP_ADDRESS }}. etcd and NATS no
longer run inside the router job, so the docs no longer depend on
DSTACK_ROUTER_INTERNAL_IP, which is going to be deprecated.

The examples now use Dynamo 1.5.1 with vLLM workers. Dynamo's default
ZMQ event plane removes the need for NATS, and the frontend uses vLLM's
GLM parsers because Dynamo's glm47 parser drops GLM-4.5 tool call
arguments. Prefill workers publish KV events for KV-aware routing.

The Router section's Dynamo tab switches from Qwen3.8-Flash-Next to
GLM-4.5-Air-FP8, which Dynamo 1.5.1 supports.

Also fix the dynamo router description and a code comment that said
workers are discovered via etcd/NATS.

Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]>

stdout should carry only command output so that `--json` stays
parseable. Logging, fatal errors from `main()`, `warn()`, the update
notice, `get` "not found" errors, and preset mirroring warnings now
go to stderr.

`error_console` resolves `sys.stderr` on each write. A plain
`Console(stderr=True)` unwraps the proxy that `rich.live.Live`
installs, so its output would garble an active spinner or live table.
While `Live` is active, `error_console` also reports no fileno.
Otherwise, on consoles without VT support, rich writes and flushes
each styled segment separately, and the proxy prints each one on a
separate line.

Fixes: dstackai#4403
Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]>