Comparing changes

Local pg_dump files sit on the same disk as Postgres, so losing the VPS loses both. This records the agreed design for porting the rclone off-host backup from submissions-checker (database only), opt-in via a compose profile, with a Grafana alert when backups go stale. Co-Authored-By: Claude Opus 5.5 <[email protected]>

Breaks the approved design into eight test-first tasks so the sidecar rewrite, compose, make targets, alerting, vps.sh and docs can each be reviewed on their own. Co-Authored-By: Claude Opus 5.5 <[email protected]>

A dump on the VPS disk dies with that disk. The sidecar now uploads each pg_dump to an off-host rclone remote, prunes by the stamp in the name only after a fully successful run, and publishes its outcome as a Prometheus textfile so a stalled backup can alert. Co-Authored-By: Claude Opus 5.5 <[email protected]>

run_once() ran inside a subshell where `set -e` does not apply, so an unguarded write_metrics call let a failed metric write pass silently: the run would log "complete" and return 0 even though the outcome never reached backup.prom, defeating the BackupStale alert. Guard it like the other steps in run_once, and add a test that makes BACKUP_METRICS_DIR a regular file so write_metrics' mkdir -p fails. Co-Authored-By: Claude Opus 5.5 <[email protected]>

Restoring must never make things worse: it now uploads a dump of the current database first, restores in one transaction, and resolves `latest` through last-success so a half-finished run is never picked. status gives a single freshness check for make targets and vps.sh. Co-Authored-By: Claude Opus 5.5 <[email protected]>

Backups need an rclone remote the template cannot know, so the sidecar becomes opt-in instead of silently writing dumps next to the database. rclone.conf is mounted read-write for token refresh and must exist, so a missing file fails loudly instead of turning into a directory. Co-Authored-By: Claude Opus 5.5 <[email protected]>

prod-restore now restores from the off-host remote by stamp and asks first. It stops Watchtower only if it was running, so restoring after a prod-rollback does not silently resume following latest. Co-Authored-By: Claude Opus 5.5 <[email protected]>

…rrors Three ways prod-restore could do the wrong thing quietly: a failed `stop` was never checked, so a failing stop still ran the restore against a live app while make exited 0; a failed `ps` was swallowed by `|| true`, so Watchtower could be assumed stopped and recreate a replica mid-restore; and STAMP was interpolated unquoted into the restore command, so a STAMP containing shell metacharacters could run arbitrary shell code. Validate STAMP against `latest` or the backup stamp shape before touching compose at all, quote it where it reaches the shell, and treat both `ps` and `stop` failures as "abort with nothing restored" rather than "carry on". Co-Authored-By: Claude Opus 5.5 <[email protected]>

$(STAMP) is make's own text substitution: it pastes the operator's value straight into the recipe's shell text before the shell ever parses it, so quoting it there does not help. A STAMP containing backticks (`` `touch x` ``) still ran as a command substitution during validation, even though round 1 added a case guard and quoted the value at the restore call. GNU make exports command-line variable overrides into each recipe line's shell environment automatically (verified against this Makefile), so reading STAMP as the shell parameter $$STAMP everywhere it appears in prod-restore — the case guard, the confirmation prompt, and the restore call — closes this off without relying on make-side quoting at all. Co-Authored-By: Claude Opus 5.5 <[email protected]>

Opt-in backups are easy to forget and a broken token fails silently. Alloy now reads the sidecar's textfile and BackupStale pages after two missed runs; no data stays quiet because backups may legitimately be off. Co-Authored-By: Claude Opus 5.5 <[email protected]>

The first place anyone looks is make vps-status; a disabled or stale backup now shows up there instead of only in Grafana, which may be off. The skill forbids Claude from running a restore itself. Co-Authored-By: Claude Opus 5.5 <[email protected]>

The old §6 told operators to copy dumps off the host themselves. It now walks through choosing a remote, verifying the first backup, restoring and undoing a restore, and records the new silent-failure invariants. Co-Authored-By: Claude Opus 5.5 <[email protected]>

pg_restore --clean drops only the objects the dump contains. Anything created after the dump survived: a new table referencing an old one made the restore fail, and a new table without references stayed next to an older databasechangelog, so Liquibase re-ran its createTable and the app crash-looped. The dump is now converted to SQL offline and applied by psql in one transaction that first replaces the public schema. Co-Authored-By: Claude Opus 5.5 <[email protected]>

rclone saves its config by writing a new file and renaming it over the old one. On a single-file bind mount that rename fails with EBUSY, so every OAuth token refresh logged "Failed to save config" and providers that rotate refresh tokens would lose access once the old one expired. Mounting the directory lets the rename happen inside it. Co-Authored-By: Claude Opus 5.5 <[email protected]>

On a new host the scheduler could upload the fresh empty database and make it `latest` before the restore ran, so the restore now happens with the profile off and an explicit stamp. Moving the always-on backup service behind a profile leaves the old container running, and turning backups off leaves the last backup.prom behind so BackupStale fires forever; both need a manual step, now documented. Co-Authored-By: Claude Opus 5.5 <[email protected]>

status read last-success with errors swallowed, so a remote it could not reach (network, expired credentials) was reported as "no successful backup yet" and sent the reader looking at the scheduler instead of the remote. It now checks the remote first; rclone's "directory not found" still counts as reachable, since a fresh remote has no folder yet. Co-Authored-By: Claude Opus 5.5 <[email protected]>

Compose accepts COMPOSE_PROFILES="backup", but vps.sh compared the raw value including its quotes and reported backups as DISABLED on a host where they run. One pair of surrounding quotes is now removed, as compose does. Co-Authored-By: Claude Opus 5.5 <[email protected]>

The restore piped `DROP SCHEMA public` and the converted dump into one psql call. sh has no pipefail, so a read error on the dump side still let psql commit the drop alone and the script reported success: the schema wiped, nothing restored. The drop and the dump now go into one file that psql runs with -f, so every step has its own exit status. Co-Authored-By: Claude Opus 5.5 <[email protected]>

deploy/backup/rclone/ holds the remote's OAuth tokens and sits next to the Dockerfile, so every build sent it to the builder. An allowlist .dockerignore sends only backup.sh, which also covers any file added there later; a probe build in deploy-config.test.sh guards it. Co-Authored-By: Claude Opus 5.5 <[email protected]>

…grees on CI failed on every push since the rclone directory mount landed: Compose v2 drops `create_host_path: false` from `docker compose config` output because false is its default, while v5 (used locally) prints it, so the rendered-output grep only passed on newer Compose. The test now reads the setting from compose.prod.yml and only uses the rendered output to reject `true` on that one mount. Co-Authored-By: Claude Opus 5.5 <[email protected]>

Navigation Menu

Repository navigation

Comparing changes

Open a pull request

Commits on Sep 28, 2026

This comparison is taking too long to generate.

Uh oh!