Automate, verify, and ship database backups off-box (#145) (#152)

The VPS now takes a daily logical dump, verifies it, encrypts it, and hands
it to a workstation; a monthly drill proves the archive restores.

- deploy/backup: pg_dump -Fc plus pg_dumpall --globals-only, pg_restore -l
  and globals-role checks before anything is promoted, row counts for the
  drill, sha256, age encryption into offsite/, retention (3d plaintext,
  14d encrypted, 5 pre-deploy sets), JSON logging and a healthchecks.io
  heartbeat
- systemd service and timer (03:15 UTC, Persistent, jittered) plus an
  OnFailure unit that logs a distinct event
- deploy/activate takes a pre-deploy dump after extracting and before
  flipping current, because migrations run in ExecStartPre; a failed dump
  aborts the activation, with SKIP_PRE_DEPLOY_BACKUP=1 as the escape hatch
- deploy/workstation/backup-pull: rsync the encrypted archives over SSH,
  verify by decrypting and checking the inner sha256, retain 7d/4w/6m, and
  fail when the newest archive is older than 48h
- deploy/workstation/backup-drill: restore the newest archive into a
  throwaway postgres:18.6, replay the globals dump as the same superuser,
  assert both roles, compare users/endgames/attempts, and append the result
  to drills.log
- user units for the daily pull and monthly drill, with a shared OnFailure
  alert template
- docs: install, key handling, retention, restore, disaster recovery, a run
  by hand section, monitoring and troubleshooting

Verified against production: the first backup completed and pinged the
heartbeat, the pull verified the archive, and the drill restored
users=1 endgames=39 attempts=0, matching the live database.
This commit is contained in:
2026-11-11 22:59:54 +01:00
committed by GitHub
parent 624ddab644
commit 9e76045862
15 changed files with 1240 additions and 36 deletions
+33
View File
@@ -0,0 +1,33 @@
# /etc/chesstrainer/backup.env -- root:root, mode 0600.
#
# Sourced by /usr/local/sbin/chesstrainer-backup. Every value below shows the
# default; only AGE_RECIPIENT and HEALTHCHECK_URL must be set. Keep secrets out
# of the repo and out of world-readable files.
# age public key, generated on the workstation with `age-keygen`.
# The private key never leaves the workstation.
AGE_RECIPIENT=age1...
# healthchecks.io (or compatible) ping URL. The script pings it after a good
# run and pings <url>/fail after a bad one. Set SKIP_HEARTBEAT=1 to run without
# monitoring (testing only).
HEALTHCHECK_URL=https://hc-ping.com/00000000-0000-0000-0000-000000000000
# Refuse to dump when the filesystem has less than this much free space.
MIN_FREE_MB=2048
# Retention. Dumps and their globals/counts/checksum siblings disappear as a
# set; only the encrypted archives in offsite/ live for the longer window.
LOCAL_PLAINTEXT_DAYS=3
LOCAL_ENCRYPTED_DAYS=14
PREDEPLOY_KEEP=5
# Postgres coordinates. The container trusts local Unix-socket connections, so
# no password belongs here.
DB_CONTAINER=chesstrainer-db
PG_USER=chesstrainer
PG_DB=chesstrainer_prod
BACKUP_DIR=/var/backups/chesstrainer
# Roles the globals dump must be able to recreate.
EXPECTED_ROLES="chesstrainer chesstrainer_ro"