Files
owenrees 9e76045862 Automate, verify, and ship database backups off-box (#145) (#152)
The VPS now takes a daily logical dump, verifies it, encrypts it, and hands
it to a workstation; a monthly drill proves the archive restores.

- deploy/backup: pg_dump -Fc plus pg_dumpall --globals-only, pg_restore -l
  and globals-role checks before anything is promoted, row counts for the
  drill, sha256, age encryption into offsite/, retention (3d plaintext,
  14d encrypted, 5 pre-deploy sets), JSON logging and a healthchecks.io
  heartbeat
- systemd service and timer (03:15 UTC, Persistent, jittered) plus an
  OnFailure unit that logs a distinct event
- deploy/activate takes a pre-deploy dump after extracting and before
  flipping current, because migrations run in ExecStartPre; a failed dump
  aborts the activation, with SKIP_PRE_DEPLOY_BACKUP=1 as the escape hatch
- deploy/workstation/backup-pull: rsync the encrypted archives over SSH,
  verify by decrypting and checking the inner sha256, retain 7d/4w/6m, and
  fail when the newest archive is older than 48h
- deploy/workstation/backup-drill: restore the newest archive into a
  throwaway postgres:18.6, replay the globals dump as the same superuser,
  assert both roles, compare users/endgames/attempts, and append the result
  to drills.log
- user units for the daily pull and monthly drill, with a shared OnFailure
  alert template
- docs: install, key handling, retention, restore, disaster recovery, a run
  by hand section, monitoring and troubleshooting

Verified against production: the first backup completed and pinged the
heartbeat, the pull verified the archive, and the drill restored
users=1 endgames=39 attempts=0, matching the live database.
2026-11-11 22:59:54 +01:00

38 lines
1.4 KiB
Bash

# ~/.config/chesstrainer/backup-pull.env -- mode 0600.
#
# Sourced by ~/.local/bin/chesstrainer-backup-pull. Values below are the
# defaults; VPS_HOST and AGE_KEY must be right. Quote anything containing
# spaces.
VPS_HOST=owen@159.69.21.225
REMOTE_DIR=/var/backups/chesstrainer/offsite
DEST="/mnt/Mint Data/chesstrainer-backups"
# Private half of the age key. Back this file up somewhere safe: without it
# every archive is unreadable.
AGE_KEY="$HOME/.config/chesstrainer/age.key"
# Fail the pull when the newest archive from the VPS is older than this, which
# means the VPS backup timer has stopped.
MAX_STALENESS_HOURS=48
# Keep every archive from the last 7 days, then one per ISO week for 4 weeks,
# then one per calendar month for 6 months.
KEEP_DAILY_DAYS=7
KEEP_WEEKLY_WEEKS=4
KEEP_MONTHLY_MONTHS=6
# Restore drill. It runs in a throwaway container matching the production
# Postgres major and appends results to DRILL_LOG.
# DRILL_IMAGE=postgres:18.6
# DRILL_DB=chesstrainer_restore
# DRILL_LOG="$DEST/drills.log"
EXPECTED_ROLES="chesstrainer chesstrainer_ro"
# Optional second heartbeat check with a long period (for example 72h), so you
# hear about it when the workstation stops pulling. Leave empty to disable.
# PULL_HEALTHCHECK_URL=https://hc-ping.com/00000000-0000-0000-0000-000000000000
# Extra ssh options, space separated.
# SSH_OPTIONS="-o BatchMode=yes -o ConnectTimeout=10 -o ServerAliveInterval=15"